Device, method and graphical user interface for electronic device interacting with stylus

Abstract

Various embodiments disclosed herein provide devices, methods, and graphical user interfaces for electronic devices that interact with a stylus. In some embodiments, an application mode is changed in response to detecting that the stylus is being held. In some embodiments, the functionality of the stylus is changed based on the stylus grip arrangement. In some embodiments, the touch input functionality is changed when the stylus is being held. In some embodiments, a displayed identifier is changed based on finger manipulation input received by the stylus. In some embodiments, the finger manipulation input received by the stylus enables navigation within a menu. In some embodiments, an identifier is created based on the hovering distance of the stylus. In some embodiments, an interactive stylus tutorial is displayed. In some embodiments, stylus settings on an electronic device are maintained. In some embodiments, picking up the stylus and then making gestures on it changes tool properties. In some embodiments, the stylus touches change the active color.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 679,461, filed June 1, 2018, and U.S. Provisional Patent Application No. 62 / 729,869, filed September 11, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates in general to electronic devices that interact with a stylus, including but not limited to user interfaces on the display of the electronic device that are affected by sensor data received from the stylus. Background Technology

[0004] The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Examples of touch-sensitive surfaces include touchpads and touchscreen displays. These surfaces are widely used to manipulate user interfaces on displays.

[0005] However, touch input (including gesture input) offers limited and inefficient control over manipulating the user interface. Therefore, to achieve a specific goal, repetitive, complex, and / or cumbersome touch input may be required to manipulate the user interface. Summary of the Invention

[0006] Therefore, a robust mechanism is needed for manipulating the user interface of a display in an electronic device. Specifically, electronic devices require faster and more efficient methods and interfaces for manipulating the user interface. Such methods and interfaces optionally supplement or replace conventional methods for manipulating the user interface. These methods and interfaces reduce the amount, extent, and / or nature of input from the user, resulting in a more efficient human-machine interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0007] The disclosed devices and methods can reduce or eliminate the aforementioned defects and other problems associated with the user interface of electronic devices having touch-sensitive surfaces. In some embodiments, the electronic device is a desktop computer. In some embodiments, the electronic device is portable (e.g., a laptop, tablet, or handheld device). In some embodiments, the electronic device is a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the electronic device has a touchpad. In some embodiments, the electronic device has a touch-sensitive display (also referred to as a "touchscreen" or "touchscreen display"). In some embodiments, the electronic device has a graphical user interface (GUI), one or more processors, memory, and one or more modules, and a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI primarily through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, when the stylus is not in physical contact with the touch-sensitive surface, the user interacts with the GUI primarily through user interaction with the stylus. In some embodiments, when the user is holding the stylus, the user interacts with the GUI primarily through finger and / or hand contact and gestures on the stylus. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, spreadsheet creation, playing games, making and receiving phone calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0008] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes obtaining information about the current state of the stylus at the electronic device via the communication interface when the electronic device is in a first state. The method further includes determining, based on the information about the current state of the stylus, that a user is holding the stylus, and displaying a visual indication on the display that the electronic device is in a second state different from the first state. The method further includes maintaining the electronic device in the first state if it is determined that the user is not holding the stylus.

[0009] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes detecting input from the stylus on the touch-sensitive surface of the electronic device. The method further includes: in response to detecting the input and based on determining that the stylus is being held in a first grip arrangement, making a first change to content displayed on the display, wherein the first grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus. The method further includes: in response to detecting the input and based on determining that the stylus is being held in a second grip arrangement different from the first grip arrangement, making a second change to the content displayed on the display, wherein the second grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus, wherein the second change to the content displayed on the display is different from the first change to the content displayed on the display.

[0010] According to some embodiments, a method is performed in an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes detecting touch input on the touch-sensitive surface. The method further includes: in response to detecting the touch input on the touch-sensitive surface and based on determining that sensor data obtained from the stylus via the communication interface indicates that the stylus is being held by a user, performing a first operation in response to the touch input. The method further includes: in response to detecting the touch input on the touch-sensitive surface and based on determining that the stylus is not being held by the user, performing a second operation in response to the touch input, wherein the second operation is different from the first operation.

[0011] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes: obtaining finger manipulation data from the stylus via the communication interface when a plurality of user interface elements are displayed on the display, wherein the finger manipulation data includes information about one or more finger manipulation inputs received by the stylus. The method further includes: performing a first operation on at least a subset of the plurality of user interface elements in response to obtaining the finger manipulation data and based on determining that the finger manipulation data indicates a first finger manipulation input on the stylus. The method further includes: performing a second operation on at least a subset of the plurality of user interface elements in response to obtaining the finger manipulation data and based on determining that the finger manipulation data indicates a second finger manipulation input on the stylus that is different from the first finger manipulation input, wherein the second operation is different from the first operation.

[0012] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes displaying a selection user interface on the display comprising a plurality of selectable items, wherein a first item among the plurality of selectable items is currently selected within the selection user interface. The method further includes obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data includes information regarding one or more finger manipulation inputs received at the stylus. The method further includes: in response to obtaining the finger manipulation data and based on determining that the finger manipulation data satisfies a navigation criterion, altering the display of the selection user interface to indicate that focus moves to a second item among the plurality of selectable items. The method further includes: in response to obtaining the finger manipulation data and based on determining that the finger manipulation data does not satisfy the navigation criterion, maintaining the display of the selection user interface, wherein the first item among the plurality of selectable items is currently focused within the selection user interface.

[0013] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes obtaining input data from the stylus via the communication interface corresponding to input detected at the stylus. The method further includes: in response to obtaining the input data from the stylus and based on determining that a distance between the stylus and the touch-sensitive display satisfies a first distance threshold when the input is detected at the stylus, displaying a first user interface element corresponding to the input. The method further includes: in response to obtaining the input data from the stylus and based on determining that the distance between the stylus and the touch-sensitive display satisfies a second distance threshold when the input is detected at the stylus, abandoning the display of the first user interface element corresponding to the input.

[0014] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a display, a touch-sensitive surface, and a communication interface provided for communicating with a stylus. The method includes: in response to detecting that the stylus is near the electronic device, pairing the electronic device with the stylus. The method further includes: in response to pairing the stylus with the electronic device: displaying a first representation of a first gesture performed on the stylus on the display; obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data indicates finger manipulation input received by the stylus; and in response to obtaining the finger manipulation data, displaying a second representation of a second gesture performed on the stylus corresponding to the finger manipulation input received by the stylus on the display.

[0015] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes detecting movement of the stylus on the touch-sensitive surface. The method includes: in response to detecting the movement of the stylus, performing a stylus operation in a user interface displayed on the display based on the movement of the stylus. The method includes: after performing the stylus operation in the user interface, obtaining finger manipulation data via the communication interface indicating finger manipulation input received at the stylus. The method includes: in response to obtaining the finger manipulation data from the stylus, changing an attribute of the stylus operation in the user interface based on the finger manipulation input; and displaying a visual indication of the change in the attribute of the stylus operation on the display of the electronic device.

[0016] According to some embodiments, a method is performed at a first electronic device having one or more processors, non-transitory memory, a display, and a communication interface provided for communicating with a stylus. The method includes detecting input corresponding to the stylus, the stylus communicating with the first electronic device via the communication interface. The method includes: in response to detecting the input corresponding to the stylus: performing a first operation at the first electronic device based on determining that the first setting of the stylus has a first value; and performing a second operation at the first electronic device, different from the first operation, based on determining that the first setting of the stylus has a second value different from the first value, wherein the value of the first setting is determined based on input at a second electronic device with which the stylus has previously communicated.

[0017] According to some embodiments, a method is performed at an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, and a display. The method includes detecting a first input on the touch-sensitive surface corresponding to a color selection indication selected by a user. The method includes: in response to detecting the first input, displaying a color picker user interface on the display, wherein the color picker user interface includes a plurality of options for selecting a user-selected color. The method includes detecting a second input on the touch-sensitive surface corresponding to a specific option among the plurality of options for selecting a user-selected color. The method includes, in response to detecting the second input: designating a first color selected based on the specific option among the plurality of options for selecting a user-selected color as an active color; stopping the display of the color picker user interface upon detection of the end of the second input, based on determining that the second input is a continuation of the first input; and maintaining the display of the color picker user interface after the detection of the end of the second input, based on determining that the second input was detected after the end of the first input and while the color picker user interface continues to be displayed on the display.

[0018] According to some embodiments, an electronic device includes a touch-sensitive surface, a display, a communication interface provided for communicating with a stylus, one or more processors, a memory, and one or more programs; the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the operations described herein. According to some embodiments, a computer-readable storage medium stores instructions therein that, when executed by an electronic device having a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus, cause the electronic device to perform or cause to perform any of the operations described herein. According to some embodiments, a graphical user interface on an electronic device having a touch-sensitive surface, a display, a communication interface provided for communicating with a stylus, a memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements of elements displayed in any of the methods described herein, the elements being updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes a touch-sensitive surface, a display, a communication interface provided for communicating with a stylus, and means for performing or causing to perform any of the operations described herein. According to some embodiments, an information processing apparatus used in an electronic device having a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus includes means for performing or causing to perform any of the methods described herein.

[0019] Therefore, electronic devices with touch-sensitive surfaces and communication interfaces provided for communicating with styluses utilize data received from the stylus. The received data indicates user input detected at the stylus's sensors. The sensors at the stylus detect various user inputs and provide data indicating these inputs to the electronic device. Based on the received data, the electronic device performs various operations, such as drawing and navigation. Thus, the electronic device can perform various operations without receiving input at its touch-sensitive surface. This improves the functionality of the electronic device in several ways, including longer battery life and less wear and tear. Additionally, the improved user interface enables more efficient and accurate user interaction with the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0021] Figure 1A This is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some embodiments.

[0022] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some embodiments.

[0023] Figure 2 A portable multi-functional device with a touchscreen is shown according to some embodiments.

[0024] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some embodiments.

[0025] Figure 4 This is a block diagram of an exemplary electronic stylus according to some embodiments.

[0026] Figures 5A to 5B The positional state of the stylus relative to the touch-sensitive surface is shown according to some embodiments.

[0027] Figure 6A An exemplary user interface for an application menu on a portable multi-functional device, according to some embodiments, is shown.

[0028] Figure 6B An exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0029] Figures 7A to 7Y An exemplary user interface for changing the state of an application is shown according to some embodiments.

[0030] Figures 8A to 8H An exemplary user interface for changing stylus functions is shown according to some embodiments.

[0031] Figures 9A to 9P An exemplary user interface for modifying touch input functionality is shown according to some embodiments.

[0032] Figures 10A to 10I An exemplary user interface for performing operations on existing identifiers based on finger manipulation input is shown according to some embodiments.

[0033] Figures 11A to 11O An exemplary user interface for performing finger manipulations on a stylus to navigate within a menu, according to some embodiments, is shown.

[0034] Figures 12A to 12O An exemplary user interface for displaying user interface elements based on the hover distance of a stylus, according to some embodiments, is shown.

[0035] Figure 13A This is a flowchart illustrating a method for processing sensor data collected at a stylus according to some embodiments.

[0036] Figure 13B This is a flowchart illustrating another method for processing sensor data collected at the stylus, according to some embodiments.

[0037] Figures 14A to 14C This is a flowchart illustrating a method for changing the state of an application according to some embodiments.

[0038] Figures 15A to 15B This is a flowchart illustrating a method for changing stylus functionality according to some embodiments.

[0039] Figures 16A to 16B This is a flowchart illustrating a method for modifying touch input functionality according to some embodiments.

[0040] Figures 17A to 17C This is a flowchart illustrating a method for performing operations on an existing identifier based on finger manipulation input according to some embodiments.

[0041] Figures 18A to 18B This is a flowchart illustrating a method for performing finger manipulations on a stylus to navigate within a menu, according to some embodiments.

[0042] Figures 19A to 19C This is a flowchart illustrating a method for displaying user interface elements based on stylus hover distance according to some embodiments.

[0043] Figures 20A to 20W This is an illustration of an exemplary user interface that provides an interactive stylus tutorial according to some embodiments.

[0044] Figures 21A to 21AB This is an illustration of an exemplary user interface for selecting stylus settings and drawing logos based on stylus settings, according to some embodiments.

[0045] Figures 22A to 22G This is an illustration of an exemplary user interface for maintaining stylus settings on an electronic device, according to some embodiments.

[0046] Figures 23A to 23Z This is an illustration of an exemplary user interface including a color picker user interface for assigning active colors, according to some embodiments.

[0047] Figures 24A to 24C This is a flowchart illustrating an exemplary user interface for providing an interactive stylus tutorial according to some embodiments.

[0048] Figures 25A to 25B This is a flowchart illustrating a method for displaying an exemplary user interface for selecting stylus settings and drawing icons based on stylus settings, according to some embodiments.

[0049] Figures 26A to 26BThis is a flowchart illustrating a method for maintaining stylus settings on an electronic device according to some embodiments.

[0050] Figures 27A to 27C This is a flowchart illustrating a method for displaying an exemplary user interface, including a color picker user interface for assigning active colors, according to some embodiments. Detailed Implementation

[0051] Many electronic devices include touch-sensitive surfaces that allow users to manipulate the user interface. For example, finger strokes on a touch-sensitive surface draw lines on a canvas in a drawing application. However, existing methods for manipulating user interfaces are slow, cumbersome, and inefficient. For instance, the number of types of input that can appear at an electronic device in response to surface touch is limited. Even multi-step surface touch (such as gestures) offers relatively few types of input because they still require touch contact with the screen.

[0052] The following embodiments address these problems by providing a separate tool (e.g., a stylus) that utilizes the user's numerous and complex hand and finger manipulations. Hand and finger manipulations arise at the stylus and are provided to the electronic device. Therefore, the user is not required to constrain his / her hands and fingers to the screen to manipulate the user interface, thus allowing for greater utilization of hand and finger manipulations. These manipulations provide the electronic device with a richer and more robust set of instructions than can be achieved using touchscreen input alone.

[0053] under, Figures 1A to 1B , Figures 2 to 4 , Figures 5A to 5B as well as Figures 6A to 6B A description of an exemplary device is provided. Figures 7A to 7Y An exemplary user interface for changing the state of an application is shown according to some embodiments. Figures 7A to 7Y The user interface in the document is used to show Figures 14A to 14C The process in. Figures 8A to 8H An exemplary user interface for changing stylus functions is shown according to some embodiments. Figures 8A to 8H The user interface in the document is used to show Figures 15A to 15B The process in. Figures 9A to 9P An exemplary user interface for modifying touch input functionality is shown according to some embodiments. Figures 9A to 9P The user interface in the document is used to show Figures 16A to 16B The process in. Figures 10A to 10I An exemplary user interface for performing operations on existing identifiers based on finger manipulation input is shown according to some embodiments. Figures 10A to 10I The user interface in the document is used to show Figures 17A to 17C The process in. Figures 11A to 11O An exemplary user interface for performing finger manipulations on a stylus to navigate within a menu, according to some embodiments, is shown. Figures 11A to 11O The user interface in the document is used to show Figures 18A to 18B The process in. Figures 12A to 12O An exemplary user interface for displaying user interface elements based on the hover distance of a stylus, according to some embodiments, is shown. Figures 12A to 12O The user interface in the document is used to show Figures 19A to 19C The process in. Figures 20A to 20W An exemplary user interface for providing interactive stylus tutorials is shown according to some embodiments. Figures 20A to 20W The user interface in the document is used to show Figures 24A to 24C The process in. Figures 21A to 21AB An exemplary user interface for selecting stylus settings and drawing logos based on stylus settings is shown according to some embodiments. Figures 21A to 21AB The user interface in the document is used to show Figures 25A to 25B The process in. Figures 22A to 22G An exemplary user interface for maintaining stylus settings on an electronic device is shown according to some embodiments. Figures 22A to 22G The user interface in the document is used to show Figures 26A to 26B The process in. Figures 23A to 23Z An exemplary user interface, including a color picker user interface for assigning active colors, is shown according to some embodiments. Figures 23A to 23Z The user interface in the document is used to show Figures 27A to 27C The process in.

[0054] Exemplary device

[0055] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0056] It will also be understood that, although in some cases the terms “first,” “second,” etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0057] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0058] As used herein, depending on the context, the term “if” is optionally interpreted as meaning “when…” followed by “at…” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrase “if determination…” or “if [the stated condition or event] is optionally interpreted as meaning “in response to determination…” or “in response to detection of [the stated condition or event]” or “in response to detection of [the stated condition or event].”

[0059] This document describes embodiments of electronic devices, user interfaces of such devices, and related processes for using such devices. In some embodiments, the electronic device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functionality. Exemplary embodiments of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). iPod and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the electronic device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0060] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0061] The electronic device typically supports a variety of applications, such as one or more of the following: note-taking applications, drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0062] Various applications running on electronic devices optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the electronic device (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0063] Now let’s turn our attention to examples of portable devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 having a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touchscreen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The electronic device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. The electronic device 100 optionally includes one or more optical sensors 164. The electronic device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the electronic device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the electronic device 100). Electronic device 100 optionally includes one or more tactile output generators 163 for generating tactile output on electronic device 100 (e.g., generating tactile output on a touch-sensitive surface such as a touch-sensitive display system 112 of electronic device 100 or a touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0064] As used in this specification and claims, the term "haptic output" refers to the physical displacement of an electronic device relative to a previous position of the electronic device, the physical displacement of a component of the electronic device (e.g., a touch-sensitive surface) relative to another component of the electronic device (e.g., a housing), or the displacement of a component relative to the center of mass of the electronic device, which is detected by a user using their tactile sense. For example, when an electronic device or a component of the electronic device comes into contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the electronic device or a component of the electronic device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. For example, even when the smoothness of the touch-sensitive surface remains unchanged, the movement of the touch-sensitive surface can optionally be interpreted or sensed by the user as the "roughness" of the touch-sensitive surface. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "release click", "press click", "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the electronic device or its components, which will generate the sensory perception of a typical (or ordinary) user.

[0065] It should be understood that electronic device 100 is merely an example of a portable multifunctional device, and electronic device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application-specific integrated circuits).

[0066] Memory 102 optionally includes high-speed random access memory, and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of electronic device 100 (such as CPU 120 and peripheral interface 118) is optionally controlled by memory controller 122.

[0067] Peripheral interface 118 can be used to couple input and output peripheral devices of electronic device to memory 102 and one or more CPUs 120. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of electronic device 100 and process data.

[0068] In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0069] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals and vice versa, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). This wireless communication optionally employs any of a variety of communication standards, protocols, and / or technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolved Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE...). 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including those not yet developed as of the date of this document submission.

[0070] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and electronic device 100. Audio circuitry 110 receives audio data from peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral device interface 118 for processing. Audio data is optionally retrieved by peripheral device interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a mono or binaural headphone) and input (e.g., a microphone).

[0071] I / O subsystem 106 couples input / output peripherals on electronic device 100, such as touch-sensitive display system 112 and other input or control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from / send electrical signals to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, stylus, and / or pointing device such as mouse. One or more buttons (e.g., Figure 2 Optionally, 208) includes an up / down button for volume control of speaker 111 and / or microphone 113. One or more buttons optionally include a push-down button (e.g., Figure 2 (206 in the middle).

[0072] The touch-sensitive display system 112 provides an input and output interface between the electronic device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. Visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, “graphics”). In some embodiments, some or all of the visual output corresponds to a user interface object. As used herein, the term “enabled representation” refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to it). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0073] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or tactile contact. The touch-sensitive display system 112 and the display controller 156 (along with any associated modules and / or instruction sets in memory 102) detect touches on the touch-sensitive display system 112 (and any movement or interruption of those touches) and translate the detected touches into interactions with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In one exemplary embodiment, the contact point between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0074] The touch-sensitive display system 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed thereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In one exemplary embodiment, projected mutual capacitance sensing technology, such as that from Apple Inc. (Cupertino, California), is used. iPod and The technology discovered in [the text].

[0075] The touch-sensitive display system 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or greater). Users optionally use any suitable object or accessory such as a stylus, finger, etc., to interact with the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the electronic device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0076] In some embodiments, in addition to a touchscreen, the electronic device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the electronic device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0077] Electronic device 100 also includes a power system 162 for supplying power to various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0078] The electronic device 100 optionally also includes one or more optical sensors 164. Figure 1A An optical sensor coupled to an optical sensor controller 158 in I / O subsystem 106 is shown. One or more optical sensors 164 optionally include charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) phototransistors. The one or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with imaging module 143 (also referred to as a camera module), the one or more optical sensors 164 optionally capture still images and / or video. In some embodiments, the optical sensor is located on the rear of the electronic device 100, opposite to the touch-sensitive display system 112 on the front of the electronic device, enabling the touchscreen to function as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the electronic device to acquire images of the user (e.g., for selfies, for video conferencing while the user views other video conferencing participants on the touchscreen, etc.).

[0079] The electronic device 100 optionally also includes one or more contact strength sensors 165. Figure 1A A contact strength sensor coupled to a strength sensor controller 159 in I / O subsystem 106 is shown. One or more contact strength sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact strength sensors 165 receive contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of electronic device 100, opposite to the touchscreen display system 112 located on the front of the electronic device 100.

[0080] The electronic device 100 optionally also includes one or more proximity sensors 166. Figure 1A A proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 is coupled to an input controller 160 in an I / O subsystem 106. In some embodiments, when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call), the proximity sensor is turned off and the touch-sensitive display system 112 is disabled.

[0081] The electronic device 100 optionally also includes one or more tactile output generators 163. Figure 1AA haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 163 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on electronic devices). The haptic output generator 163 receives haptic feedback generation instructions from haptic feedback module 133 and generates a haptic output on electronic device 100 that can be felt by a user of electronic device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of electronic device 100) or laterally (e.g., backward and forward in the same plane as the surface of electronic device 100). In some embodiments, at least one tactile output generator sensor is located on the rear of the electronic device 100, opposite to the touch-sensitive display system 112 located on the front of the electronic device 100.

[0082] Electronic device 100 may optionally also include one or more accelerometers 167, gyroscopes 168 and / or magnetometers 169 (e.g., as part of an inertial measurement unit (IMU)) for obtaining information about the position (e.g., attitude) of the electronic device. Figure 1A Sensors 167, 168, and 169, coupled to peripheral interface 118, are shown. Alternatively, sensors 167, 168, and 169 may be coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from the one or more accelerometers. Electronic device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the position and orientation (e.g., portrait or landscape) of electronic device 100.

[0083] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a position module (or instruction set) 131, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 stores device / global internal state 157, as shown in Figures 1A and 1B. Figure 3As shown in the diagram. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views or other information occupy the various areas of the touch-sensitive display system 112; sensor state, including information obtained from the various sensors of the electronic device and other input or control devices 116; and position and / or orientation information regarding the position and / or attitude of the electronic device.

[0084] Operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0085] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, Wireless LAN, etc.). In some embodiments, the external ports are compatible with some Apple Inc. (Cupertino, California) devices. iPod and The device uses the same or similar and / or compatible multi-pin (e.g., 30-pin) connectors as the 30-pin connector used in the device. In some embodiments, the external port is compatible with some Apple Inc. (Cupertino, California) connectors. iPod and The device uses the same or similar and / or compatible Lightning connectors.

[0086] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., via a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations can optionally be applied to single-point contact (e.g., single-finger contact or stylus contact) or multi-point simultaneous contact (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0087] The contact / motion module 130 optionally detects user gesture input. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the icon location). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event. Similarly, stylus taps, swipes, drags, and other gestures are optionally detected by detecting specific contact patterns of the stylus.

[0088] In conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169, position module 131 optionally detects position information about the electronic device, such as the attitude of the electronic device in a particular reference frame (e.g., roll, pitch, and / or yaw). Position module 130 includes software components for performing various operations related to detecting the position of the electronic device and detecting changes in the position of the electronic device. In some embodiments, position module 131 uses information received from a stylus used with the electronic device to detect position information about the stylus, such as detecting the positional state of the stylus relative to the electronic device and detecting changes in the positional state of the stylus.

[0089] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, and non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0090] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like to specify the graphic to be displayed, and, if necessary, also receives coordinate data and other graphic attribute data, and then generates screen image data to output to the display controller 156.

[0091] The haptic feedback module 133 includes various software components for generating instructions that are used by the haptic output generator 163 to generate haptic output at one or more locations on the electronic device 100 in response to user interaction with the electronic device 100.

[0092] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0093] GPS module 135 determines the location of the electronic device and provides that information for use in various applications (e.g., to phone 138 for location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services such as weather desktop apps, local yellow pages desktop apps, and map / navigation desktop apps).

[0094] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0095] • Contacts module 137 (sometimes called address book or contact list);

[0096] • Telephone module 138;

[0097] • Video conferencing module 139;

[0098] • Email client module 140;

[0099] • Instant Messaging (IM) module 141;

[0100] Fitness support module 142;

[0101] • Camera module 143 for still images and / or video images;

[0102] • Image management module 144;

[0103] • Browser module 147;

[0104] • Calendar module 148;

[0105] • Desktop mini-program module 149, which optionally includes one or more of the following: weather desktop mini-program 149-1, stock market desktop mini-program 149-2, calculator desktop mini-program 149-3, alarm clock desktop mini-program 149-4, dictionary desktop mini-program 149-5, and other desktop mini-programs obtained by the user, and desktop mini-programs created by the user 149-6;

[0106] • Desktop app creator module 150 for creating user-created desktop apps 149-6;

[0107] • Search module 151;

[0108] • A video and music player module 152, optionally composed of a video player module and a music player module;

[0109] • Memo module 153;

[0110] • Map module 154;

[0111] • Online video module 155; and / or

[0112] • Annotation application 195, for providing annotations to the user interface and optionally storing and / or accessing saved annotations 196 in memory 102.

[0113] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0114] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contact module 137 includes executable instructions for managing an address book or contact list (e.g., in the application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communication via telephone 138, video conferencing 139, email 140, or instant messaging 141; and so on.

[0115] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 includes executable instructions for performing the following operations: inputting a character sequence corresponding to a telephone number, accessing one or more telephone numbers in the address book 137, modifying an input telephone number, dialing a corresponding telephone number, initiating a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communication optionally employs any of a variety of communication standards, protocols, and technologies.

[0116] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0117] Incorporating RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0118] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for internet-based instant messaging), receiving an instant message, and viewing a received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0119] Incorporating RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (in sports equipment and smartwatches); receiving fitness sensor data; calibrating sensors used for monitoring fitness; selecting and playing music for fitness activities; and displaying, storing, and transmitting fitness data.

[0120] In conjunction with the touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for performing the following operations: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0121] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, displaying (e.g., in a digital slideshow or photo album), and storing still images and / or video images.

[0122] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) according to user instructions.

[0123] Incorporating RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0124] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, desktop applet module 149 is optionally a micro-application downloaded and used by a user (e.g., weather desktop applet 149-1, stock market desktop applet 149-2, calculator desktop applet 149-3, alarm clock desktop applet 149-4, and dictionary desktop applet 149-5) or a user-created micro-application (e.g., user-created desktop applet 149-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop applet includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop applet).

[0125] Combining RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, desktop applet creator module 150 includes executable instructions for creating desktop applets (e.g., transferring user-specified portions of web pages to desktop applets).

[0126] In conjunction with the touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0127] In conjunction with the touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, the video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on the touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, the electronic device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0128] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0129] Combining RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and map-related data (e.g., driving routes; data on shops and other points of interest at or near specific locations; and other location-based data) according to user instructions.

[0130] In conjunction with the touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes executable instructions that allow users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen 112 or on an external display connected wirelessly or via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, the instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos.

[0131] Each module and application identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures described above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0132] In some embodiments, electronic device 100 is an electronic device whose operation of a predefined set of functions is uniquely performed via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating electronic device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on electronic device 100 is optionally reduced.

[0133] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates the electronic device 100 from any user interface displayed on the electronic device 100 to a main menu, home menu, or root menu. In such embodiments, a touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0134] Figure 1B This is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A (middle) or memory 370 ( Figure 3 The (in the middle) includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0135] Event classifier 170 receives event information and determines the application 136-1 to which the event information should be delivered and the application view 191 of application 136-1. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display system 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0136] In some embodiments, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0137] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display system 112 as part of a multi-touch gesture). Peripheral device interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (via audio circuitry 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or touch-sensitive surfaces.

[0138] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0139] In some embodiments, the event classifier 170 further includes a hit view determination module 172 and / or an activity event recognizer determination module 173. When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that a user can see on the display.

[0140] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0141] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that forms an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.

[0142] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, the higher-level view in the hierarchy will still remain the actively participating view.

[0143] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver module 182.

[0144] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.

[0145] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0146] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0147] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the electronic device rotating from one orientation to another (e.g., from a portrait orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the current orientation of the electronic device (also referred to as device orientation).

[0148] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display system 112, and lifting off the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0149] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on a touch-sensitive display system 112, when a touch is detected on the touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0150] In some embodiments, the definition of the corresponding event 187 also includes delay actions that delay the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0151] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0152] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, identifiers, and / or lists indicating how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists indicating how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists indicating whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0153] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and delaying) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws an identifier associated with the identified event, and the event handler 190 associated with that identifier receives the identifier and executes a predefined procedure.

[0154] In some embodiments, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and executes a predetermined procedure.

[0155] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or video files used in storage video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0156] In some embodiments, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0157] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; movement of the electronic device; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0158] Figure 2 It is shown that some embodiments have a touch screen (e.g., Figure 1AA portable multi-functional device 100 (a touch-sensitive display system 112). The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figures) or one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of one or more graphics occurs when the user interrupts contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the electronic device 100. In some embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0159] The stylus 203 includes a first end 276 and a second end 277. In various embodiments, the first end 276 corresponds to the tip of the stylus 203 (e.g., the tip of a pencil), and the second end 277 corresponds to the opposite end or bottom of the stylus 203 (e.g., the eraser of a pencil).

[0160] The stylus 203 includes a touch-sensitive surface 275 to receive touch input from a user. In some embodiments, the touch-sensitive surface 275 corresponds to a capacitive touch element. The stylus 203 includes a sensor or group of sensors that detect input from the user based on tactile and / or sensory contact with the touch-sensitive surface 275. In some embodiments, the stylus 203 includes any of a variety of touch sensing technologies now known or to be developed thereafter, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive surface 275, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies. Because the stylus 203 includes a variety of sensors and sensor types, the stylus 203 can detect a wide variety of different inputs from the user, including gestures disclosed herein with respect to the touchscreen of the portable multifunction device 100. In some embodiments, one or more sensors can detect a single touch input or continuous touch input in response to a user tapping once or multiple times on the touch-sensitive surface 275. In some embodiments, one or more sensors can detect a swipe input on the stylus 203 in response to a user swiping along the touch-sensitive surface 275 with one or more fingers. In some embodiments, if the speed at which a user swipes along the touch-sensitive surface 275 exceeds a threshold, one or more sensors detect a flick input instead of a swipe input.

[0161] The stylus 203 also includes one or more sensors, such as an accelerometer, magnetometer, gyroscope, etc., for detecting the orientation (e.g., angular position relative to the electronic device) and / or movement of the stylus 203. The one or more sensors can detect various rotational movements of the stylus 203 by the user, including the type and direction of rotation. For example, the one or more sensors can detect the user rolling and / or rotating the stylus 203, and can detect the direction of rolling / rotating (e.g., clockwise or counterclockwise). In some embodiments, the detected input depends on the angular position of the first end 276 and the second end 277 of the stylus 203 relative to the electronic device. For example, in some embodiments, if the stylus 203 is substantially perpendicular to the electronic device and the second end 277 (e.g., an eraser) is closer to the electronic device, contacting the surface of the electronic device with the second end 277 results in an erasing operation. On the other hand, if the stylus 203 is substantially perpendicular to the electronic device and the first end 276 (e.g., a tip) is closer to the electronic device, contacting the surface of the electronic device with the first end 276 results in a marking operation.

[0162] Electronic device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on electronic device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on a touchscreen display.

[0163] In some embodiments, the electronic device 100 includes a touchscreen display, a menu button 204, a push-button 206 for powering on / off the electronic device and for locking the electronic device, a volume control button 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the electronic device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the electronic device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the electronic device or initiate an unlocking process. In some embodiments, the electronic device 100 also accepts voice input via a microphone 113 for activating or deactivating certain functions. The electronic device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112 and / or one or more haptic output generators 163 for generating haptic outputs for a user of the electronic device 100.

[0164] Figure 3This is a block diagram of an exemplary multi-functional device 300 having a display and a touch-sensitive surface according to some embodiments. The electronic device 300 need not be portable. In some embodiments, the electronic device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). The electronic device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. The device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on the electronic device 300 (e.g., similar to the reference above). Figure 1A The tactile output generator 163 and sensor 359 (e.g., similar to the one mentioned above) are described. Figure 1A The sensors 112, 164, 165, 166, 167, 168, and 169 are touch sensors, optical sensors, contact strength sensors, proximity sensors, acceleration sensors, attitude sensors, and / or magnetic sensors. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores data with portable multifunction device 100 (…). Figure 1A The programs, modules, and data structures stored in the memory 102 of the portable multifunction device 100 are similar to, or subsets thereof. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 ( Figure 1A The memory 102 may optionally not store these modules.

[0165] Figure 3Each of the elements identified above is optionally stored in one or more of the previously mentioned memory devices. Each of the modules identified above corresponds to an instruction set for performing the functions described above. The modules or programs identified above (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures described above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0166] Figure 4 This is a block diagram of an exemplary electronic stylus 203 according to some embodiments. The electronic stylus 203 is sometimes simply referred to as a stylus. The stylus 203 includes a memory 402 (which optionally includes one or more computer-readable storage media), a memory controller 422, one or more processing units (CPUs) 420, a peripheral device interface 418, RF circuitry 408, an input / output (I / O) subsystem 406, and other input or control devices 416. The stylus 203 optionally includes an external port 424 and one or more optical sensors 464. The stylus 203 optionally includes one or more intensity sensors 465 for detecting the intensity of contact of the stylus 203 on device 100 (e.g., when the stylus 203 is used with a touch-sensitive surface such as a touch-sensitive display system 112 of electronic device 100) or on other surfaces (e.g., a table surface). The stylus 203 optionally includes one or more haptic output generators 463 for generating haptic output on the stylus 203. These components can optionally communicate via one or more communication buses or signal lines 403.

[0167] In some embodiments, the term "haptic output" discussed above refers to the physical displacement of an accessory (e.g., stylus 203) of an electronic device (e.g., electronic device 100) relative to a previous position of the accessory, the physical displacement of a component of the accessory relative to another component of the accessory, or the displacement of the component relative to the center of mass of the accessory, as detected by a user using their tactile senses. For example, when the accessory or a component of the accessory comes into contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the accessory or a component of the accessory. For example, movement of a component (e.g., the housing of stylus 203) may optionally be interpreted by the user as a "click" on a physically actuated button. In some cases, the user will feel a tactile sensation such as a "click" even when the physically actuated button associated with the stylus, which is physically pressed (e.g., displaced) by the user's movement, does not move. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when tactile output is described as corresponding to a specific sensory perception of a user (e.g., “click”), unless otherwise stated, the generated tactile output corresponds to the physical displacement of an electronic device or its components that would generate the sensory perception of a typical (or ordinary) user.

[0168] It should be understood that stylus 203 is merely an example of an electronic stylus, and stylus 203 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 4 The various components shown are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application-specific integrated circuits).

[0169] Memory 402 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more flash memory devices or other non-volatile solid-state memory devices. Access to memory 402 by other components of stylus 203 (such as one or more CPUs 420 and peripheral interface 418) is optionally controlled by memory controller 422.

[0170] Peripheral interface 418 can be used to couple the stylus's input and output peripherals to one or more CPUs 420 and memory 402. The one or more processors 420 run or execute various software programs and / or instruction sets stored in memory 402 to perform various functions of stylus 203 and process data.

[0171] In some embodiments, the peripheral interface 418, CPU 420, and memory controller 422 are optionally implemented on a single chip, such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0172] RF (Radio Frequency) circuit 408 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 408 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with electronic device 100 or electronic device 300, communication networks, and / or other communication devices via electromagnetic signals. RF circuit 408 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 408 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). The wireless communication may optionally use any of a variety of communication standards, protocols, and / or technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolved Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE... 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including those not yet developed as of the date of this document submission.

[0173] The I / O subsystem 406 couples input / output peripherals such as other input or control devices 416 on the stylus 203 to the peripheral interface 418. The I / O subsystem 406 optionally includes an optical sensor controller 458, an intensity sensor controller 459, a haptic feedback controller 461, and one or more input controllers 460 for the other input or control devices. The one or more input controllers 460 receive electrical signals from / send electrical signals to the other input or control devices 416. The other input or control devices 416 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, click wheels, etc. In some alternative embodiments, one or more input controllers 460 are optionally coupled to (or not coupled to) any of the following: an infrared port and / or a USB port.

[0174] The stylus 203 also includes a power system 462 for supplying power to various components. The power system 462 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in portable devices and / or portable accessories.

[0175] The stylus 203 may optionally also include one or more optical sensors 464. Figure 4 An optical sensor 464 coupled to an optical sensor controller 458 in an I / O subsystem 406 is shown. The optical sensor 464 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 464 receives light projected through one or more lenses from the environment and converts the light into data representing an image.

[0176] The stylus 203 may optionally also include one or more contact intensity sensors 465. Figure 4 A contact strength sensor coupled to a strength sensor controller 459 in I / O subsystem 406 is shown. One or more contact strength sensors 465 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a surface). The contact strength sensor 465 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to the tip of stylus 203.

[0177] The stylus 203 may optionally also include one or more proximity sensors 466. Figure 4 A proximity sensor 466 coupled to a peripheral device interface 418 is shown. Alternatively, the proximity sensor 466 is coupled to an input controller 460 in an I / O subsystem 406. In some embodiments, the proximity sensor determines the proximity of the stylus 203 to an electronic device (e.g., electronic device 100).

[0178] The stylus 203 may optionally also include one or more haptic output generators 463. Figure 4 A haptic output generator coupled to a haptic feedback controller 461 in I / O subsystem 406 is shown. The haptic output generator 463 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on electronic devices). One or more haptic output generators 463 receive haptic feedback generation instructions from haptic feedback module 433 and generate a haptic output on stylus 203 that can be felt by a user of stylus 203. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to the length (e.g., body or housing) of stylus 203, and optionally, the haptic output is generated by moving stylus 203 vertically (e.g., in a direction parallel to the length of stylus 203) or laterally (e.g., in a direction perpendicular to the length of stylus 203).

[0179] The stylus 203 may optionally also include one or more accelerometers 467, gyroscopes 468 and / or magnetometers 469 (e.g., as part of an inertial measurement unit (IMU)) for obtaining information about the position and position state of the stylus 203. Figure 4 Sensors 467, 468, and 469, coupled to peripheral interface 418, are shown. Alternatively, sensors 467, 468, and 469 may be coupled to input controller 460 in I / O subsystem 406. Stylus 203 may optionally include a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of stylus 203.

[0180] The stylus 203 includes a touch-sensitive system 432. The touch-sensitive system 432 detects input received at the touch-sensitive surface 275. These inputs include those discussed herein with respect to the touch-sensitive surface 275 of the stylus 203. For example, the touch-sensitive system 432 can detect tap, swivel, roll, flick, and swipe inputs. The touch-sensitive system 432 cooperates with a touch interpretation module 477 to interpret specific types of touch input received at the touch-sensitive surface 275 (e.g., swivel / roll / flick / swipe, etc.).

[0181] In some embodiments, the software components stored in memory 402 include an operating system 426, a communication module (or instruction set) 428, a contact / motion module (or instruction set) 430, a location module (or instruction set) 431, and a Global Positioning System (GPS) module (or instruction set) 435. Furthermore, in some embodiments, memory 402 stores device / global internal state 457, such as... Figure 4 As shown. Furthermore, although not shown, memory 402 includes a touch interpretation module 477. Device / global internal state 457 includes one or more of the following: sensor state, including information obtained from various sensors of the stylus and other input or control devices 416; position state, including information about the stylus's position relative to the electronic device (e.g., electronic device 100) (e.g., position, orientation, tilt, roll, and / or distance, such as...). Figure 5A and Figure 5B Information (as shown); and location information about the stylus position (e.g., determined by GPS module 435).

[0182] The operating system 426 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.) and facilitates communication between various hardware and software components.

[0183] Communication module 428 is optionally adapted to communicate with other devices via one or more external ports 424, and also includes various software components for processing data received by RF circuitry 408 and / or external ports 424. External ports 424 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., Internet, Wireless LAN, etc.). In some embodiments, the external port is associated with some devices from Apple Inc. (Cupertino, California). iPod and The device uses the same or similar and / or compatible Lightning connectors.

[0184] The contact / motion module 430 optionally detects contact with the stylus 203 and other touch-sensitive devices of the stylus 203 (e.g., buttons or other touch-sensitive components of the stylus 203). The contact / motion module 430 includes software components for performing various operations related to the detection of contact (e.g., detection of contact between the tip of the stylus and a touch-sensitive display such as the touchscreen 112 of the electronic device 100 or with another surface such as a table surface), such operations as determining whether a contact has occurred (e.g., detecting a touch press event), determining the intensity of the contact (e.g., the force or pressure of the contact or a substitute for force or pressure), determining whether there is movement of the contact and tracking across that movement (e.g., across the touchscreen 112 of the electronic device 100), and determining whether the contact has stopped (e.g., detecting a lift-off event or a contact interruption). In some embodiments, the contact / motion module 430 receives contact data from the I / O subsystem 406. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. As described above, in some embodiments, one or more of these operations related to the detection of contact are performed by an electronic device using contact / motion module 130 (contact / motion module 430 is used as a supplement to or in place of a stylus).

[0185] The contact / motion module 430 optionally detects gesture input from the stylus 203. Different gestures performed with the stylus 203 have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single tap gesture includes detecting a touch press event, followed by detecting a lift-off event at the same (or substantially the same) location as the touch press event (e.g., at the location of an icon). As another example, detecting a swipe gesture includes detecting a touch press event, followed by detecting one or more stylus drag events, and then detecting a lift-off event. As described above, in some embodiments, gesture detection is performed by the electronic device using the contact / motion module 130 (as a supplement to or alternative to the stylus).

[0186] In conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469, position module 431 optionally detects position information about the stylus, such as the stylus's attitude (roll, pitch, and / or yaw) in a particular frame of reference. In conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469, position module 431 optionally detects stylus movement gestures, such as flicking, tapping, and rotating the stylus. Position module 431 includes software components for performing various operations related to detecting the stylus's position and detecting changes in the stylus's position in a particular frame of reference. In some embodiments, position module 431 detects the stylus's positional state relative to the electronic device and detects changes in the stylus's positional state relative to the electronic device. As described above, in some embodiments, electronic device 100 or electronic device 300 uses position module 431 to determine the stylus's positional state relative to the electronic device and changes in that stylus's positional state (using position module 431 as a supplement to or alternative to the stylus).

[0187] The haptic feedback module 433 includes various software components for generating instructions used by one or more haptic output generators 463 to produce haptic output at one or more locations on the stylus 203 in response to user interaction with the stylus 203.

[0188] GPS module 435 determines the location of the stylus and provides that information for use in various applications (e.g., applications that provide location-based services, such as applications for finding lost devices and / or accessories).

[0189] Touch interpretation module 477 cooperates with touch-sensitive system 432 to determine (e.g., interpret or recognize) the type of touch input received at touch-sensitive surface 275 of stylus 203. For example, if a user swipes a sufficient distance on touch-sensitive surface 275 for a sufficiently short period of time, touch interpretation module 477 determines that the touch input corresponds to a swipe input (rather than a tap). Similarly, if the user swipes on touch-sensitive surface 275 fast enough to correspond to a voice input that is a swipe, touch interpretation module 477 determines that the touch input corresponds to a flick input (rather than a swipe). The threshold speed of the swipe can be preset and varied. In various embodiments, the pressure and / or force of the touch received at touch-sensitive surface determines the type of input. For example, a light touch may correspond to a first type of input, while a stronger touch may correspond to a second type of input.

[0190] Each module and application identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 402 optionally stores a subset of the modules and data structures described above. Furthermore, memory 402 optionally stores additional modules and data structures not described above.

[0191] Figures 5A to 5B The diagram illustrates the positional state of a stylus 203 relative to a touch-sensitive surface (e.g., the touchscreen 112 of the electronic device 100) according to some embodiments. In some embodiments, the positional state of the stylus 203 corresponds to (or indicates): the projected position of the stylus tip (or other representative portion) on the touch-sensitive surface (e.g., Figure 5A The (x,y) position 504), the orientation of the stylus relative to the touch-sensitive surface (e.g., Figure 5A Orientation 506), the tilt of the stylus relative to the touch-sensitive surface (e.g., Figure 5B The tilt (512) and / or the distance of the stylus relative to the touch-sensitive surface (e.g., Figure 5B (distance 514 in the distance). In some embodiments, the position state of the stylus 203 corresponds to (or indicates) the pitch, yaw, and / or rotation of the stylus (e.g., the attitude of the stylus relative to a particular reference frame such as a touch-sensitive surface (e.g., touchscreen 112) or the ground). In some embodiments, the position state includes a set of position parameters (e.g., one or more position parameters). In some embodiments, the position state is detected based on one or more measurements sent from the stylus 203 to an electronic device (e.g., electronic device 100). For example, the stylus measures the tilt of the stylus (e.g., Figure 5B Inclination 512) and / or orientation (e.g., Figure 5A The orientation 506 in the stylus is measured and the results are sent to the electronic device 100. In some embodiments, instead of or in combination with the position state detected based on one or more measurements from the stylus 203, the position state is detected based on the raw output from one or more electrodes in the stylus sensed by a touch-sensitive surface (e.g., the touchscreen 112 of the electronic device 100). For example, the touch-sensitive surface receives the raw output from one or more electrodes in the stylus and calculates the stylus tilt and / or orientation based on the raw output (optionally, in combination with position state information provided by the stylus based on sensor measurements generated by the stylus).

[0192] Figure 5AThe image shows a stylus 203 relative to a touch-sensitive surface, viewed from a perspective directly above the touch-sensitive surface (e.g., the touchscreen 112 of electronic device 100), according to some embodiments. Figure 5A In the figure, the z-axis 594 points out of the page (i.e., in the direction perpendicular to the plane of the touch screen 112), the x-axis 590 is parallel to the first edge (e.g., length) of the touch screen 112, the y-axis 592 is parallel to the second edge (e.g., width) of the touch screen 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0193] Figure 5A The tip of stylus 203 is shown at (x,y) position 504. In some embodiments, the tip of stylus 203 is the end of the stylus, configured to determine the proximity of the stylus to a touch-sensitive surface (e.g., touchscreen 112). In some embodiments, the projection of the stylus tip onto the touch-sensitive surface is an orthogonal projection. In other words, the projection of the stylus tip onto the touch-sensitive surface is at the end of a line perpendicular to the surface of the touch-sensitive surface from the stylus tip (e.g., (x,y) position 504, where the stylus tip would touch the touch-sensitive surface if the stylus were moved directly along a path perpendicular to the touch-sensitive surface). In some embodiments, the (x,y) position at the lower left corner of touchscreen 112 is position (0,0) (e.g., (0,0) position 502), and other (x,y) positions on touchscreen 112 are relative to the lower left corner of touchscreen 112. Alternatively, in some embodiments, the (0,0) position is located at another location on the touchscreen 112 (e.g., at the center of the touchscreen 112) and other (x,y) positions are relative to the (0,0) position on the touchscreen 112.

[0194] in addition, Figure 5A The stylus 203 is shown to have an orientation 506. In some embodiments, orientation 506 is the orientation of the projection of the stylus 203 onto the touchscreen 112 (e.g., an orthogonal projection of the length of the stylus 203 or a line corresponding to the line between the projections of two different points of the stylus 203 onto the touchscreen 112). In some embodiments, orientation 506 is relative to at least one axis in a plane parallel to the touchscreen 112. In some embodiments, orientation 506 is relative to a single axis in a plane parallel to the touchscreen 112 (e.g., axis 508, having a clockwise rotation angle varying from 0 degrees to 360 degrees starting from axis 508, such as...). Figure 5A (As shown). Alternatively, in some embodiments, orientation 506 is relative to a pair of axes in a plane parallel to touchscreen 112 (e.g., x-axis 590 and y-axis 592, as shown). Figure 5A As shown, or a pair of axes associated with the application displayed on the touchscreen 112.

[0195] In some embodiments, an indication (e.g., indication 516) is displayed on a touch-sensitive display (e.g., the touchscreen 112 of electronic device 100). In some embodiments, indication 516 indicates the location on the touch-sensitive display that the stylus will touch (or mark) before the stylus touches the touch-sensitive display. In some embodiments, indication 516 is part of an indication drawn on the touch-sensitive display. In some embodiments, indication 516 is separate from the indication drawn on the touch-sensitive display and corresponds to a virtual “pen tip” or other element indicating the location on the touch-sensitive display where an indication will be drawn.

[0196] In some embodiments, the indicator 516 is displayed based on the position state of the stylus 203. For example, in some cases, the indicator 516 is shifted from the (x,y) position 504 (e.g., Figure 5A and Figure 5B As shown), and in other cases, indicator 516 does not shift from (x,y) position 504 (e.g., when tilt 512 is zero degrees, indicator 516 is displayed at or near (x,y) position 504). In some embodiments, indicator 516 is displayed with varying colors, sizes (or radii or areas), opacities, and / or other characteristics depending on the stylus's position state. In some embodiments, the displayed indicator takes into account the thickness of the glass layer on the touch-sensitive display so that the indicator is applied to the "pixels" of the touch-sensitive display, rather than displayed on the "glass" covering the pixels.

[0197] Figure 5B The image shows a stylus 203 relative to a touch-sensitive surface (e.g., a touchscreen 112 of an electronic device 100) as viewed from a side view, according to some embodiments. Figure 5B In the figure, the z-axis 594 points in a direction perpendicular to the plane of the touch screen 112, the x-axis 590 is parallel to the first edge (e.g., length) of the touch screen 112, the y-axis 592 is parallel to the second edge (e.g., width) of the touch screen 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0198] Figure 5B The stylus 203 is shown to have a tilt 512. In some embodiments, the tilt 512 is an angle relative to the normal to the surface of the touch-sensitive surface (also simply referred to as the normal to the touch-sensitive surface) (e.g., normal 510). Figure 5B As shown, when the stylus is perpendicular to / perpendicular to the touch-sensitive surface (e.g., when stylus 203 is parallel to normal 510), the tilt 512 is zero, and the tilt increases as the stylus tilts closer to being parallel to the touch-sensitive surface.

[0199] in addition, Figure 5BThe distance 514 between the stylus 203 and the touch-sensitive surface is shown. In some embodiments, the distance 514 is the distance from the tip of the stylus 203 to the touch-sensitive surface in a direction perpendicular to the touch-sensitive surface. For example, in Figure 5B In the middle, distance 514 is the distance from the tip of stylus 203 to position 504 at (x,y).

[0200] Although this document uses the terms "x-axis," "y-axis," and "z-axis" to illustrate certain directions in specific figures, it should be understood that these terms do not refer to absolute directions. In other words, the "x-axis" can be any corresponding axis, and the "y-axis" can be a specific axis different from the x-axis. Typically, the x-axis is perpendicular to the y-axis. Similarly, the "z-axis" is different from the "x-axis" and "y-axis," and is typically perpendicular to both the "x-axis" and "y-axis."

[0201] in addition, Figure 5B The rotation 518 is shown—a rotation about the length (major axis) of the stylus 203.

[0202] Now turn attention to an embodiment of the user interface (“UI”) optionally implemented on the portable multifunction device 100.

[0203] Figure 6A An exemplary user interface for a menu for an application on a portable multifunction device 100, according to some embodiments, is shown. A similar user interface may optionally be implemented on an electronic device 300. In some embodiments, the user interface 600 includes the following elements or a subset or superset thereof:

[0204] • One or more signal strength indicators 602 for one or more wireless communications such as cellular signals and Wi-Fi signals;

[0205] • Time 604;

[0206] Bluetooth indicator 605;

[0207] • Battery status indicator 606;

[0208] • Tray 608 with icons for commonly used applications, such as:

[0209] The telephone module 138 has an icon 616 labeled "telephone", which optionally includes an indicator 614 indicating the number of missed calls or voicemails;

[0210] The email client module 140 has an icon 618 labeled "email", which optionally includes an indicator 610 for the number of unread emails;

[0211] The icon 620 labeled "Browser" in browser module 147; and

[0212] ο Video and music player module 152 (also known as (Apple Inc. trademark) Module 152) with icon 622 labeled "iPod"; and

[0213] • Icons of other applications, such as:

[0214] Icon 624 labeled "Message" in οIM module 141;

[0215] The icon 626 labeled "Calendar" in the calendar module 148;

[0216] The icon 628 labeled "Photo" in the image management module 144;

[0217] The icon 630 of the camera module 143 is labeled "camera";

[0218] The icon 632 labeled "Video Editing" in the online video module 155;

[0219] The icon labeled "Stock Market" in the Stock Market Desktop Mini Program 149-2 is 634.

[0220] The icon 636 labeled "map" in map module 154;

[0221] The icon labeled "Weather" in the Weather Desktop Mini Program 149-1 is 638.

[0222] The icon labeled "Clock" in the alarm clock desktop mini-program 149-4 is 640;

[0223] The icon 642 labeled "Fitness Support" in the Fitness Support module 142;

[0224] The icon 644 labeled "Notepad" in the Notepad module 153; and

[0225] An icon 646 is used to set up applications or modules, providing access to settings for the electronic device 100 and its various applications 136.

[0226] It should be pointed out that Figure 6AThe icon labels shown are merely exemplary. For example, in some embodiments, the icon 622 of the video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a corresponding application icon includes the name of the application corresponding to that application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0227] Figure 6B A touch-sensitive surface 651 separate from the display 650 is shown (e.g., Figure 3 Electronic devices (e.g., tablets or touchpads 355) Figure 3 An exemplary user interface on device 300. Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contact on the tactile surface 651 and / or one or more tactile output generators 357 for generating tactile outputs for the user of device 300.

[0228] Figure 6B A touch-sensitive surface 651 separate from the display 650 is shown (e.g., Figure 3 Electronic devices (e.g., tablets or touchpads 355) Figure 3 An exemplary user interface on device 300. Although many examples of input on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given in the following examples, in some embodiments, the electronic device 100 detects input on a touch-sensitive surface separate from the display, such as Figure 6B As shown. In some embodiments, the touch-sensitive surface (e.g., Figure 6B 651) has a main axis (e.g., on the display (e.g., 650) that is aligned with the main axis on the display (e.g., 651). Figure 6B The principal axis corresponding to 653 in the middle (e.g., Figure 6B (652 in the text). According to these embodiments, the electronic device 100 detects contact with the touch-sensitive surface 651 at a position corresponding to a corresponding position on the display (e.g., ...). Figure 6B (660 and 662 in the example) Figure 6B In this context, 660 corresponds to 668 and 662 corresponds to 670. Thus, when the touch-sensitive surface (e.g., Figure 6B 651 in the middle) and the display of a multi-functional device (e.g., Figure 6BWhen 650 is separated, user input detected by the electronic device on the touch-sensitive surface (e.g., touches 660 and 662 and their movement) is used by the electronic device to manipulate the user interface on the display. It should be understood that similar methods may be optionally used for other user interfaces described herein.

[0229] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture, etc.) and / or stylus input, it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the touch). Similarly, a tap gesture is optionally replaced by a mouse click when the cursor is above the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mouse and finger (or stylus) touch are optionally used simultaneously.

[0230] User interface and related processes

[0231] Now let’s turn our attention to electronic devices such as the portable multi-functional device 100 in Figure 1 or Figure 3 An embodiment of a user interface (“UI”) and associated processes implemented on an electronic device 300, the electronic device having a touch-sensitive display and optional one or more sensors for detecting signals from a stylus associated with the electronic device.

[0232] Figures 7A to 7Y Exemplary user interfaces for changing application state are shown according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 14A to 14C The process in the process. Although some examples in the following examples will refer to input on a touchscreen display (where a touch-sensitive surface and a display are combined, such as on touchscreen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 separate from display 650, such as Figure 6B As shown. In various embodiments, the electronic device 100 changes the application state based on data received from the stylus 203.

[0233] In various embodiments, the touch-sensitive surface of the stylus 203 (e.g., Figure 2 and Figures 5A to 5BThe touch-sensitive surface 275 in the stylus 203 detects touch input and gesture input, or the absence of touch input and gesture input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0234] In various embodiments, the stylus 203 uses orientation and / or motion sensors (e.g., accelerometers, magnetometers, gyroscopes) to detect orientation / motion input or the absence of orientation / motion input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, the pen barrel rotation and / or its direction, rotation and / or its direction, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0235] Figures 7A to 7C The diagram illustrates a sequence in which the electronic device 100 transitions from a first state to a second state based on the determination that the stylus 203 is being held by a user, and displays a visual indication associated with the second state. Figure 7A An electronic device 100 in a first state is shown, in which the stylus 203 is not held by the hand of a user 702. The stylus 203 includes a tip 276 and an end portion 277 opposite the tip 276. Figure 7A As shown, the electronic device 100 displays a navigation area 704, a canvas area 706, and a toolbar area 708. The navigation area 704, canvas area 706, and toolbar area 708 are associated with stylus-compatible applications such as drawing applications (e.g., Notepad or drawing applications).

[0236] like Figure 7B As shown, stylus 203 detects that it is being held by user 702's hand. This can occur when user 702's hand is holding stylus 203. In response to receiving data from stylus 203 indicating that it is being held by user 702's hand, electronic device 100 transitions from a first state to a second state. Figure 7C As shown, in the second state, the electronic device 100 stops displaying the navigation area 704, canvas area 706, and toolbar area 708. The electronic device 100 displays an enlarged canvas area 710 and a visual indicator 712 to indicate that the electronic device 100 is in the second state. The visual indicator 712 includes a marker icon 714 with a solid tip 716 to indicate that the stylus 203 will form a solid marker mark on the enlarged canvas area 710. Those skilled in the art will understand that the visual indicator 712 can take various forms.

[0237] Figures 7C to 7H Various operations performed in the second state based on the manipulation input received at stylus 203 are shown. As described above, Figure 7C An electronic device 100 in its second state is shown. (Example) Figure 7C As shown, the stylus 203 detects a downward swipe gesture 718. In response to receiving manipulation input data indicating a downward swipe gesture 718 from the stylus 203, in Figure 7D In this embodiment, the electronic device 100 displays a color palette 720 near a visual indicator 712. The color palette 720 includes four color indicators, each corresponding to a different color or pattern of a marker pen. Those skilled in the art will understand that the color indicators in the color palette 720 can include various styles and colors. Figure 7D A solid fill indicator 720a associated with a solid mark is shown to be currently selected (e.g., currently selected) within the color palette 720. For example, solid fill indicator 720a corresponds to the solid tip 716 of marker icon 714.

[0238] like Figure 7D As shown, stylus 203 detects a counter-clockwise (from above) rotational manipulation 722. Those skilled in the art will understand that stylus 203 can be rotated according to any number of angular manipulations. In response to receiving an instruction from stylus 203... Figure 7D The rotation control 722's control input data, in Figure 7E In the display, the electronic device 100 shows a diagonally filled indicator 720b with focus and a solid filled indicator 720a without focus. Furthermore, the electronic device 100 displays... Figure 7E The marker icon 714 in the middle has a corresponding diagonal striped tip 724, so as to indicate that the stylus 203 will form a diagonal striped mark on the enlarged canvas area 710.

[0239] like Figure 7E As shown, stylus 203 detects an upward swipe gesture 726. In response to receiving an instruction from stylus 203... Figure 7E The upward swipe gesture 726 in the middle of the operation input data, Figure 7F In this process, the electronic device 100 stops displaying the color palette 720. However, the electronic device 100 continues to display a visual indicator 712, including a marker icon 714 with diagonally striped tips 724.

[0240] like Figure 7F As shown, stylus 203 detects a tap gesture 728. In response to receiving an instruction from stylus 203... Figure 7F The tap gesture 728's operation input data, electronic device 100 update Figure 7GThe visual indicator 712 is replaced by a pencil icon 730. Figures 7C to 7F The marker icon 714 indicates that the stylus 203 will form a pencil mark on the enlarged canvas area 710.

[0241] like Figure 7G As shown, the stylus detects a subsequent tap gesture 732. This is in response to receiving an instruction from the stylus 203. Figure 7G The subsequent tap gesture 732's operation input data, electronic device 100 update Figure 7H The visual indicator 712 replaces the pencil icon 730 with a ruler icon 734. This indicates that the stylus 203 will use the pen as a ruler relative to the enlarged canvas area 710.

[0242] Figures 7H to 7J The sequence shown illustrates how an electronic device transitions from a second state to a first state based on the determination that the stylus 203 is no longer held by the user. Figure 7H An electronic device 100 in a second state is shown, in which the stylus 203 is held by the hand of a user 702. (See diagram below.) Figure 7H As shown, the electronic device 100 displays a visual indicator 712 including a ruler icon 734.

[0243] like Figure 7I As shown, stylus 203 detects that it is not being held by user 702's hand. This can occur when user 702 puts down stylus 203. In response to receiving data from stylus 203 indicating that it is not being held by user 702's hand, electronic device 100 transitions from a second state to a first state. Figure 7J As shown, in the first state, the electronic device 100 stops displaying the enlarged canvas area 710 and the visual indicator 712. Figure 7J In the middle, electronic devices display something similar to Figures 7A to 7B The navigation area 704, the canvas area 706, and the toolbar area 708.

[0244] Figures 7K to 7M Another sequence is shown whereby the electronic device transitions from a first state to a second state based on the determination that the stylus 203 is being held by the user, and displays a visual indication associated with the second state. Figure 7K An electronic device 100 is shown in a first state in which the stylus 203 is not held by the hand of the user 702. Figure 7K As shown, electronic device 100 displays lock screen 736.

[0245] like Figure 7LAs shown, stylus 203 detects that it is being held by user 702's hand. This can occur when user 702's hand is holding stylus 203. In response to receiving data from stylus 203 indicating that it is being held by user 702's hand, electronic device 100 transitions from a first state to a second state where electronic device 100 is not in a locked mode. Figure 7M As shown, in the second state, the electronic device 100 stops displaying the lock screen 736. The electronic device 100 displays something similar to... Figure 7C An enlarged canvas area 710 and a visual indicator 712. Although the visual indicator 712 corresponds to a marker icon 714 with a solid tip 716, those skilled in the art will understand that the visual indicator 712 can take various forms.

[0246] Figures 7M to 7O Another sequence is shown whereby the electronic device transitions from a second state to a first state and stops displaying visual indicators once it is determined that the stylus 203 is no longer being held by the user. Figure 7M An electronic device 100 in a second state is shown, in which the stylus 203 is held by the hand of a user 702. (See diagram below.) Figure 7M As shown, the electronic device 100 displays something similar to Figure 7C and Figure 7M An enlarged canvas area 710 and a visual indicator 712 associated with stylus-compatible applications such as drawing applications (e.g., notepad applications).

[0247] like Figure 7N As shown, the stylus 203 detects that it is not being held by the user 702's hand. In response to receiving data from the stylus 203 indicating that it is not being held by the user 702's hand, the electronic device 100 transitions from a second state to a first state where the electronic device 100 is in a locked mode. Figure 7O As shown, in the first state, the electronic device 100 stops displaying the enlarged canvas area 710 and the visual indicator 712. The electronic device 100 displays the lock screen 736.

[0248] Figures 7P to 7R This illustrates yet another sequence in which an electronic device transitions from a first state to a second state based on the determination that the stylus 203 is being held by a user, and displays visual indications associated with that second state. Figure 7P An electronic device 100 is shown in a first state in which the stylus 203 is not held by the hand of the user 702. Figure 7P As shown, electronic device 100 displays lock screen 736.

[0249] like Figure 7QAs shown, the stylus 203 detects that it is being held by the hand of user 702. In response to receiving data from the stylus 203 indicating that it is being held by the hand of user 702, in Figure 7R The electronic device 100 displays a prompt interface 738 superimposed on the lock screen 736. The prompt interface 738 includes a "yes" power indicator 740 and a "no" power indicator 742, allowing the user 702 to enter a drawing application or cancel the prompt interface 738 accordingly. The user can interact with the power indicators 740 and 742 via touch input pointing towards the touch-sensitive surface of the electronic device 100 at positions corresponding to 740 and 742. The following is related to... Figures 7R to 7S These interactions are described in further detail. In some embodiments, the electronic device 100 stops displaying the prompt interface 738 in response to receiving data from the stylus 203 indicating that the stylus 203 is no longer held by the user 702's hand. In some embodiments, the electronic device 100 continues to display the prompt interface 738 in response to receiving data from the stylus 203 indicating that the stylus 203 is no longer held by the user 702's hand.

[0250] Figures 7R to 7S This illustrates the shift from the lock screen to a restricted user interface associated with a drawing application. (For example...) Figure 7R As shown, the electronic device 100 displays a prompt interface 738 superimposed on the lock screen 736. In response to detecting a corresponding... Figure 7R The "Yes" indicator in the code signifies touch input to 740. The electronic device 100 then stops displaying the lock screen 736 and the prompt interface 738, and subsequently displays a restricted user interface 744 (e.g., associated with a drawing application) and visual indicators 712, such as... Figure 7S As shown. In response to detecting a touch input corresponding to the "No" enable indicator 742, the electronic device 100 stops displaying the prompt interface 738 and continues to display the lock screen 736 (not shown).

[0251] Figures 7S to 7U Another sequence is shown whereby the electronic device transitions from a second state to a first state and stops displaying visual instructions once it is determined that the stylus 203 is no longer being held by the user. Figure 7S An electronic device 100 in a second state is shown, in which the stylus 203 is held by the hand of a user 702.

[0252] like Figure 7T As shown, the stylus 203 detects that it is not being held by the user 702's hand. In response to receiving data from the stylus 203 indicating that it is not being held by the user 702's hand, the electronic device 100 transitions from a second state to a first state. As... Figure 7U As shown, in the first state, the electronic device 100 stops displaying the visual indicator 712 and the restricted user interface 744. Then, as... Figure 7U As shown, the electronic device 100 displays something similar to Figure 7J The navigation area 704, canvas area 706, and toolbar area 708 in the first state.

[0253] Figures 7V to 7X This illustrates yet another sequence in which an electronic device transitions from a first state to a second state based on the determination that the stylus 203 is being held by a user, and displays visual indications associated with that second state. Figure 7V An electronic device 100 is shown in a first state in which the stylus 203 is not held by the hand of the user 702. Figure 7V As shown, electronic device 100 displays a home screen 746. The home screen 746 includes a matrix of application icons (e.g., applications) arranged in the main area 748 of the display. The home screen 746 includes a taskbar 750 comprising a row of taskbar icons. Those skilled in the art will understand that the number and arrangement of application icons and / or taskbar icons may vary.

[0254] like Figure 7W As shown, the stylus 203 detects that it is being held by the hand of user 702. In response to receiving data from the stylus 203 indicating that it is being held by the hand of user 702, the electronic device 100 transitions from a first state to a second state. As... Figure 7X As shown, in the second state, the electronic device 100 displays a prompt interface 752 superimposed on the home screen 746. The prompt interface 752 includes a "yes" power indicator 754 and a "no" power indicator 756, allowing the user to enter the drawing application or cancel the prompt interface 752 accordingly. The user can interact with the power indicators 754 and 756 via touch input pointing towards the touch-sensitive surface of the electronic device 100 at positions corresponding to the power indicators 754 and 756.

[0255] Figures 7X to 7Y This illustrates the transition from the home screen to the user interface associated with the drawing application. (For example...) Figure 7X As shown, the electronic device 100 displays a prompt interface 752 superimposed on the home screen 746. In response to detecting a corresponding... Figure 7X The "Yes" indicator in the code signifies touch input 754. The electronic device 100 stops displaying the home screen 746 and the prompt interface 752, and subsequently displays a restricted user interface 744 (e.g., associated with a drawing application) and a visual indicator 712, such as... Figure 7Y As shown. In response to detecting a touch input corresponding to the "No" indication 756, the electronic device 100 stops displaying the prompt interface 752 and continues to display the home screen 746 (not shown).

[0256] Figures 8A to 8H Exemplary user interfaces for changing the functionality of stylus 203 are shown according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including… Figures 15A to 15B The process in the process. Although some examples in the following examples will refer to input on a touchscreen display (where a touch-sensitive surface and a display are combined, such as on touchscreen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 separate from display 650, such as Figure 6B As shown. In various embodiments, the electronic device 100 changes the function of the stylus 203 based on data received from the stylus 203.

[0257] In various embodiments, the touch-sensitive surface of the stylus 203 (e.g., Figure 2 and Figures 5A to 5B The touch-sensitive surface 275 in the stylus 203 detects touch input and gesture input, or the absence of touch input and gesture input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0258] In various embodiments, the stylus 203 uses orientation and / or motion sensors (e.g., accelerometers, magnetometers, gyroscopes) to detect orientation / motion input or the absence of orientation / motion input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, the pen barrel rotation and / or its direction, rotation and / or its direction, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0259] Figures 8A to 8B This illustrates a first sequence in which the displayed content is modified based on the determination that the stylus is being held in a first gripping arrangement. For example... Figure 8A As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application, including content 804 (e.g., a gray rectangle). Figure 8A In this scenario, when a user holds the stylus 203 in his / her hand 802 in a first grip arrangement 815, the electronic device 100 detects input 810 (e.g., drawing strokes or markings) from the stylus 203. The first grip arrangement 815 corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing towards the electronic device 100), wherein the fingers of the hand 802 are close to the tip 276 of the stylus 203.

[0260] In response to the detection that the stylus 203 is being held in the first gripping arrangement 815, Figures 8A to 8B In this context, the electronic device 100 displays an indicator 812 associated with the first marking tool (e.g., a felt-tip marker) within a user interface 800. For example... Figure 8B As shown, electronic device 100 is based on Figure 8A The input 810 and the first marking tool (e.g., felt-tip marker) associated with the first grip arrangement 815 are used to display a first change 820 (e.g., a stroke or mark) to the user interface 800.

[0261] Figures 8C to 8D This illustrates a second sequence in which the displayed content is altered based on the determination that the stylus is being held in a second grip arrangement. For example... Figure 8C As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application, including content 804 (e.g., a gray rectangle). Figure 8C In this scenario, when a user holds the stylus 203 in his / her hand 802 in a second grip arrangement 835, the electronic device 100 detects input 810 (e.g., drawing strokes or markings) from the stylus 203. The second grip arrangement 835 corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing towards the electronic device 100), wherein the fingers of the hand 802 are close to the end 277 of the stylus 203 opposite the tip 276 of the stylus 203.

[0262] In response to the detection that the stylus 203 is being held in the second gripping arrangement 835, Figures 8C to 8D In this context, the electronic device 100 displays an indicator 832 associated with a second marking tool (e.g., a watercolor brush) within a user interface 800. Figure 8D As shown, electronic device 100 is based on Figure 8C The input 810 and the second marking tool (e.g., a watercolor brush) associated with the second grip arrangement 835 are used to display a second change 840 (e.g., a stroke or mark) to the user interface 800.

[0263] Figures 8E to 8F This illustrates a third sequence in which the displayed content is modified based on the determination that the stylus is being held in a third grip arrangement. For example... Figure 8E As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application, including content 804 (e.g., a gray rectangle). Figure 8EIn this process, when the user is holding the stylus 203 in his / her hand 802 in a third grip arrangement 855, the electronic device 100 detects input 810 (e.g., drawing strokes or markings) from the stylus 203. The third grip arrangement 855 corresponds to holding the stylus 203 in an inverted orientation (e.g., the tip 276 of the stylus 203 points outward from the electronic device 100) and close to the end 277 of the stylus 203 opposite to the tip 276 of the stylus 203.

[0264] In response to the detection that the stylus 203 is being held in the third grip arrangement 855, Figures 8E to 8F In this context, electronic device 100 displays an indicator 852 associated with a third marking tool (e.g., an eraser) within user interface 800. For example... Figure 8F As shown, electronic device 100 is based on Figure 8E The input 810 and the third marking tool (e.g., eraser) associated with the third grip arrangement 855 are used to display a third change 860 (e.g., stroke or mark) to the user interface 800.

[0265] Figures 8G to 8H This illustrates a fourth sequence that makes a fourth change to the displayed content based on determining that the stylus is being held in a fourth grip arrangement. For example... Figure 8G As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application, including content 804 (e.g., a gray rectangle). Figure 8G In this scenario, when the user is holding the stylus 203 in his / her hand 802 in a fourth grip arrangement 875, the electronic device 100 detects input 810 (e.g., drawing strokes or markings) from the stylus 203. The fourth grip arrangement 875 corresponds to holding the stylus 203 in an inverted orientation (e.g., with the tip 276 of the stylus 203 pointing outwards from the electronic device 100) and close to the tip 276 of the stylus 203.

[0266] In response to the detection that the stylus 203 is being held in the fourth gripping arrangement 875, Figures 8G to 8H In this context, electronic device 100 displays an indicator 872 associated with a fourth marking tool (e.g., a spray can) within user interface 800. For example... Figure 8H As shown, electronic device 100 is based on Figure 8G The input 810 and the fourth marking tool (e.g., inkjet can) associated with the fourth grip arrangement 875 are used to display a fourth change 880 (e.g., stroke or mark) to the user interface 800.

[0267] Those skilled in the art will understand, refer to Figures 8A to 8HThe specific description of the gripping arrangement to the identifier type in the described sequence is arbitrary and variable. Those skilled in the art will understand that, although in reference... Figures 8A to 8H The same input 810 is shown in the sequence, but in various other embodiments, other input vectors may be detected when the stylus is held in a gripping arrangement.

[0268] Figures 9A to 9P Exemplary user interfaces for modifying stylus input functionality are illustrated according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including… Figures 16A to 16B The process in the process. Although some examples in the following examples will refer to input on a touchscreen display (where a touch-sensitive surface and a display are combined, such as on touchscreen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 separate from display 650, such as Figure 6B As shown. In various embodiments, the electronic device 100 modifies the touch input function based on data received from the stylus 203.

[0269] In various embodiments, the touch-sensitive surface of the stylus 203 (e.g., Figure 2 and Figures 5A to 5B The touch-sensitive surface 275 in the stylus 203 detects touch input and gesture input, or the absence of touch input and gesture input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0270] In various embodiments, the stylus 203 uses orientation and / or motion sensors (e.g., accelerometers, magnetometers, gyroscopes) to detect orientation / motion input or the absence of orientation / motion input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, the pen barrel rotation and / or its direction, rotation and / or its direction, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0271] Figures 9A to 9C An example is shown where the first operation is performed based on determining that the stylus is being held. For example... Figure 9AAs shown, electronic device 100 displays a user interface 900 associated with a drawing or notepad application, including content 904 (e.g., an icon) and a visual indicator 906 indicating that a stylus 203 is being held by the hand of user 902. The visual indicator 906 corresponds to a solid-tipped marker icon, such that the stylus 203 will form a solid-tipped marker icon on the user interface 900. Those skilled in the art will understand that content 904 and / or visual indicator 906 can take various forms.

[0272] like Figure 9A As shown, electronic device 100 detects a left swipe gesture 908 on its touch-sensitive surface. In response to detecting the left swipe gesture 908 and determining, based on data received from stylus 203, that stylus 203 is being held by user 902's hand, electronic device 100 performs an erase or undo operation relative to content 904. Therefore, electronic device 100 stops displaying content 904 (e.g., ...) on user interface 900. Figure 9B (as shown) and keep displaying visual indicator 906.

[0273] like Figure 9B As shown, electronic device 100 detects a right swipe gesture 910 on its touch-sensitive surface. In response to detecting the right swipe gesture 910 and determining, based on data received from stylus 203, that stylus 203 is being held by user 902's hand, electronic device 100 performs a restore operation relative to content 904. Therefore, electronic device 100 redisplays content 904 on user interface 900 (e.g., ...). Figure 9C (as shown) and keep displaying visual indicator 906.

[0274] Figures 9D to 9E An example is shown of performing a second operation based on determining that the stylus is not being held. Electronic device 100 in Figure 9D Content 904 is displayed on the user interface 900. The electronic device 100 determines that the stylus 203 is not being held by the user 902's hand based on data received from the stylus 203 and / or based on the lack (e.g., absence) of data received from the stylus 203. Therefore, as... Figures 9D to 9E As shown, electronic device 100 does not display. Figures 9A to 9C The visual indicator 906 is shown.

[0275] like Figure 9D As shown, electronic device 100 detects a left swipe gesture 908 (e.g., a stroke) on the touch-sensitive surface of electronic device 100. In response to detecting a left swipe gesture 908 (e.g., similar to...), Figure 9A The electronic device 100 performs a left swipe gesture (908) and, based on the determination that the stylus 203 is not held by the user's hand (902), the electronic device 100... Figure 9EDrawing operations are performed on the user interface 900. That is, as... Figure 9E As shown, the electronic device 100 displays the corresponding Figure 9D The left swipe gesture 908 is identified by icon 914, and the displayed content 904 is maintained. This drawing operation differs from the erase / undo operation performed in response to the left swipe gesture 908 made while the stylus 203 is held, which is in Figures 7A to 7B Shown in.

[0276] Figures 9F to 9H Another example is shown where the first operation is performed based on the determination that the stylus is held. For example... Figure 9F As shown, the electronic device 100 displays a user interface 900 that includes content 904 (e.g., logo) and a visual indicator 906 indicating that a stylus 203 is being held by the hand of a user 902.

[0277] like Figure 9F As shown, electronic device 100 detects a looping gesture 916 (e.g., a lasso gesture) on its touch-sensitive surface. The looping gesture 916 corresponds to enclosing (e.g., substantially enclosing) content 904. In other words, the looping gesture 916 corresponds to surrounding and / or encircling displayed content, including a portion of the displayed content (e.g., the upper half of a circle, a segment of a line, a small piece of an image, half of a stanza, etc.). Those skilled in the art will understand that the looping gesture 916 may include various lines (e.g., regular polygonal lines, irregular polygonal lines, circular lines, oval lines, lines with various curvatures, or combinations thereof) and / or may enclose multiple or one types of displayed content (e.g., lines, polygons, logos, images, text, etc.).

[0278] In response to detection Figure 9F The electronic device 100 changes the content 904 surrounded by the circular gesture 916 and, based on data received from the stylus 203, determines that the stylus 203 is being held by the user 902's hand. This change is done to indicate that the content 904 is currently being held. Figure 9G It is selected from among them. That is, such as Figures 9F to 9G As shown, electronic device 100 changes content 904 from a solid line marker to content 920 marked with a dashed line. Those skilled in the art will understand that electronic device 100 can change content 904 in various ways to indicate the detection of a looping gesture 916.

[0279] like Figure 9G As shown, the electronic device 100 detects a drag gesture 922 including a start point 924 and an end point 926. In response to detecting the drag gesture 922 and determining, based on data received from the stylus 203, that the stylus 203 is being held by the user 902's hand, the electronic device 100 moves content 920 according to the drag gesture 922, such as... Figure 9H As shown. That is, as Figure 9H As shown, the electronic device 100 moves (e.g., changes its display position) the content 920 to the endpoint 926 of the drag gesture 922 and restores the display of the content 904 to the solid line marker.

[0280] Figures 9I to 9J Another example is shown of performing a second operation based on determining that the stylus is not being held. Electronic device 100 determines that the stylus 203 is not being held by the user 902's hand based on data received from the stylus 203 and / or based on the lack (e.g., absence) of data received from the stylus 203. Figure 9I As shown, in response to determining that the stylus 203 is not being held by the user 902's hand, the electronic device 100 does not display anything. Figures 9F to 9H The visual indicator 906 is shown. Electronic device 100 in... Figure 9I The user interface 900 displays a navigation area 928, a canvas area 930, and a toolbar area 932. The navigation area 928, the canvas area 930, and the toolbar area 932 are associated with stylus-compatible applications such as drawing applications (e.g., Notepad or drawing applications).

[0281] like Figure 9I As shown, electronic device 100 detects a looping gesture 916 surrounding content 904 (e.g., similar to...). Figure 9F (Circular gesture 916 in the image). However, because the stylus 203 is not held by the user 902's hand, the electronic device 100 performs a gesture relative to... Figures 9F to 9H The second operation differs from the first operation. That is, as... Figure 9J As shown, in response to detecting a looping gesture 916 and determining that the stylus 203 is not being held by the user 902's hand based on data received from the stylus 203 and / or based on the lack of such data, the electronic device 100 displays an identifier 934 corresponding to the looping gesture 916.

[0282] Figures 9K to 9M Another example is shown where the first operation is performed based on determining that the stylus is being held. For example... Figure 9K As shown, the electronic device 100 displays a user interface 900 that includes text 936 and a visual indicator 906 indicating that the stylus 203 is being held by the hand of the user 902.

[0283] like Figure 9K As further shown, the electronic device 100 detects a rightward swipe gesture 938 on its touch-sensitive surface. In response to the detection... Figure 9K The electronic device 100 selects a portion of text 936, such as a right swipe gesture 938, and determines, based on data received from the stylus 203, that the stylus 203 is being held by the user 902's hand. Figure 9LAs shown. That is, as Figure 9L As shown, the electronic device displays the selected text 940 with a selection indicator 941 that indicates the selection.

[0284] like Figure 9L As further shown, the electronic device 100 detects a dragging gesture 942 including a start point 944 and an end point 946. In response to the detection... Figure 9L The electronic device 100 moves the selected text 940 according to the drag gesture 942 and, based on data received from the stylus 203, determines that the stylus 203 is being held by the user 902's hand. Figure 9M As shown. That is, as Figure 9M As shown, the electronic device 100 moves (e.g., changes its display position) the selected text 940 to the endpoint 946 of the drag gesture 942. Therefore, as... Figure 9M As shown, the electronic device 100 displays modified text 948 corresponding to text 936, which does not have the moved selected text 940.

[0285] Figures 9N to 9P Another example is shown where a second operation is performed based on the determination that the stylus is not being held. For example... Figure 9N As shown, the electronic device 100 displays a user interface 900 including text 936. The electronic device 100 displays a navigation area 928, a canvas area 930, and a toolbar area 932 on the user interface 900. The navigation area 928, canvas area 930, and toolbar area 932 are associated with stylus-compatible applications such as drawing applications (e.g., Notepad or drawing applications). In response to data received from the stylus 203 indicating that the stylus 203 is not held by the user 902's hand and / or the lack of such data, [the following is unclear and likely incomplete] Figures 9K to 9M Electronic devices 100 Figure 9N Visual indicator 906 is not displayed.

[0286] like Figure 9N As further shown, the electronic device 100 detects a right swipe gesture 938 on its touch-sensitive surface. In response to detecting the right swipe gesture 938 and determining, based on data received from the stylus 203 and / or the absence of such data, that the stylus 203 is not being held by the user 902's hand, the electronic device 100 highlights a portion of text 936, such as... Figure 9O As shown. That is, as Figure 9O As shown, the electronic device 100 displays highlighted text 950 with a highlight indicator 952 indicating highlighting. This highlighting operation differs from that when the stylus 203 is held by the user 902's hand. Figures 9K to 9L The selection operation that occurred.

[0287] like Figure 9OAs further shown, the electronic device 100 detects a dragging gesture 942 including a start point 944 and an end point 946. In response to the detection... Figure 9O The electronic device 100 performs a drag gesture 942 and determines, based on data received from the stylus 203 and / or the absence of such data, that the stylus 203 is not being held by the user 902's hand. Figure 9P The display shows an identifier 954 corresponding to the drag gesture 942. This identifier indicates an operation different from that when the stylus 203 is held by the user 902's hand. Figures 9L to 9M The movement operation that occurred. For example... Figure 9P As further shown, the electronic device 100 maintains the display text 936, the highlighted text 950, and the highlighted indicator 952.

[0288] Figures 10A to 10I Exemplary user interfaces for performing operations on existing identifiers based on finger manipulation input are illustrated according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 17A to 17C The process in the process. Although some examples in the following examples will refer to input on a touchscreen display (where a touch-sensitive surface and a display are combined, such as on touchscreen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 separate from display 650, such as Figure 6B As shown. In various embodiments, the electronic device 100 performs operations on existing identifiers based on data received from the stylus 203.

[0289] In various embodiments, the touch-sensitive surface of the stylus 203 (e.g., Figure 2 and Figures 5A to 5B The touch-sensitive surface 275 in the stylus 203 detects touch input and gesture input, or the absence of touch input and gesture input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0290] In various embodiments, the stylus 203 uses orientation and / or motion sensors (e.g., accelerometers, magnetometers, gyroscopes) to detect orientation / motion input or the absence of orientation / motion input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, the pen barrel rotation and / or its direction, rotation and / or its direction, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0291] Figures 10A to 10B This illustrates a sequence of selecting user interface elements within the user interface. For example... Figure 10A As shown, the electronic device 100 displays a user interface 1000 associated with a drawing or notepad application, which includes pre-existing content: a star 1004a and a lightning bolt 1004b. Figure 10A In the process, when the user is holding the stylus 203 in his / her hand 1002 with one or more fingers 202 around the stylus 203 while making a fist, the electronic device 100 detects input 1010 from one or more fingers 202 in a generally circular shape (e.g., drawing strokes or symbols) around the lightning bolt 1004b.

[0292] In response to the detection of selection Figure 10A The input 1010 of the Lightning 1004b, and the electronic device 100 are... Figure 10B The selected state, indicated by a dashed outline, displays lightning bolt 1004b' to indicate that lightning bolt 1004b' is currently selected. Figure 10B In the middle, star 1004a is shown with a solid outline corresponding to star 1004a not selected by the user.

[0293] Figures 10B to 10C This illustrates a sequence in which a first operation (e.g., enlarging) is performed on a user interface element based on a first finger manipulation input (e.g., counter-clockwise scrolling of the stylus) determined from finger manipulation data received from the stylus. Figure 10B In the image, electronic device 100 displays lightning bolt 1004b' at a first size 1015a. (As shown...) Figure 10B As shown, when the user is holding the stylus 203 in his / her hand 1002 and rolling the stylus 203 counterclockwise, the stylus 203 detects input 1020a (e.g., the counterclockwise rolling of the stylus 203).

[0294] In response to receiving instructions Figure 10B The input of finger manipulation data in 1020a, in Figure 10C In the process, the electronic device 100 displays lightning bolt 1004b' increasing in size from the first size 1015a to the second size 1015b within the user interface 1000, and lightning bolt 1004c' increasing in size from the first size 1015a.

[0295] Figures 10C to 10D This illustrates a sequence in which a first operation (e.g., enlarging) is performed again on a user interface element based on a first finger manipulation input (e.g., counter-clockwise scrolling of the stylus) determined from finger manipulation data from the stylus. Figure 10CAs shown, when the user is holding the stylus 203 in his / her hand 1002 and scrolling the stylus 203 counterclockwise, the stylus 203 detects input 1020b (e.g., counterclockwise scrolling of the stylus 203). In response to receiving the instruction... Figure 10C Input 1020b finger manipulation data, in Figure 10D In the process, the electronic device 100 displays lightning bolt 1004c' within the user interface 1000, which further increases in size from the second size 1015b to the third size 1015c, displaying lightning bolt 1004d'.

[0296] Figures 10D to 10E This illustrates a sequence in which a second operation (e.g., resizing) is performed on a user interface element based on a second finger manipulation input (e.g., clockwise scrolling of the stylus) determined from finger manipulation data received from the stylus. Figure 10D As shown, when the user is holding the stylus 203 in his / her hand 1002 and rolling the stylus 203 clockwise, the stylus 203 detects input 1020c (e.g., clockwise rolling of the stylus 203). In response to receiving the instruction... Figure 10D Input 1020c finger manipulation data, in Figure 10E In the process, the size of the lightning bolt 1004d' displayed by the electronic device 100 within the user interface 1000 is further reduced, from the third size 1015c to the lightning bolt 1004e' under the fourth size 1015d.

[0297] Figures 10E to 10F This illustrates a sequence in which a second operation (e.g., resizing) is performed on a user interface element based on a second finger manipulation input (e.g., clockwise scrolling of the stylus) determined from finger manipulation data received from the stylus. Figure 10E As shown, when the user is holding the stylus 203 in his / her hand 1002 and rolling the stylus 203 clockwise, the stylus 203 detects input 1020d (e.g., clockwise rolling of the stylus 203). In response to receiving the instruction... Figure 10E Input 1020d finger manipulation data in Figure 10F In the process, the size of the lightning bolt 1004e' displayed by the electronic device 100 within the user interface 1000 is further reduced, from the fourth size 1015d to the fifth size 1015e of the lightning bolt 1004e'.

[0298] Figures 10G to 10H Another sequence is shown in which a first operation (e.g., a cut operation) is performed on a user interface element based on a third finger manipulation input (e.g., an upward swipe on the stylus) determined from finger manipulation data from the stylus. Figure 10GAs shown, the electronic device 100 displays a user interface 1000 associated with a drawing or notepad application, which includes pre-existing content: a triangle 1004d. (As shown...) Figure 10G As shown, when the user is holding the stylus 203 in his / her hand 1002, the stylus 203 detects input 1040 at the position of the stylus 203 relative to the electronic device 100 (e.g., an upward swipe on the stylus 203), instructing the user to select a clipping triangle 1004d from the user interface 1000.

[0299] In response to receiving instructions Figure 10G Input 1040 finger manipulation data, in Figure 10H In this process, electronic device 100 performs a first operation (e.g., a cut operation) on triangle 1004d within user interface 1000. In some embodiments, the first operation corresponds to a copy operation. Figure 10H As shown, in response to detecting an upward swipe on the stylus 203 corresponding to the user cutting (or, in some embodiments, copying) triangle 1004d, the electronic device 100 no longer displays triangle 1004d on the user interface 1000.

[0300] Figures 10H to 10I This illustrates a sequence in which a second operation (e.g., a paste operation) is performed on a user interface element based on a fourth finger manipulation input (e.g., a downward swipe gesture on the stylus) determined from finger manipulation data received from the stylus. Figure 10H As shown, when the user is holding the stylus 203 in his / her hand 1002, the stylus 203 detects input 1050 (e.g., a downward swipe on the stylus 203) at the position of the stylus 203 relative to the electronic device 100.

[0301] In response to receiving instructions Figure 10H Input 1050 finger manipulation data in Figure 10I In this process, the electronic device 100 performs a second operation (e.g., a paste operation) on the triangle 1004d within the user interface 1000. For example... Figure 10I As shown, in response to detecting a downward swipe on the stylus 203 corresponding to the user pasting triangle 1004d onto the user interface 1000, the electronic device displays triangle 1004d on the user interface 1000 at the position of the stylus 203 relative to the electronic device 100.

[0302] Figures 11A to 11O Exemplary user interfaces for performing finger manipulations on a stylus 203 to navigate within menus, according to some embodiments, are shown. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 18A to 18BThe process in the process. Although some examples in the following examples will refer to input on a touchscreen display (where a touch-sensitive surface and a display are combined, such as on touchscreen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 separate from display 650, such as Figure 6B As shown. In various embodiments, the electronic device 100 navigates within a menu based on data received from the stylus 203.

[0303] In various embodiments, the touch-sensitive surface of the stylus 203 (e.g., Figure 2 and Figures 5A to 5B The touch-sensitive surface 275 in the stylus 203 detects touch input and gesture input, or the absence of touch input and gesture input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0304] In various embodiments, the stylus 203 uses orientation and / or motion sensors (e.g., accelerometers, magnetometers, gyroscopes) to detect orientation / motion input or the absence of orientation / motion input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, the pen barrel rotation and / or its direction, rotation and / or its direction, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0305] Figures 11A to 11B This shows a first sequence that makes the first change to the displayed content. For example... Figure 11A As shown, the electronic device 100 displays a user interface 1100 associated with a drawing or notepad application. Figure 11A In the process, when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1110 from the stylus 203 (e.g., drawing strokes or markings). In response to the electronic device 100 detecting input 1110, Figure 11B In the middle, electronic device 100 is based on Figure 11A Input 1110 in the middle is used to display the first change 1106 to the user interface 1100 (e.g., a stroke or logo) to display the user interface element 1104.

[0306] Figures 11C to 11DThis illustrates another sequence in which a first action (e.g., opening a menu) is performed on a user interface element based on a first finger manipulation input (e.g., an upward swipe on the stylus) determined from finger manipulation data from the stylus. Figure 11C As shown, when the user is holding the stylus 203 in his / her hand 1102, the stylus 203 detects input 1120a (e.g., an upward swipe on the stylus 203) at the position of the stylus 203 relative to the electronic device 100.

[0307] In response to receiving instructions Figure 11C The input of finger manipulation data in 1120a, in Figure 11D In this configuration, electronic device 100 displays menu 1114 on user interface 1100. Menu 1114 includes four visual indicators: a solid indicator 1114a, a striped indicator 1114b, a dotted indicator 1114c, and a blank indicator 1114d, where the solid indicator 1114a has focus by default (as shown by focus indicator 1114i). Figure 11D As shown, menu 1114 is a radial menu with four visual indicators arranged in a circle. Additionally, the focus indicator 1114i corresponds to a star or other icon near the selected item with focus, a ring around the selected item with focus, magnifying the selected item in the focus area, changing the color or appearance of the selected item with focus, etc. Those skilled in the art will understand that menu 1114 can include any number of visual indicator types with various characteristics, wherein any one of these visual indicators has focus by default.

[0308] Figures 11D to 11E Another sequence is shown of performing a second operation based on finger manipulation data determined from the stylus, instructing a second finger manipulation input on the stylus (e.g., clockwise rotation of the stylus). In some embodiments, the electronic device 100 may change which indicator is in focus in response to a user's hand 1102 manipulating the stylus 203. For example, in response to receiving finger manipulation data from the stylus 203 instructing a clockwise rotation 1130a of the stylus 203, the electronic device 100 moves clockwise through (e.g., changes the display) menu 1114, causing the focus to change from a solid indicator 1114a to a striped indicator 1114b.

[0309] Figures 11E to 11FThe diagram illustrates a sequence in which a second operation is performed again based on finger manipulation data determined from the stylus, instructing a second finger manipulation input on the stylus (e.g., clockwise rotation of the stylus). For example, in response to receiving finger manipulation data from the stylus 203 instructing a clockwise rotation 1130b of the stylus 203, the electronic device 100 further moves clockwise through (e.g., changes the display) menu 1114, causing the focus to change from the striped indicator 1114b to the dot indicator 1114c.

[0310] Figures 11F to 11G Another sequence of operations is shown, instructing a third finger manipulation input (e.g., counter-clockwise rotation of the stylus) based on finger manipulation data determined from the stylus. For example, in response to receiving finger manipulation data from the stylus 203 instructing a counter-clockwise rotation 1130c of the stylus 203, the electronic device 100 moves counter-clockwise through (e.g., changes the display) menu 1114, causing the focus to shift from the dot indicator 1114c back to the stripe indicator 1114b.

[0311] Figures 11G to 11H Another sequence is shown in which an action (e.g., a selection action) is performed on a user interface element based on a finger manipulation data determined from the stylus indicating a manipulation input (e.g., a tap on the stylus). Figure 11G As shown, when the user is holding the stylus 203 in his / her hand 1102, the stylus 203 detects input 1140a (e.g., a tap on the stylus 203) at the position of the stylus 203 relative to the electronic device 100. In response to receiving an instruction... Figure 11G The input of finger manipulation data in the 1140a is in Figure 11H In this process, electronic device 100 removes menu 1114 from user interface 1110 on display. Additionally, Figure 11G The indicator 1112a associated with the first marking tool (e.g., a felt-tip marker) that is a solid line is in Figure 11H The middle part is transformed into an indicator 1112b associated with the first marking tool, which is striped.

[0312] Figures 11H to 11I This illustrates another sequence that makes a second change to the displayed content. Figure 11H In the process, when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1150 (e.g., drawing strokes or markings) from the stylus 203. In response to the electronic device 100 detecting input 1150, Figure 11I In the middle, electronic device 100 is based on Figure 11H Input 1150 is used to display a second change 1116 (e.g., a stroke or symbol) to the user interface 1100 to display user interface element 1124. For example... Figure 11IAs shown, user interface element 1124 is a stripe line corresponding to tool 1112b.

[0313] Figures 11J to 11K This illustrates another sequence that makes a third change to the displayed content. Figure 11J In the process, when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1160 (e.g., drawing strokes or markings) from the stylus 203. In response to the electronic device 100 detecting input 1160, Figure 11K In the middle, electronic device 100 is based on Figure 11J Input 1160 is used to display a third change 1126 (e.g., a stroke or symbol) to the user interface 1100 to display user interface element 1134. For example... Figure 11J As shown, user interface element 1134 is a solid line corresponding to tool 1112a.

[0314] Figures 11K to 11L Another sequence is shown in which an action (e.g., opening a menu) is performed on a user interface element based on finger manipulation input (e.g., a tap on the stylus) determined from finger manipulation data from the stylus. Figure 11K As shown, when the user is holding the stylus 203 in his / her hand 1102, the stylus 203 detects input 1120b (e.g., a tap on the stylus 203) at the position of the stylus 203 relative to the electronic device 100.

[0315] In response to receiving instructions Figure 11K The input of finger manipulation data in 1120b is in Figure 11L In this configuration, electronic device 100 displays menu 1144 on user interface 1110. Menu 1144 includes five tool indicators: felt-tip marker tool indicator 1144a, brush tool indicator 1144b, eraser tool indicator 1144c, pencil tool indicator 1144d, and chisel marker tool indicator 1144e, wherein felt-tip marker tool indicator 1144a has focus by default (as shown by focus indicator 1144i). Those skilled in the art will understand that menu 1144 can include any number of tool indicator types with various characteristics, wherein any one of these tool indicators has focus by default.

[0316] Figures 11L to 11MAnother sequence of operations is shown, instructing finger manipulation input (e.g., counter-clockwise rotation of the stylus) based on finger manipulation data determined from the stylus. For example, in response to receiving finger manipulation data from stylus 203 instructing counter-clockwise rotation 1130d of stylus 203, electronic device 100 moves counter-clockwise through (e.g., changes display) menu 1144, causing the focus to change from felt-tip marker tool indicator 1144a to brush tool indicator 1144b.

[0317] Figures 11M to 11N This illustrates performing an action (e.g., a selection action) on a user interface element based on a determined finger manipulation data from the stylus instructing a manipulation input (e.g., a tap on the stylus). Figure 11M As shown, when the user is holding the stylus 203 in his / her hand 1102, the stylus 203 detects input 1140b (e.g., a tap on the stylus 203) at the position of the stylus 203 relative to the electronic device 100. In response to... Figure 11M The finger manipulation data obtained from the instruction input 1140b and the brush tool selection indicator 1144b are used in... Figure 11N In this process, the electronic device 100 removes the menu 1144 from the user interface 1110 on the display. Additionally, Figure 11M The indicator 1112a associated with the first marking tool (e.g., a felt-tip marker) is in Figure 11N The middle part becomes an indicator 1112b associated with a second marking tool (e.g., a brush).

[0318] Figures 11N to 11O This illustrates another sequence that makes a fourth change to the displayed content. Figure 11N In the process, when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1170 (e.g., drawing strokes or markings) from the stylus 203. In response to the electronic device 100 detecting input 1170, Figure 11O In the middle, electronic device 100 is based on Figure 11N Input 1170 is used to display a fourth change 1136 (e.g., a stroke or symbol) to the user interface 1100 to display user interface element 1154. For example... Figure 11O As shown, user interface element 1154 is a drawing stroke corresponding to tool 1112c.

[0319] Figures 12A to 12O Exemplary user interfaces for displaying user interface elements based on the hovering distance of stylus 203, according to some embodiments, are shown. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 19A to 19CThe process in the process. Although some examples in the following examples will refer to input on a touchscreen display (where a touch-sensitive surface and a display are combined, such as on touchscreen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 separate from display 650, such as Figure 6B As shown. In various embodiments, the electronic device 100 displays user interface elements based on the hovering distance of the stylus 203, which is based on data received from the stylus 203.

[0320] In various embodiments, the touch-sensitive surface of the stylus 203 (e.g., Figure 2 and Figures 5A to 5B The touch-sensitive surface 275 in the stylus 203 detects touch input and gesture input, or the absence of touch input and gesture input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0321] In various embodiments, the stylus 203 uses orientation and / or motion sensors (e.g., accelerometers, magnetometers, gyroscopes) to detect orientation / motion input or the absence of orientation / motion input. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides the electronic device 100 with data indicating one or more of the following: whether the stylus is being held, the pen barrel rotation and / or its direction, rotation and / or its direction, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0322] Figures 12A to 12C An example is shown where an icon is displayed based on the stylus hovering distance meeting a first distance threshold. Figure 12A The images include a bird's-eye view 1202 and a side view 1204 of the electronic device 100. As shown in the bird's-eye view 1202, the electronic device 100 displays a user interface 1206 (e.g., associated with a drawing or notepad application) including a visual indicator 1208 that indicates the stylus 203 is being held by the hand of the user 1210. The visual indicator 1208 corresponds to a solid-tipped marker icon so as to indicate that the stylus 203 will form a solid mark on the user interface 1206. Those skilled in the art will understand that the visual indicator 1208 can take various forms.

[0323] Bird's-eye view 1202 and side view 1204 include a first position 1212 on the touch-sensitive surface of the electronic device 100, located below the tip 276 of the stylus 203. In some embodiments, the first position 1212 corresponds to the end of a straight vertical line starting from the tip 276 of the stylus 203. Those skilled in the art will understand that the first position 1212 may vertically correspond to various points on the stylus 203, such as the end 277 of the stylus 203, the midpoint of the stylus 203, etc.

[0324] Figure 12A It also includes a rangefinder 1214. The rangefinder 1214 indicates a first hovering distance 1216. The first hovering distance 1216 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above a first position 1212 on the touch-sensitive surface of the electronic device 100. The electronic device 100 determines the first hovering distance 1216 based on data from the stylus 203 (e.g., data indicating input detected at the stylus), data generated at the electronic device 100 (e.g., sensor information at the electronic device 100), or a combination thereof. The rangefinder 1214 also indicates a first distance threshold 1218 and a second distance threshold 1220.

[0325] like Figure 12A As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1222. In response to detecting the tap gesture 1222 and based on determining that a first hovering distance 1216 satisfies (e.g., reaches or exceeds) a first distance threshold 1218, the electronic device 100 displays a first cube 1224a associated with the first position 1212. Therefore, in Figure 12B In the process, electronic device 100 displays the first cube 1224a and maintains the display of visual indicator 1208. For example, in Figure 12B In the user interface 1206, the first cube 1224a is displayed at a position corresponding to the first position 1212 (e.g., the first cube 1224a is centered at the first position 1212). Although the electronic device 100 displays a cube, those skilled in the art will understand that the electronic device 100 may display one or more user interface elements of various types, such as logos, text, menus, bullet points, objects, etc.

[0326] like Figures 12A to 12B As shown, stylus 203 is moved. Therefore, as Figure 12B As shown, bird's-eye view 1202 and side view 1204 show a second position 1226 on the electronic device 100. The second hover distance 1228 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the second position 1226 on the electronic device 100.

[0327] like Figure 12B As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1230. In response to detecting the tap gesture 1230 and based on determining that the second hover distance 1228 satisfies (e.g., reaches or exceeds) a first distance threshold 1218, the electronic device 100 displays a second cube 1224b associated with the second position 1226. Therefore, in Figure 12C In this configuration, electronic device 100 displays the second cube 1224b while maintaining the display of the first cube 1224a and visual indicator 1208. For example, in... Figure 12C In this context, the second cube 1224b is displayed within the user interface 1206 at a position corresponding to the second position 1226 (e.g., the second cube 1224b is centered on the second position 1226). Figure 12C As shown, because the first hovering distance 1216 and the second hovering distance 1228 satisfy the first distance threshold 1218, the first cube 1224a and the second cube 1224b shown thereby share the same properties (e.g., they are the same cubes).

[0328] Figures 12C to 12D An example is shown where an icon is displayed based on the stylus hovering distance meeting a second distance threshold. For example... Figures 12B to 12C As shown, stylus 203 is moved to a position above the third position 1234. Figure 12C As shown, bird's-eye view 1202 and side view 1204 indicate a third position 1234 on the electronic device 100. The third hovering distance 1236 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the third position 1234 on the electronic device 100.

[0329] like Figure 12C As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1238. In response to detecting the tap gesture 1238 and based on determining that the third hover distance 1236 satisfies (e.g., reaches or exceeds) a second distance threshold 1220 (e.g., exceeds), the electronic device 100 displays a third cube 1240 associated with the third position 1234. Therefore, in Figure 12D In this context, the electronic device 100 displays a third cube 1240 while maintaining the display of a first cube 1224a, a second cube 1224b, and a visual indicator 1208. For example, in... Figure 12D In the user interface 1206, the third cube 1224c is displayed at the position corresponding to the third position 1234 (for example, the third cube 1224c is centered on the third position 1234).

[0330] It is worth noting that the electronic device 100 behaves differently depending on whether the stylus hovering distance meets the first distance threshold 1218 or the second distance threshold 1220. That is, based on meeting the first threshold 1218, the electronic device 100... Figures 12B to 12C The first cube 1224a and the second cube 1224b are displayed; and the electronic device displays according to the second distance threshold 1220. Figure 12D A larger third cube 1240. Those skilled in the art will understand that a user interface element corresponding to satisfying the first distance threshold 1218 can differ in various ways from a user interface element corresponding to satisfying the second distance threshold 1220.

[0331] Figures 12E to 12F Another example is shown where an icon is displayed based on the stylus hovering distance meeting a first distance threshold. For example... Figure 12E As shown in the bird's-eye view 1202, the electronic device 100 displays a user interface 1206 (e.g., associated with a drawing or notepad application) including a visual indicator 1208 that indicates the stylus 203 is being held by the user 1210's hand. Figure 12D As further shown, the bird's-eye view 1202 and the side view 1204 indicate a fourth position 1242 on the touch-sensitive surface of the electronic device 100, located below the tip 276 of the stylus 203. The fourth hovering distance 1244 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the fourth position 1242 on the electronic device 100.

[0332] like Figure 12E As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1246. In response to detecting the tap gesture 1246 and based on determining that the fourth hover distance 1244 satisfies (e.g., reaches or exceeds) a first distance threshold 1218, the electronic device 100 displays a solid ellipse 1248 associated with the fourth position 1242. Therefore, in Figure 12F In this context, electronic device 100 displays a solid oval 1248 and maintains the display of visual indicator 1208. For example, in... Figure 12F In the user interface 1206, a solid ellipse 1248 is displayed at a position corresponding to the fourth position 1242 (e.g., the solid ellipse 1248 is centered on the fourth position 1242). Although the electronic device 100 displays a solid ellipse 1248, those skilled in the art will understand that the electronic device 100 may display one or more user interface elements of various types, such as logos, menus, bullet points, objects, etc.

[0333] Figures 12F to 12GAnother example is shown where an icon is displayed based on the stylus hovering distance meeting a second distance threshold. For example... Figures 12E to 12F As shown, stylus 203 is moved to a position above the fifth position 1250. Figure 12F In the bird's-eye view 1202 and side view 1204, a fifth position 1250 is indicated on the electronic device 100. The fifth hovering distance 1252 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the fifth position 1250 on the electronic device 100.

[0334] like Figure 12F As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1254. In response to detecting the tap gesture 1254 and based on determining that the fifth hover distance 1252 meets (e.g., reaches or exceeds) a second distance threshold 1220, the electronic device 100 displays a splash icon 1256 associated with the fifth position 1250. Therefore, in Figure 12G In this configuration, electronic device 100 displays a sputtering mark 1256 and maintains a display of a solid oval 1248 and a visual indicator 1208. For example, in... Figure 12G In the interface 1206, the sputtering identifier 1256 is displayed at a position corresponding to the fifth position 1250 (e.g., the sputtering identifier 1256 is centered on the fifth position 1250).

[0335] It is worth noting that the electronic device 100 behaves differently depending on whether the stylus hovering distance meets the first distance threshold 1218 or the second distance threshold 1220. That is, based on meeting the first threshold 1218, the electronic device 100 displays... Figure 12F The solid ellipse 1248 in the middle; and according to the second distance threshold 1220, the electronic device 100 displays. Figure 12G The splashing identifier in the text is 1256.

[0336] Figures 12H to 12I Another example is shown where bullet points are displayed based on the stylus hover distance meeting a first distance threshold. For example... Figure 12H As shown, bird's-eye view 1202 and side view 1204 show a sixth position 1258 on the touch-sensitive surface of the electronic device 100, located below the tip 276 of the stylus 203. The sixth hover distance 1260 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the sixth position 1258 on the electronic device 100.

[0337] like Figure 12HAs shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1262. In response to detecting the tap gesture 1262 and based on determining that the sixth hover distance 1260 meets (e.g., reaches or exceeds) a first distance threshold 1218, the electronic device 100 displays a bullet point 1264 associated with the sixth position 1258 near the text box 1266. Therefore, in Figure 12I In this context, the electronic device 100 displays bullet points 1264 near the text box 1266 and maintains the display of visual indicators 1208. For example, in... Figure 12I In the user interface 1206, bullet 1264 and text box 1266 are displayed at positions corresponding to the sixth position 1258 (e.g., bullet 1264 and text box 1266 are centered at the sixth position 1258).

[0338] In some embodiments, when displaying text box 1266, electronic device 100 displays bullet point 1266. In some embodiments, electronic device 100 displays both bullet point 1264 and text box 1266 simultaneously. Those skilled in the art will understand that electronic device 100 can display one or more user interface elements from a variety of user interface elements, such as logos, menus, bullet points, objects, etc.

[0339] Figures 12J to 12K An example is shown where a second distance threshold is met based on the stylus hover distance, and no bullet points are displayed. Figure 12J As shown in the bird's-eye view 1202, the electronic device 100 displays a user interface 1206 and a visual indicator 1208 indicating that the stylus 203 is being held by the hand of the user 1210. (See also...) Figure 12J As further shown, the bird's-eye view 1202 and the side view 1204 indicate a seventh position 1268 on the touch-sensitive surface of the electronic device 100, located below the tip 276 of the stylus 203. The seventh hovering distance 1270 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the seventh position 1268 on the electronic device 100.

[0340] like Figure 12J As shown, the electronic device 100 receives data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1272. In response to detecting the tap gesture 1272 and determining that a seventh hover distance 1270 satisfies (e.g., reaches or exceeds) a second distance threshold 1220, the electronic device 100 does not display a bullet point or text box. Therefore, in Figure 12K In this case, electronic device 100 does not display bullet points or text boxes and maintains the display of visual indicator 1208.

[0341] It is worth noting that the electronic device 100 behaves differently depending on whether the stylus hovering distance meets a first distance threshold 1218 or a second distance threshold 1220. That is, depending on whether the first threshold 1218 is met, the electronic device 100... Figure 12I In the middle, the electronic device 100 displays bullet points 1264 near the text box 1266; and according to satisfying the second distance threshold 1220, in Figure 12K In this case, electronic device 100 does not display either text boxes or bullet points.

[0342] Figures 12L to 12M An example is shown where a menu is displayed based on the stylus hover distance meeting a first distance threshold. For example... Figure 12L As shown in the bird's-eye view 1202, the electronic device 100 displays a user interface 1206 and a visual indicator 1208 indicating that the stylus 203 is being held by the hand of the user 1210. (See also...) Figure 12L As further shown, the bird's-eye view 1202 and the side view 1204 indicate an eighth position 1274 on the touch-sensitive surface of the electronic device 100, below the tip 276 of the stylus 203. The eighth hovering distance 1276 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the eighth position 1274 on the electronic device 100.

[0343] like Figure 12L As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1278. In response to detecting the tap gesture 1278 and determining that the eighth hover distance 1276 meets (e.g., reaches or exceeds) a first distance threshold 1218, the electronic device 100 displays a menu 1280 associated with the eighth position 1274. Therefore, in Figure 12M In this context, electronic device 100 displays menu 1280 and maintains the display of visual indicator 1208. For example, in... Figure 12M In this context, menu 1280 is displayed within user interface 1206 at a position corresponding to the eighth position 1274 (e.g., menu 1280 is centered at the eighth position 1274). Menu 1280 includes four visual indicators, wherein the solid indicator 1280a has focus by default. Those skilled in the art will understand that menu 1280 may include any number of visual indicator types with various characteristics, wherein any one of these visual indicators has focus by default.

[0344] In some embodiments, the electronic device 100 may change which indicator is in focus in response to the user 1210’s hand manipulating the stylus 203. For example, in response to receiving data from the stylus 203 indicating clockwise rotation of the stylus 203, the electronic device 100 moves clockwise through (e.g., changes the display) menu 1280, causing the focus to change from the solid indicator 1280a to the dashed indicator 1280b (not shown).

[0345] Figures 12N to 12O An example is shown where a menu is not displayed based on the stylus hover distance meeting a second distance threshold. Figure 12N As shown in the bird's-eye view 1202, the electronic device 100 displays a user interface 1206 and a visual indicator 1208 indicating that the stylus 203 is being held by the hand of the user 1210. (See also...) Figure 12N As shown, bird's-eye view 1202 and side view 1204 indicate a ninth position 1282 on the touch-sensitive surface of the electronic device 100, located below the tip 276 of the stylus 203. The ninth hover distance 1284 corresponds to the distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the ninth position 1282 on the electronic device 100.

[0346] like Figure 12N As shown, the electronic device 100 receives data from the stylus 203 indicating that the stylus 203 has detected a tap gesture 1286. In response to detecting the tap gesture 1286 and determining that a ninth hover distance 1284 satisfies (e.g., reaches or exceeds) a second distance threshold 1220, the electronic device 100 does not display a menu. Therefore, in Figure 12O In this case, electronic device 100 does not display a menu and keeps displaying visual indicator 1208.

[0347] It is worth noting that the electronic device 100 behaves differently depending on whether the stylus hovering distance meets a first distance threshold 1218 or a second distance threshold 1220. That is, depending on whether the first threshold 1218 is met, the electronic device 100... Figure 12M In the middle, the electronic device 100 displays menu 1280; and according to satisfying the second distance threshold 1220, in Figure 12O In the middle, electronic device 100 does not display a menu.

[0348] Figure 13AThis is a flowchart illustrating a method 1300 for processing sensor data collected at a stylus according to some embodiments. Method 1300 envisions an electronic device 100 processing sensor data obtained from a stylus 203. As shown in block 1302, the stylus 203 detects user input. This user input corresponds to one of the various user inputs described in this disclosure. As shown in step 1304, the stylus 203 provides sensor information to the electronic device 100. The sensor information indicates whether the stylus detected user input or not (e.g., when the stylus 203 is not held). As shown in block 1306, based on the sensor information, the electronic device 100 determines (e.g., processes, interprets, translates, decodes, etc.) the input type. This input type corresponds to one of the various input types described in this disclosure. As shown in block 1308, the electronic device 100 performs an operation based on the input type. This operation corresponds to one of the various operations described in this disclosure.

[0349] Figure 13B This is a flowchart illustrating another method 1310 for processing sensor data collected at a stylus according to some embodiments. As shown in block 1312, stylus 203 detects user input. This user input corresponds to one of the various user inputs described in this disclosure. As shown in block 1314, stylus 203 determines the input type based on the detected user input. In various embodiments, stylus 203 determines (e.g., processes, interprets, translates, decodes, etc.) the input type. In some embodiments, a touch interpretation module 477 of stylus 203 determines the input type. Although not shown, in various embodiments, stylus 203 and electronic device 100 jointly (e.g., collaboratively) determine the input type. In other words, stylus 203 and electronic device 100 share the processing corresponding to determining the input type. This input type corresponds to one of the various input types described in this disclosure. As shown in step 1316, stylus 203 provides information indicating the input type to electronic device 100. In various embodiments, stylus 203 and electronic device 100 jointly determine the input type, and stylus 203 does not provide information indicating the input type. In other words, method 1310 does not execute box 1316. As shown in box 1318, electronic device 100 performs an operation based on the input type. This operation corresponds to one of the various operations described in this disclosure.

[0350] Figures 14A to 14C This is a flowchart illustrating a method 1400 for changing the state of an application according to some embodiments. In an electronic device having a touch-sensitive surface, a display, and a communication interface (e.g., a Bluetooth interface) provided for communicating with a stylus (e.g., ...), Figure 3 Electronic devices 300 or Figure 1AMethod 1400 is performed at a portable multi-functional device 100. In some embodiments, the touch-sensitive surface and display are combined into a touchscreen display (e.g., a mobile phone or tablet). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1400 are optionally combined, and / or the order of some operations is optionally changed.

[0351] Using sensor data from the stylus to transition an electronic device from a first application state to a second application state reduces the amount of input required to perform the transition. This reduction in input enhances the operability of the electronic device and makes the interface more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the electronic device), which in turn reduces the device's power consumption and wear and tear.

[0352] Method 1400 envisions an electronic device utilizing data received from a stylus to take advantage of numerous detectable input types at the stylus. The stylus detects input from the user's hand when the user is holding the stylus and detects input when the user is not holding the stylus. Due to the complex and varied hand manipulation capabilities of users, the stylus can detect many types of user input. The stylus provides data to the electronic device indicating these user inputs. Therefore, method 1400 envisions an electronic device receiving from the stylus various types of data indicating the various user inputs detected at the stylus.

[0353] This enhances the operability of electronic devices and makes their interfaces more efficient and robust. As mentioned above, users can provide various input types to the stylus (e.g., finger manipulation, gestures, rotational movement, etc.). On the other hand, the touch-sensitive surface of an electronic device can receive a single input type (e.g., touch input). A single input type limits the user's ability to interact with the electronic device and can lead to erroneous user input. Therefore, the shift of at least some user input from the touch-sensitive surface of the electronic device to the stylus provides a more efficient user interface and reduces the number of erroneous inputs. Furthermore, this shift to less touch input on the touch-sensitive surface of the electronic device reduces wear and tear and power consumption. This extends the battery life of the electronic device by enabling users to use it more quickly and efficiently. For battery-powered electronic devices, allowing users to input less on the touch-sensitive surface saves power and increases the time interval between battery charges.

[0354] Compared to Figure 14AWhen the electronic device is in the first state, it obtains (1402) information about the current state of the stylus via a communication interface. For example, this information corresponds to sensor data collected by the stylus's magnetometer, accelerometer, capacitive touch element on the stylus barrel, or touch-sensitive surface. For example, the sensor data is transmitted / received via Bluetooth connection, IEEE 802.11x connection, etc.

[0355] For example, refer to Figure 7B The electronic device 100 receives data from the stylus 203 indicating that the stylus is being held by the hand of the user 702. For example, in... Figure 7I In this context, electronic device 100 receives data from stylus 203 indicating that the stylus is not being held by user 702's hand. For example, in... Figure 7D In this process, device 100 receives data from stylus 203 indicating that stylus 203 is undergoing a rotational movement 722 performed by user 702's hand. For example, see reference... Figure 7F The electronic device 100 receives data from the stylus 203 indicating that the stylus 203 is detecting a tapping gesture 728 from the user 702's hand.

[0356] In some embodiments, when the electronic device is in a first state, it operates in an inactive mode (1404). Operating the electronic device in an inactive mode when in the first state enhances its operability and makes it more efficient, which extends its battery life. For example, the display of the electronic device is off and does not display a user interface in the first state. For example, such as... Figure 7K As shown, the electronic device 100 displays a lock screen 736 and provides limited functionality, resulting in lower power consumption. For example, as... Figure 7V As shown, the electronic device 100 displays a home screen 746 and has no active foreground applications running, resulting in less power consumption.

[0357] In some embodiments, when the electronic device is in a first state, the electronic device displays (1406) a first interface on the display. For example, the first interface corresponds to a lock screen. For example, such as... Figure 7K As shown, when the electronic device 100 is operated in the first state while the stylus 203 is not held by the user, the electronic device displays a lock screen 736 (e.g., a first interface). For example, the first interface corresponds to the home screen 746, such as... Figure 7V As shown. For example, the first interface corresponds to the drawing interface 706, as shown... Figure 7A As shown.

[0358] In some embodiments, at least a portion of the information about the current state of the stylus corresponds (1408) to touch sensor data from one or more touch sensors on the stylus. Making some of the information about the current state of the stylus correspond to stylus touch sensor data enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, as... Figure 7A As shown, the electronic device 100 receives data (e.g., information) from the stylus 203 indicating that the user is not holding the stylus 203.

[0359] Based on information about the current state of the stylus, it is determined that the user is holding the stylus, and the electronic device displays (1410) a visual indication on the display that the electronic device is in a second state different from the first state. For example, one or more sensors on the stylus (such as a magnetometer, accelerometer, and capacitive touch elements or touch-sensitive surfaces on the stylus barrel) are used to make this determination. Alternatively, in some embodiments, to avoid false alarms, sensor data is used to indicate that the user is holding the stylus based on two or more inputs (e.g., accelerometer, capacitive touch) indicating that the user is holding the stylus. Furthermore, the visual indication is a representation of the stylus, such as an icon, a pen tip, an image of an icon, etc.

[0360] For example, such as Figures 7A to 7C As shown, electronic device 100 from Figure 7A The first state transition shown in the diagram is to Figure 7C The second state shown in the image. Based on the determination in... Figure 7B The user is holding the stylus 203, in Figure 7C In the middle, electronic device 100 displays visual indicator 712 ( Figure 7A (Not shown in the image) to indicate that the electronic device is in the second state.

[0361] For example, such as Figures 7K to 7M As shown, electronic device 100 from Figure 7K The first state transition shown in the diagram is to Figure 7M The second state shown in the image. Based on the determination in... Figure 7L The user is holding the stylus 203, in Figure 7M In the middle, electronic device 100 displays visual indicator 712 ( Figure 7K (Not shown in the image) so as to indicate that the electronic device 100 is in the second state.

[0362] In some embodiments, when the electronic device is in the second state, it operates in an active mode (1412). For example, in the second state, the display of the electronic device is on and displays an interface. For example, such as Figure 7CAs shown, when the electronic device 100 is operated in the second state while the stylus 203 is held by the user, the electronic device displays an enlarged canvas 710 and visual indicators 712.

[0363] In some embodiments, when the electronic device is in a second state, it displays (1414) a second interface different from the first interface on a display, along with a visual indication that the electronic device is in a second state. Displaying a different interface based on data from the stylus enhances the operability of the electronic device and reduces the amount of input required. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear. For example, the second interface corresponds to a home screen or application interface. For example, such as... Figures 7K to 7M As shown, the electronic device 100 transitions from a first state to a second state and in Figure 7M The application interface, including a magnified canvas 710 and visual indicators 712, is displayed.

[0364] In some embodiments, the visual indication corresponds to (1416) a drawing canvas associated with a drawing application. Displaying visual indications indicating that the electronic device is in a second state based on data received from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, as Figure 7M As shown, when the electronic device 100 is operated in the second state, it displays an application interface including a magnified canvas 710 and visual indicators 712.

[0365] In some embodiments, the visual indication corresponds to (1418) an application icon associated with a drawing application. Displaying a visual indication indicating that the electronic device is in a second state based on data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, when an application icon is selected, the electronic device runs (e.g., executes) and displays the drawing application in the foreground. As another example, when the stylus is no longer held, the electronic device stops displaying the application icon.

[0366] In some embodiments, the electronic device (1420) displays a drawing application interface on a display and stops displaying one or more user interface elements associated with the drawing application interface on the display. Stopping the display of user interface elements based on data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, stopping the display of one or more user interfaces corresponds to removing a displayed set of tools, such as a set of marker tools or a color palette. For example, in response to detecting... Figure 7B The stylus 203 is being held, in Figure 7C In the middle, the electronic device 100 stops displaying the navigation area 704, the canvas area 706, and the toolbar area 708.

[0367] In some embodiments, the visual indication corresponds to (1422) a first marking tool, wherein the first marking tool is the currently active marking tool. For example, the visual indication corresponds to an image, icon, text, etc., of the currently active marking tool. Displaying visual indications indicating that the electronic device is in a second state based on data from the stylus enhances the operability of the electronic device and reduces the amount of input required to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, as... Figure 7C As shown, the electronic device 100 displays a visual indicator 712 including a marker icon 716 corresponding to a marking tool. Continuing with this example, Figure 7C The visual indicator 712 shown corresponds to a color (e.g., hue, shade, etc.) to indicate that the markings made on the enlarged canvas 710 will have that color. Those skilled in the art will understand that the visual indicator 712 may include any form of indicator.

[0368] Compared to Figure 14B Based on the determination that the user is not holding the stylus, the electronic device maintains (1424) the electronic device in a first state. For example, the electronic device receives data from the stylus indicating that the user is not holding the stylus due to its absence or lack of data (e.g., below a threshold). Alternatively, to conserve the battery life of the stylus and / or the device, the stylus provides data to the electronic device in response to detecting specific (e.g., significant) touch input (such as gestures on the stylus (e.g., taps, swipes, etc.), manipulation of the stylus itself (e.g., scrolling, turning, etc.)).

[0369] For example, the electronic device remains in a first state without displaying a visual indication that it is in a second state. For instance, such as... Figures 7A to 7B As shown, the electronic device 100 maintains itself in the first state and does not display anything. Figure 7CThe visual indicator 712 indicates that the electronic device 100 is in a second state.

[0370] In some embodiments, the electronic device (1426) displays a drawing application interface on a display, and displays one or more user interface elements associated with the drawing application interface on the display. For example, the one or more user interface elements correspond to a tool set including a drawing implementation, drawing tools, a color palette, etc. As another example, Figure 7A As shown, the electronic device 100 displays user interface elements in the toolbar area 708, including drawing tools (e.g., markers, pencils, rulers) and color palettes.

[0371] In some embodiments, when the electronic device is in the second state, the electronic device (1428): obtains updated information about the current state of the stylus via a communication interface, wherein the updated information indicates that the user is no longer holding the stylus; in response to obtaining the updated information, the electronic device stops displaying the second interface and visual indications on the display; and redisplays the first interface on the display. Stopping and redisplaying the interface based on data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first interface corresponds to a lock screen, and the second interface corresponds to... Figures 7M to 7O The drawing interface shown. In this example, in response to detecting that the stylus 203 is no longer being held, Figures 7M to 7N The electronic device 100 stops displaying the enlarged canvas 710 and visual indicator 712, and... Figure 7O Redisplay in Figure 7K The lock screen is displayed in the middle. For example, the first interface corresponds to... Figure 7J The first drawing interface is shown, and the second interface corresponds to, as shown in the figure. Figure 7H The second drawing interface is shown.

[0372] In some embodiments, when the electronic device is in a second state, the electronic device (1430): obtains first finger manipulation data from the stylus via a communication interface, wherein the first finger manipulation data characterizes one or more finger manipulation inputs received at the stylus; and in response to obtaining the first finger manipulation data: changes the currently active marking tool to a second marking tool; and updates the visual indication to correspond to the second marking tool. Changing the active marking tool and updating the visual indication based on data received from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, one or more finger manipulation inputs correspond to gestures made on the tactile surface of the stylus, such as swiping down, swiping up, tapping, etc. As another example, one or more finger manipulation inputs correspond to manipulating the stylus, such as rolling the stylus barrel clockwise or counterclockwise, rotating the stylus clockwise or counterclockwise, etc. As yet another example, finger manipulation data corresponds to data collected by the stylus's magnetometer, stylus's accelerometer and / or capacitive touch elements or tactile surfaces on the stylus barrel.

[0373] For example, finger manipulation data can be transmitted / received via Bluetooth, IEEE 802.11x, or similar connections. In some embodiments, the finger manipulation data includes information about finger movement on the stylus or stylus movement relative to the user's finger (e.g., data indicating how the finger and / or stylus move). In some embodiments, the finger manipulation data includes a processed representation of finger movement on the stylus or stylus movement relative to the user's finger. This processed representation may indicate a gesture or manipulation performed at the stylus, such as a swipe or rotation gesture, and optionally includes information indicating the direction and / or magnitude of the gesture or movement.

[0374] For example, in response to receiving Figure 7F The electronic device 100 transmits the data of a tapping gesture at the stylus 203 to the visual indicator 712. Figure 7F The marker 714 in the middle has been updated to Figure 7G The pencil 730 in the example. For instance, users can scan through a column of marking tools by scrolling the stylus. This column of marking tools could correspond to a customized column of tools, a default column of tools, recently used and / or frequently used tools, etc.

[0375] refer to Figure 14CIn some embodiments, when the electronic device is in a second state, the electronic device (1432): obtains first finger manipulation data from the stylus via a communication interface, wherein the first finger manipulation data represents an upward swipe gesture received on the stylus; and in response to obtaining the first finger manipulation data, displays a color palette on the display near a visual indicator. Displaying a color palette based on data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the color palette corresponds to a user interface area with a variety of different colors to choose from, such as a color wheel, a grid with different color areas, a column of colors, etc. For example, in Figure 7C In the process, the electronic device 100 receives data from the stylus 203 indicating a detected downward swipe gesture 718, and... Figure 7D In the electronic device 100, a proximity visual indicator 712 displays a color palette 720. Continuing this example, in response to receiving... Figure 7D The electronic device 100 will display the data of rotational manipulation (e.g., scrolling) of the indicator stylus 203 from the visual indicator 712. Figure 7D The marker 714 with a solid tip 716 has been updated to... Figure 7E The marker 714 has a striped tip 724.

[0376] In some embodiments, when the electronic device is in a second state, the electronic device (1434): obtains second finger manipulation data from a stylus via a communication interface, wherein the second finger manipulation data characterizes a rotational gesture received at the stylus; and in response to obtaining the second finger manipulation data: changes the color associated with the currently active marking tool; and updates the visual indication to correspond to the color. Updating the color and visual indication associated with the active marking tool based on data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear on the electronic device. For example, in order to change the color and update the visual indication at the electronic device, the stylus rotation is greater than a threshold angular distance and / or greater than a threshold angular velocity. For example, the electronic device displays a first indicator (e.g., a star) next to the selected color and / or displays a second indicator (e.g., a ring) around the selected color. For example, the electronic device displays a color icon that changes color. In this example, the electronic device increases the size of the icon corresponding to the currently selected color.

[0377] For example, in response to Figure 7D The device receives data indicating rotational manipulation 722 (e.g., scrolling) of the stylus 203. Figure 7EIn this case, the electronic device 100 stops displaying the solid fill indicator 720a with focus and displays the diagonal fill indicator 720b with focus. Alternatively, scrolling the stylus 203 in one direction (e.g., clockwise) moves the focus downwards (e.g., from 720a to 720b), while scrolling the stylus 203 in another direction (e.g., counterclockwise) moves the focus downwards (e.g., from 720b to 720a). Those skilled in the art will understand that the nature (e.g., direction) of the stylus rotation can have different effects on the user interface.

[0378] In some embodiments, when the electronic device is in a second state, the electronic device (1436): obtains third-finger manipulation data from a stylus via a communication interface, wherein the third-finger manipulation data characterizes a downward swipe gesture received at the stylus; and in response to obtaining the third-finger manipulation data, removes the display of the palette on the screen. Removing the palette based on data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, in response to in Figure 7E The stylus receives data at stylus 203 indicating an upward swipe gesture 726. Figure 7F In the middle, electronic device 100 stops displaying the color palette 720.

[0379] It should be understood that, Figures 14A to 14C The specific order in which the operations described herein are presented is merely exemplary and is not intended to indicate that this order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein.

[0380] The above references Figures 14A to 14C The operation is optionally performed by Figures 1A to 1BThe components depicted are used to implement this. For example, obtaining operation 1402, determining operations 1410 and 1424, and displaying operations 1406, 1414, 1420, and 1426 are optionally implemented by event classifier 170, event recognizer 180, and event handler 190. Event monitor 171 in event classifier 170 detects a touch (or proximity) on touch-sensitive display 112, and event assigner module 174 transmits the event information to application 136-1. The corresponding event recognizer 180 of application 136-1 compares the event information with the corresponding event definition 186 and determines whether a first touch (or proximity) at a first location on the touch-sensitive surface (or rotation of the electronic device) corresponds to a predefined event or sub-event, such as selection of an object on the user interface or rotation of the electronic device from one orientation to another. When a corresponding predefined event or sub-event is detected, event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. Event handler 190 optionally uses or invokes data updater 176 or object updater 177 to update the application's internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will readily understand that based on... Figure 1A-Figure 1B How the components described herein can implement other processes.

[0381] It should be noted that the details of the process described above with respect to method 1400 also apply in a similar manner to other methods described herein (e.g., 1500, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700). For example, the stylus, stylus state, touch-sensitive surface, display, and communication interface described above with reference to method 1400 optionally have one or more of the attributes of the stylus, stylus state, touch-sensitive surface, display, and communication interface described herein with reference to other methods described herein (e.g., 1500, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700).

[0382] Figures 15A to 15B This is a flowchart illustrating a method 1500 for changing stylus functionality according to some embodiments. In an electronic device (e.g., having a touch-sensitive surface, a display, and a communication interface (e.g., Bluetooth interface) provided for communicating with a stylus... Figure 3 Electronic devices 300 or Figure 1AMethod 1500 is performed at a portable multi-functional device 100. In some embodiments, the touch-sensitive surface and display are combined into a touchscreen display (e.g., a mobile phone or tablet). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1500 are optionally combined, and / or the order of some operations is optionally changed.

[0383] Changing stylus functionality based on sensor data from the stylus reduces the amount of input required to perform these changes. This reduction in input enhances the operability of electronic devices and makes electronic device interfaces more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with electronic devices), which in turn reduces the power consumption and wear and tear of electronic devices.

[0384] Method 1500 envisions an electronic device utilizing data received from a stylus to take advantage of numerous detectable input types at the stylus and the stylus's orientation relative to the electronic device. For example, the electronic device receives data from the stylus indicating how the stylus is being held (e.g., a grip arrangement). Due to the complex and varied hand manipulation capabilities of users, the stylus can detect many types of user input. The stylus provides data to the electronic device indicating these user inputs. Therefore, method 1500 envisions an electronic device receiving various types of data from the stylus indicating various types of user inputs detected at the stylus. Additionally, method 1500 envisions the received data including information about the stylus's orientation relative to the electronic device.

[0385] This enhances the operability of electronic devices and makes their interfaces more efficient and robust. As mentioned above, users can provide various input types to the stylus (e.g., grip arrangement) and change the stylus's orientation (e.g., which end of the stylus contacts the electronic device). On the other hand, current systems envision the touch-sensitive surface of the electronic device receiving a single type of input: touch input from the user's finger. This single input type limits the user's ability to interact with the electronic device and can lead to erroneous user input. Therefore, shifting at least some input from finger touch input to the aforementioned stylus input provides a more efficient user interface with the electronic device and reduces the number of erroneous inputs at the electronic device. Furthermore, this shift to less finger touch input at the touch-sensitive surface of the electronic device reduces wear and tear and power consumption. This extends the battery life of the electronic device by allowing users to use it more quickly and efficiently. For battery-powered electronic devices, enabling users to input less on the touch-sensitive surface saves power and increases the time interval between battery charges.

[0386] refer to Figure 15A Electronic device 100 detects (1502) input from a stylus on the touch-sensitive surface of the electronic device. For example, reference Figures 8A to 8B The electronic device 100 detects an input 810 (e.g., a drawing stroke or mark) corresponding to the contact vector between the stylus 203 and the touch-sensitive surface of the electronic device 100.

[0387] In some embodiments, the electronic device (1504): acquires sensor data from the stylus; and determines a grip arrangement characterizing the user's manipulation of the stylus based on the sensor data from the stylus, wherein the grip arrangement is determined during input detection. Acquiring grip arrangement data to influence operations performed by the electronic device enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the sensor data corresponds to the magnetometer of the stylus 203, the accelerometer of the stylus 203, the capacitive touch element on the stylus 203 barrel, or the touch-sensitive surface (e.g., such as...) Figure 2 and Figures 5A to 5B The data collected by the stylus 203 (such as the touch-sensitive surface 275) is shown. In some embodiments, the sensor data is transmitted from the stylus 203 to the electronic device 100 via Bluetooth connection, IEEE 802.11x connection, etc.

[0388] For example, the user holds the stylus 203 near the tip 276, where the tip 276 points towards the electronic device 100 to form an identifier associated with the pen marking tool (e.g., as shown in the image). Figures 8A to 8B (As shown). Alternatively, the user holds the stylus 203 near the end 277 opposite the tip 276, where the tip 276 points towards the electronics 100 to form an identifier associated with a paintbrush marking tool (e.g., as shown). Figures 8C to 8D (As shown). For example, the user holds the stylus 203 near the end 277 opposite the tip 276, where the tip 276 points outward from the electronics 100 to form a mark associated with an eraser marking tool (e.g., as shown). Figures 8E to 8F (As shown). For example, the user holds the stylus 203 near the tip 276, which points outward from the electronics 100 to form markings associated with a smearing or spraying marking tool (e.g., as shown). Figures 8G to 8H (As shown).

[0389] For example, refer to Figures 8A to 8BThe electronic device 100 determines, based on sensor data from the stylus 203, that the user is holding the stylus 203 in his / her hand 802 in a first grip arrangement 815, which corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing towards the electronic device 100), wherein the fingers of the hand 802 are close to the tip of the stylus 203. In this example, reference... Figures 8A to 8B When input 810 (e.g., drawing strokes or markings) is detected via the touch-sensitive surface of electronic device 100, electronic device 100 determines that the user is holding stylus 203 in his / her hand 802 in a first grip arrangement 815. For example, see reference... Figures 8C to 8D The electronic device 100 determines, based on sensor data from the stylus 203, that the user is holding the stylus 203 in his / her hand 802 in a second grip arrangement 835, which corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing toward the electronic device 100), wherein the fingers of the hand 802 are close to the end 277 of the stylus 203 opposite to the tip 276 of the stylus 203.

[0390] In some embodiments, the grip arrangement is determined (1506) based on at least one of grip style, grip position, or stylus orientation relative to a reference frame. Obtaining grip arrangement data to influence operations performed by the electronic device enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, grip style corresponds to the position of contact points on the stylus relative to each other (e.g., the position of different fingers). For example, grip position corresponds to the position of contact points on the stylus relative to the stylus (e.g., a finger near the tip of the stylus). For example, stylus orientation corresponds to the position of the stylus relative to the electronic device, gravity, and / or the Earth's magnetic field.

[0391] In some embodiments, the grip style is determined at least in part based on touch input in contact with the stylus, such as the number of fingers on the stylus, which fingers are on the stylus, pinch grip, or first grip. In some embodiments, the grip position is determined at least in part based on the position of the grip (e.g., touch input) relative to the stylus. For example, the grip position includes the location of the touch input on the stylus (e.g., near the tip of the stylus or near the base of the stylus). In some embodiments, the grip style is determined at least in part based on the orientation of the stylus relative to the electronic device (e.g., upright or inverted).

[0392] For example, refer to Figures 8A to 8BThe electronic device 100 determines a first grip arrangement 815. The first grip arrangement 815 corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing towards the electronic device 100), wherein the fingers of the hand 802 are close to the tip 276 of the stylus 203. For example, see reference... Figures 8E to 8F The electronic device 100 determines a third gripping arrangement 855. The third gripping arrangement 855 corresponds to gripping the stylus 203 in an inverted orientation (e.g., the tip 276 of the stylus 203 points outward from the electronic device 100) and near the end 277 of the stylus 203 opposite the tip 276. For example, see reference... Figures 8G to 8H The electronic device 100 determines a fourth grip arrangement 875, which corresponds to gripping the stylus 203 in an inverted orientation (e.g., the tip 276 of the stylus 203 points outward from the electronic device 100) and close to the tip 276 of the stylus 203.

[0393] In some embodiments, at least a portion of the sensor data corresponds (1508) to touch sensor data obtained from one or more touch sensors on the stylus. Obtaining sensor data from the stylus to influence operations performed by the electronic device enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, a grip style and / or grip position is determined based on a portion of the sensor data indicating the location of a touch detected on the touch-sensitive surface of the stylus. For example, referencing Figures 8A to 8B The electronic device 100 determines a first grip arrangement 815 based on sensor data received from the stylus 203 indicating that the stylus 203 is being held in an upright orientation (e.g., the tip 276 of the stylus 203 is pointing towards the electronic device 100), wherein the fingers of the hand 802 hold the stylus 203 near the tip 276 of the stylus 203.

[0394] In response to detecting the input and determining that the stylus is being held in a first grip arrangement, the electronic device makes a (1510) first change to the content displayed on the display, wherein the first grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus. For example, sensors at the stylus (e.g., a capacitive touch sensor, accelerometer, magnetometer, or gyroscope) detect the first grip arrangement. For example, the first change corresponds to drawing lines, spraying, erasing, etc., with a paintbrush / pencil / airbrush. For example, the first change is associated with a first marking tool corresponding to the first grip arrangement.

[0395] For example, refer to Figures 8A to 8BThe electronic device 100 detects input 810 and determines a first grip arrangement 815 based at least in part on sensor data detected by the stylus 203. Therefore, in Figure 8B In the process, the electronic device 100 makes a first change 820 (e.g., a stroke or symbol) to the user interface 800 based on the input 810.

[0396] In some embodiments, making the first change includes (1512) displaying a first user element based on a first marking tool corresponding to a first grip arrangement. Displaying the user element based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first grip arrangement (e.g., upright stylus orientation, grip position near the tip of the stylus relative to the electronic device) invokes a writing marking tool (e.g., pencil, marker, etc.). For example, refer to Figures 8A to 8B The electronic device 100 makes a first change 820 based on a first marking tool (e.g., a felt-tip marker) corresponding to the first gripping arrangement 815.

[0397] In some embodiments, making the first change includes (1514) altering an existing label displayed on the display based on a first marking tool corresponding to a first grip arrangement. Changing the existing label based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first grip arrangement (e.g., an inverted stylus orientation, the grip position near the bottom of the stylus relative to the electronic device) invokes an eraser marking tool.

[0398] For example, refer to Figures 8E to 8F The electronic device 100 detects input 810 and determines a third grip arrangement 855 based at least in part on sensor data detected by the stylus 203. The third grip arrangement 855 corresponds to a third marking tool (e.g., an eraser), as indicated by the eraser indicator 852. Therefore, in Figure 8F In this process, the electronic device 100 alters the existing sign 804 by displaying a white stroke / mark 860 (e.g., erase) corresponding to the input 810 at an appropriate location on a portion of the existing sign 804.

[0399] In some embodiments, a first grip arrangement is detected (1516) based on detecting that the stylus is in its upright orientation and detecting touch input near the first tip of the stylus, and a first change is made including displaying strokes on the display based on a writing instrument corresponding to the first grip arrangement. Changing the displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, upright orientation is based on the physical properties of the stylus, such as the stylus tip pointing upwards. For example, the first tip of the stylus corresponds to the writing tip of the stylus. For example, the first grip arrangement corresponds to a pencil, pen, marker, etc.

[0400] For example, refer to Figures 8A to 8B The electronic device 100 determines, based on sensor data from the stylus 203, that the user is holding the stylus 203 in an upright orientation (e.g., the tip of the stylus 203 is pointing towards the electronic device 100), with the fingers of the hand 802 close to the tip of the stylus 203. Therefore, in response to input 810, in Figure 8B In the process, the electronic device 100 makes a first change 820, which corresponds to the display of a stroke on the display according to a marking tool (e.g., a felt-tip marker) indicated by the indicator 812.

[0401] refer to Figure 15B In response to the detection of input and based on the determination that the stylus is being held in a second grip arrangement different from the first grip arrangement, the electronic device makes a second change (1518) to the content displayed on the display, wherein the second grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus, and wherein the second change to the content displayed on the display is different from the first change to the content displayed on the display. This reduces wear and tear and battery consumption of the electronic device because the user interface is changed without additional touch to the touch-sensitive surface of the electronic device. For example, sensors at the stylus (e.g., capacitive touch sensor, accelerometer, magnetometer, or gyroscope) detect the second grip arrangement. For example, the second change corresponds to drawing lines, spraying, erasing, etc., with a paintbrush / pencil / airbrush. For example, the first and second are associated with a second marking tool corresponding to the second grip arrangement.

[0402] For example, refer to Figures 8C to 8D Based at least in part on sensor data received from stylus 203, electronic device 100 determines that the stylus is being held in a second grip arrangement 835. The second grip arrangement 835 differs from... Figure 8B The first gripping arrangement 815 in the middle. Therefore, in response to the detection of input 810, in Figure 8D In this process, the electronic device 100 makes a second change 840 to the user interface 800 (e.g., a stroke or symbol). The second change 840 differs from... Figure 8B The first change in 830.

[0403] In some embodiments, making the second change includes (1520) displaying a second user element based on a second marking tool corresponding to the second grip arrangement. Changing the displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second grip arrangement (e.g., upright stylus orientation, grip position near the top of the stylus relative to the electronic device) invokes a drawing marking tool (e.g., a paintbrush, etc.). For example, refer to Figures 8A to 8B Electronic device 100 determines a first gripping arrangement 815. Electronic device 100 determines that the first gripping arrangement 815 corresponds to a felt-tip marker tool and displays an indicator 812 indicating the marker tool. Therefore, in response to detecting input 810, electronic device 100 makes a first change 820 corresponding to a felt-tip marker stroke. For example, see reference... Figures 8C to 8D The electronic device 100 determines a second gripping arrangement 835. The electronic device 100 determines that the second gripping arrangement 835 corresponds to a paintbrush marking tool and displays an indicator 832 that indicates the marking tool. Therefore, in response to the detected input 810, the electronic device 100 makes a second change 840 corresponding to a paintbrush stroke.

[0404] In some embodiments, making the second change includes (1522) altering the existing markings displayed on the display based on a second marking tool corresponding to the second grip arrangement. Changing the existing markings based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second grip arrangement (e.g., an inverted stylus orientation, with the stylus tip near the grip position relative to the electronic device) invokes the marking tool.

[0405] For example, refer to Figures 8E to 8F Electronic device 100 determines a third gripping arrangement 855. Electronic device 100 determines that the third gripping arrangement 855 corresponds to an eraser marking tool and displays an indicator 852 indicating the marking tool. Therefore, in response to detecting input 810, electronic device 100 changes the existing mark 804 by displaying a white stroke 860 (e.g., erasure) above the existing mark 804.

[0406] In some embodiments, a second grip arrangement is detected (1524) based on detecting that the stylus is in an inverted orientation and detecting touch input near a second end of the stylus that is different from the first end of the stylus, and a second change is made including removing existing markings displayed on the display based on an erasing tool corresponding to the second grip arrangement. Changing the displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the inverted orientation is based on the physical properties of the stylus, such as the stylus tip pointing downwards towards the electronic device. For example, the second end corresponds to the eraser tip of the stylus or the end opposite the writing tip of the stylus.

[0407] For example, refer to Figures 8E to 8F The electronic device 100 determines, at least in part, that the stylus 203 is being held in a third grip arrangement 855 based on data received from the stylus 203. The third grip arrangement 855 corresponds to holding the stylus 203 in an inverted orientation (e.g., with the tip 276 of the stylus 203 pointing outwards from the electronic device 100) near the end 277 of the stylus 203 opposite the tip 276. The electronic device 100 determines that the third grip arrangement 855 corresponds to an eraser marking tool and displays an indicator 852 indicating this marking tool. Therefore, in response to the detection of input 810, the electronic device 100 changes the existing mark 804 by displaying a white stroke 860 (e.g., erasure) above the existing mark 804.

[0408] In some embodiments, a second grip arrangement is detected (1526) based on detecting that the stylus is in its upright orientation and detecting touch input near the second end of the stylus, and the second change includes displaying strokes based on a drawing tool corresponding to the first grip arrangement. Changing the displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second end of the stylus corresponds to the eraser tip of the stylus or the end opposite the writing tip of the stylus. For example, the drawing tool corresponds to a pencil, pen, marker, etc.

[0409] For example, refer to Figures 8C to 8DElectronic device 100 determines a second grip arrangement 835. The second grip arrangement 835 corresponds to holding stylus 203 in an upright orientation (e.g., with the tip 276 of stylus 203 pointing towards electronic device 100), wherein the fingers of hand 802 are close to the end 277 of stylus 203 opposite to the tip 276. Electronic device 100 determines that the second grip arrangement 835 corresponds to a paintbrush, as indicated by indicator 832. Therefore, in response to the detection of input 810, electronic device 100 makes a second change 840 corresponding to displaying a paintbrush stroke 840.

[0410] In some embodiments, a second grip arrangement is detected (1528) based on detecting that the stylus is in an inverted orientation and detecting touch input near the first tip of the stylus, and a second change is made by removing existing markings displayed on the display based on a smearing tool corresponding to the second grip arrangement. Changing the displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first tip of the stylus corresponds to the writing tip of the stylus.

[0411] It should be understood that, Figures 15A to 15B The specific order in which the operations described herein are presented is merely exemplary and is not intended to indicate that this order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein.

[0412] The above references Figures 15A to 15B The operation is optionally performed by Figures 1A to 1BThe components depicted are used to implement this. For example, the detection operation 1502, the acquisition and determination operation 1504, and the performance operations 1510 and 1518 are optionally implemented by the event classifier 170, the event recognizer 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects a touch (or proximity) on the touch-sensitive display 112, and the event distributor module 174 transmits the event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first touch (or proximity) at the first position on the touch-sensitive surface (or whether the rotation of the electronic device) corresponds to a predefined event or sub-event, such as the selection of an object on the user interface, or the rotation of the electronic device from one orientation to another. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. Event handler 190 optionally uses or invokes data updater 176 or object updater 177 to update the application's internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will readily understand that based on... Figure 1A-Figure 1B How the components described herein can implement other processes.

[0413] It should be noted that the details of the process described above with respect to method 1500 also apply in a similar manner to other methods described herein (e.g., 1400, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700). For example, the stylus, grip arrangement, display, touch-sensitive surface, and communication interface described above with reference to method 1500 optionally have one or more of the properties of the stylus, grip arrangement, display, touch-sensitive surface, and communication interface described herein with reference to other methods described herein (e.g., 1400, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700).

[0414] Figures 16A to 16B This is a flowchart illustrating a method 1600 for modifying touch input functionality according to some embodiments. In an electronic device having a touch-sensitive surface, a display, and a communication interface (e.g., a Bluetooth interface) provided for communicating with a stylus (e.g., ...), ... Figure 3 Electronic devices 300 or Figure 1AMethod 1600 is performed at a portable multifunction device 100. In some embodiments, the touch-sensitive surface and display are combined into a touchscreen display (e.g., a mobile phone or tablet). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1600 are optionally combined, and / or the order of some operations is optionally changed.

[0415] Modifying touch input functionality based on sensor data from the stylus reduces the amount of input required to execute changes to stylus functions. This reduction in input enhances the operability of electronic devices and makes electronic device interfaces more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with electronic devices), which in turn reduces the power consumption and wear and tear of electronic devices.

[0416] Method 1600 envisions an electronic device utilizing data received from and / or the absence of data received from a stylus. The stylus detects input from the user's hand when the user is holding it and detects input when the user is not holding the stylus. This enhances the operability of the electronic device and makes the electronic device interface more efficient and robust. That is, the functionality of touch input to the electronic device depends on whether the stylus is being held by the user, as indicated by the presence and / or absence of data received from the stylus. In other words, the electronic device can perform multiple functions in response to detecting a specific touch input on the touch-sensitive surface of the electronic device.

[0417] Therefore, compared to current systems where electronic devices perform a single operation in response to the detection of a specific touch input on the touch-sensitive surface of the electronic device, method 1600 achieves a richer set of functions. The single operation envisioned in current systems limits the user's ability to interact with the electronic device and may lead to erroneous user input. Therefore, extending the functions generated by a specific touch input based on whether the stylus is held provides a more efficient user interface with the electronic device and reduces the number of erroneous inputs at the electronic device. Additionally, this reduces wear and tear on the electronic device and power consumption. This extends the battery life of the electronic device by enabling users to use it more quickly and efficiently. For battery-powered electronic devices, it allows users to input less on the touch-sensitive surface of the electronic device, saving power and increasing the time interval between two battery charges.

[0418] refer to Figure 16A Electronic device 100 detects (1602) touch input on a touch-sensitive surface. For example, this touch input corresponds to finger touch input, such as tapping, swiping, or gestures. For example, see reference... Figure 9AElectronic device 100 detects a leftward swipe gesture 908 made by the user's (902) hand. For example, see reference... Figure 9I The electronic device 100 detects a looping gesture 916 (e.g., a lasso gesture) surrounding the content 904.

[0419] In some embodiments, the electronic device obtains (1604) sensor data from a stylus via a communication interface, and at least a portion of this sensor data corresponds to touch sensor data from one or more touch sensors on the stylus. Obtaining sensor data from the stylus to influence touch input functionality enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the sensor data corresponds to a capacitive touch element or touch-sensitive surface on the stylus barrel. For example, sensor data is transmitted / received via Bluetooth connectivity, IEEE 802.11x connectivity, etc. For example, refer to Figure 9H The electronic device 100 obtains sensor data from the stylus 203 indicating that the stylus 203 is not being held by the user 902's hand. For example, see Reference Figure 9A The electronic device 100 obtains sensor data from the stylus 203 indicating that the stylus 203 is being held by the hand of the user 902.

[0420] In response to detecting touch input on a touch-sensitive surface and determining, based on sensor data obtained from the stylus via a communication interface, that the stylus is being held by a user, the electronic device performs a first operation (1606) in response to the touch input. For example, the electronic device determines that the stylus is being held based on data obtained from the stylus indicating that the stylus is detecting one or more fingers in contact with the stylus. For example, the first operation is performed based on the directionality, speed, acceleration, displacement, etc., of the touch input.

[0421] For example, refer to Figures 9A to 9B The electronic device 100, in response to detecting a left swipe gesture 908 and determining, based on data obtained from the stylus 203, that the stylus 203 is being held by the user 902's hand, performs an undo / erase operation on the content 904. For example, see [reference 100] Figures 9F to 9H The electronic device 100 performs a selection and movement operation on the content 904. In this example, in response to detecting a looping gesture 916 relative to the content 904 and a subsequent dragging gesture 922, and determining based on data obtained from the stylus 203 that the stylus 203 is being held by the user 902's hand, the electronic device 100 selects and moves the content 904 according to the dragging gesture 922.

[0422] In some embodiments, the electronic device performs (1608) a first operation including modifying one or more pre-existing user interface elements displayed on the display. Modifying pre-existing user interface elements based on sensor data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, modifying user elements includes copying and pasting icons / objects / text, cutting and pasting icons / objects / text, undoing and redoing icons / objects / text, erasing icons / objects / text, or combinations thereof. For example, the first operation corresponds to lassoing / selecting icons / objects / text to move them to different locations (e.g., the first operation corresponds to a cut and paste operation or a copy and paste operation) and / or to change their appearance.

[0423] For example, refer to Figures 9A to 9C The first operation corresponds to the electronic device 100 performing an erase / undo operation relative to content 904 in response to detecting a left swipe gesture 908. Continuing this example, the first operation corresponds to the electronic device 100 performing a restore operation relative to content 904 in response to detecting a right swipe gesture 910. Also, see reference... Figures 9F to 9G The first operation corresponds to the electronic device 100 performing a selection operation relative to content 904 in response to detecting a looping gesture 916. For example, see reference... Figures 9K to 9L The first operation corresponds to the electronic device 100 responding to the detection Figure 9K The rightward swipe gesture in 938 and like Figure 9L The selection operation is performed relative to the selected text 940.

[0424] In some embodiments, based on sensor data obtained from the stylus via a communication interface indicating that the stylus is being held by a user with a hand different from the hand corresponding to the touch input, the electronic device performs (1610) a second operation in response to the touch input. Performing the second operation based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation corresponds to inkjet can operation, ink printing operation, grid line placement operation, ruler operation, etc. In some embodiments, the electronic device determines the distinction between the hand holding the stylus and the hand making the touch input based on proximity (e.g., closeness). For example, the electronic device determines that the stylus is being held by a user with a hand different from the hand corresponding to the touch input based on determining that the stylus is at least a threshold distance from the electronic device. For example, the electronic device determines the proximity of the stylus to the electronic device based on data received from the stylus, sensor data generated at the electronic device, or a combination thereof.

[0425] In response to detecting touch input on the touch-sensitive surface and determining that the stylus is not being held by the user, the electronic device performs a second operation (1612) in response to the touch input, and this second operation is different from the first operation. This reduces wear and extends battery life because the electronic device does not need to detect the first and second touch inputs to perform the first and second operations. For example, the second operation can be performed based on the directionality, speed, acceleration, displacement, etc., of the touch input.

[0426] In some embodiments, the electronic device determines that the stylus is not being held based on the absence of sensor data, such as the stylus not being paired with the electronic device. In some embodiments, to extend battery life and reduce processing, the stylus provides sensor data only when it experiences a change / event, rather than continuously providing sensor data. In some embodiments, the electronic device determines that the stylus is not being held based on data received from the stylus. For example, the electronic device receives data from the stylus indicating that the stylus is not being held based on the stylus detecting a contact being lifted off the stylus.

[0427] For example, refer to Figures 9D to 9E The second operation corresponds to the electronic device 100 performing a display operation. That is, the electronic device 100 displays the identifier 914 in response to detecting a left swipe gesture 908. For example, see reference... Figures 9N to 9P The second operation corresponds to the electronic device 100 performing a highlight operation relative to the highlighted text 950 in response to detecting a right swipe gesture 938. Continuing with this example, the second operation corresponds to the electronic device 100 performing a display operation on the identifier 954 in response to detecting a drag gesture 942.

[0428] In some embodiments, determining that the stylus is not being held by the user includes (1614) detecting the absence of sensor data from the stylus. Performing a second operation based on the absence of sensor data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device.

[0429] In some embodiments, performing the second operation includes (1616) displaying one or more user interface elements on a display. Displaying user interface elements based on sensor data from a stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation includes displaying a new logo / object / text, highlighting an existing logo / object / text, marking an existing logo / object / text, etc. For example, refer to Figures 9D to 9E The electronic device 100 performs a second operation, the second operation including responding to the detection Figure 9DThe corresponding left swipe gesture in 908 is... Figure 9E The display shows identifier 914. For example, see reference... Figures 9I to 9J The electronic device 100 performs a second operation, the second operation including responding to the detection Figure 9I 916 circular gestures were detected in the middle. Figure 9J The display shows identifier 934. For example, see reference... Figures 9N to 9O The electronic device 100 performs a second operation, the second operation including responding to the detection of a corresponding Figure 9N The highlighted text in 950 and the right swipe gesture in 938 are used to display text in the middle. Figure 9O Medium highlighting of highlighted text 950.

[0430] In some embodiments, performing the second operation includes (1618) navigating within a user interface displayed on the display. For example, the second operation includes panning and / or zooming the canvas. Navigating within the user interface based on sensor data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation corresponds to navigation through user interface elements such as marking tools (e.g., pen, marker, pencil, ruler, etc.). For example, referencing... Figure 9O The electronic device 100 performs a second operation, including enlarging and highlighting text 950 or re-centering text 936 (not shown).

[0431] refer to Figure 16B In some embodiments, the electronic device (1620) detects input from a stylus on a touch-sensitive surface of the electronic device and performs a second operation in response to detecting the input from the stylus. Performing the second operation based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation corresponds to drawing lines. For example, refer to... Figures 9I to 9J Electronic device 100 in response to detecting Figure 9I The loop input 916 is executed in Figure 9J The second operation, identified by flag 904, is shown in the image. Continuing with this example, [the text continues with...]. Figure 9I As shown, the stylus 203 (instead of the user 902's hand) makes a looped input 916 on the touch-sensitive surface of the electronic device 100.

[0432] In some embodiments, the electronic device (1622) detects input from a stylus on a touch-sensitive surface of the electronic device and performs a third operation in response to detecting the input from the stylus, and the third operation is different from the first and second operations. Performing the third operation based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the third operation corresponds to the electronic device displaying paintbrush marks, strokes, inkjet prints, grid lines, etc. on a user interface (not shown).

[0433] It should be understood that, Figures 16A to 16B The specific order in which the operations described herein are presented is merely exemplary and is not intended to indicate that this order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein.

[0434] The above references Figures 16A to 16B The operation is optionally performed by Figures 1A to 1B The components depicted are used to implement this. For example, detection operations 1602, 1620, and 1622, determination operations 1606 and 1612, and execution operations 1616 and 1618 are optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects a touch (or proximity) on the touch-sensitive display 112, and an event dispatcher module 174 transmits the event information to the application 136-1. The corresponding event recognizer 180 in the application 136-1 compares the event information with a corresponding event definition 186 and determines whether a first touch (or proximity) at a first location on the touch-sensitive surface (or rotation of the electronic device) corresponds to a predefined event or sub-event, such as the selection of an object on the user interface or rotation of the electronic device from one orientation to another. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. Event handler 190 optionally uses or invokes data updater 176 or object updater 177 to update the application's internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will readily understand that based on... Figure 1A-Figure 1B How the components described herein can implement other processes.

[0435] It should be noted that the details of the process described above with respect to method 1600 also apply in a similar manner to other methods described herein (e.g., 1400, 1500, 1700, 1800, 1900, 2400, 2500, 2600, 2700). For example, the stylus, sensor data, display, touch-sensitive surface, input, and communication interface described above with reference to method 1600 optionally have one or more of the properties of the stylus, sensor data, display, touch-sensitive surface, input, and communication interface described herein with reference to other methods described herein (e.g., 1400, 1500, 1700, 1800, 1900, 2400, 2500, 2600, 2700).

[0436] Figures 17A to 17C This is a flowchart illustrating a method 1700 for performing operations on existing identifiers displayed on an interface based on finger-operated input, according to some embodiments. In an electronic device (e.g., having a touch-sensitive surface, a display, and a communication interface (e.g., Bluetooth interface) provided for communicating with a stylus), Figure 3 Electronic devices 300 or Figure 1A Method 1700 is performed at a portable multifunction device 100. In some embodiments, the touch-sensitive surface and display are combined into a touchscreen display (e.g., a mobile phone or tablet). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1700 are optionally combined, and / or the order of some operations is optionally changed.

[0437] Using finger-based input data from the stylus to manipulate existing icons displayed on the interface reduces the amount of input required to change stylus functionality. This reduction in input enhances the operability of electronic devices and makes their interfaces more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with electronic devices), which in turn reduces power consumption and wear and tear on the devices.

[0438] Method 1700 envisions an electronic device using data received from a stylus to perform operations on existing icons displayed on a tablet computer based on finger manipulation input received by the stylus. These operations include cut / paste, copy / paste, resize / desize, increase / decrease thickness, etc. Many different types of finger manipulation input can be used for many different types of operations. For example, finger manipulation input received by the stylus includes tapping, flicking, swiping, scrolling, rotating, etc.

[0439] This enhances the operability of electronic devices and makes their interfaces more efficient and robust. As mentioned above, users can interact with a stylus in many different ways, while the touch-sensitive surface of an electronic device can receive a single type of input (touch input). Furthermore, the shift to less touch input on the touch-sensitive surface of an electronic device reduces wear and tear and power consumption. This extends the battery life of the electronic device by allowing users to use it more quickly and efficiently. For battery-powered electronic devices, enabling users to input less on the touch-sensitive surface saves power and increases the time interval between battery charges.

[0440] Compared to Figure 17A When multiple user interface elements are displayed on the screen, the electronic device obtains (1702) finger manipulation data from the stylus via a communication interface, wherein the finger manipulation data includes information about one or more finger manipulation inputs received by the stylus. For example, the multiple user interface elements correspond to icons, objects, vector graphics, and / or objects, etc. For example, see reference... Figures 10G to 10I One or more finger manipulation inputs received by the stylus 203 include upward or downward swipe gestures relative to the electronic device 100 on the stylus 203. (e.g., inputs 1040 and 1050). In another example, refer to... Figures 10B to 10F One or more finger manipulation inputs received by the stylus 203 include rotating the stylus 203 barrel in a counter-clockwise or clockwise manner (e.g., inputs 1020a to 1020d). In some embodiments, the finger manipulation data corresponds to data collected by the stylus's magnetometer, accelerometer, and capacitive touch elements or touch-sensitive surfaces on the stylus barrel. In some embodiments, the finger manipulation data is transmitted / received via Bluetooth, IEEE 802.11x, or the like.

[0441] In some embodiments, finger manipulation data includes information about finger movement on the stylus or stylus movement relative to the user's finger (e.g., data indicating how the finger and / or stylus move). In some embodiments, finger manipulation data includes a processed representation of finger movement on the stylus or stylus movement relative to the user's finger (e.g., data indicating gestures or manipulations performed at the stylus, such as swipe or rotation gestures that optionally include information indicating the direction and / or magnitude of the gesture or movement). In one example, finger manipulation data indicates gestures or manipulations performed at the stylus, such as swipe or rotation gestures that optionally include information indicating the direction and / or magnitude of the gesture movement.

[0442] In some embodiments, at least a portion of the finger manipulation data corresponds (1704) to touch sensor data from one or more touch sensors on the stylus. Obtaining finger manipulation data from the stylus to influence operation at the electronic device enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the sensor data corresponds to data collected by the stylus's magnetometer, accelerometer, and capacitive touch elements or touch-sensitive surfaces on the stylus barrel (e.g., transmitted and / or received sensor data via Bluetooth connection, IEEE 802.11x connection, etc.). For example, as Figure 10G As shown, stylus 203 detects input 1040 (e.g., an upward swipe on stylus 203). In another example, as... Figure 10H As shown, stylus 203 detects input 1050 (e.g., a downward swipe on stylus 203).

[0443] In some embodiments, when multiple user interface elements are displayed on a display, the electronic device displays (1706) a drawing application interface including a canvas having multiple pre-existing identifiers displayed in response to previous input from a stylus detected on a touch-sensitive surface of the electronic device. Displaying the canvas based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. In some embodiments, the multiple pre-existing identifiers are generated by writing or drawing strokes from the stylus. In some embodiments, the multiple pre-existing identifiers are generated by one or more vector graphics operations. Vector graphics include, for example, closed objects such as triangles, squares, or any polygons. For example, referencing Figure 10A The user interface 1000 associated with a drawing or notepad application includes pre-existing content: a star 1004a and a lightning bolt 1004b. In another example, reference... Figure 10G The user interface 1000 associated with a drawing or notepad application includes pre-existing content: triangle 1004d.

[0444] In some embodiments, the plurality of user interface elements correspond to (1708) a subset of a plurality of pre-existing identifiers selected by the user. Utilizing data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, in Figure 10A In this context, pre-existing identifiers (e.g., star 1004a and lightning bolt 1004b) appear on the user interface 1000, and lightning bolt 1004b is selected, as shown below. Figure 10B As shown. In another example, the user selects a pre-existing identifier (e.g., triangle 1004d) to perform the cut operation.

[0445] In some embodiments, the plurality of user interface elements correspond (1710) to a subset of pre-existing identifiers selected based on the position of the stylus relative to the electronic device during the detection of one or more finger manipulation inputs. Utilizing finger manipulation data from the stylus enhances the operability of the electronic device and reduces the amount of input to the electronic device. Reducing the amount of input makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, refer to... Figure 10A Pre-existing identifiers (e.g., star 1004a and lightning bolt 1004b) appear on the user interface. In response to the user selecting lightning bolt 1004b (a subset of the multiple pre-existing identifiers), the electronic device 100 displays the selected lightning bolt 1004b' with a dashed outline to indicate that lightning bolt 1004b' is currently selected, while star 1004a remains unchanged.

[0446] In some embodiments, in response to detecting finger manipulation data and determining that the finger manipulation data indicates a first finger manipulation input on the stylus, the electronic device performs (1712) a first operation on at least a subset of the plurality of user interface elements. In one example, in response to detecting... Figure 10B The counter-clockwise rotation of the stylus 203 in the device increases the size of a subset of user interface elements (e.g., in...). Figures 10B to 10C In the example, lightning bolt 1004b' increases in size from the first size 1015a to lightning bolt 1004c' under the second size 1015b. In another example, in response to detecting an upward swipe on stylus 203, a subset of user interface elements (e.g., Figure 10G The triangle 1004d in the diagram performs a cut operation. In yet another example, a subset of user interface elements includes objects, vector graphics, or other similar elements.

[0447] In some embodiments, the first finger manipulation input corresponds to a first gesture type detected on the stylus (1714). Obtaining finger manipulation data from the stylus to influence the execution of an operation enhances the operability of the electronic device and reduces the amount of in...

Claims

1. A method for interacting with a stylus, comprising: At an electronic device having one or more processors, non-transitory memory, a display, a touch-sensitive surface, and a communication interface provided to communicate with a stylus: In response to detecting that the stylus is in proximity to the electronic device, pairing the electronic device with the stylus; and In response to pairing the stylus with the electronic device: displaying, on the display, a first representation of a first gesture performed on the stylus; obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data indicates finger manipulation input received by the stylus; and In response to obtaining the finger manipulation data, displaying on the display a second representation of a second gesture performed on the stylus corresponding to the finger manipulation input received by the stylus.

2. The method according to claim 1, further comprising: obtaining additional finger manipulation data from the stylus via the communication interface, wherein the additional finger manipulation data indicates a second finger manipulation input received by the stylus corresponding to movement of a finger on the stylus; and In response to obtaining the additional finger manipulation data, the second representation of the second gesture performed on the stylus is changed on the display according to the second finger manipulation input. 3 . The method of claim 1 , wherein the stylus being close to the electronic device corresponds to the stylus not being in contact with the electronic device. 4 . The method of claim 1 , wherein the stylus approaching the electronic device corresponds to the stylus contacting the electronic device at a connection point on the electronic device. The method of claim 1 , wherein the first representation of the first gesture is predetermined.

6. The method of claim 1, further comprising detecting one or more inputs on the touch-sensitive surface corresponding to a request to select a particular tutorial, wherein the first representation of the first gesture is based on the particular tutorial. The method of claim 1 , wherein the first representation of the first gesture is displayed without user intervention.

8. The method according to claim 1, further comprising: obtaining, from the stylus via the communication interface, state information regarding one or more states of the stylus; and In response to obtaining the status information, one or more status indicators indicating the one or more statuses of the stylus are displayed on the display.

9. The method of claim 8, wherein the one or more status indicators include a battery level indicator.

10. The method of claim 8, wherein the one or more status indicators are displayed along a side of the display corresponding to a connection point on the electronic device that the stylus is contacting.

11. The method according to claim 8, further comprising: determining whether the status information indicates a prompt condition associated with the stylus; and In response to determining that the status information indicates the prompt condition, a prompt message indicating the prompt condition is displayed on the display. 12 . The method of claim 1 , wherein the first representation of the first gesture and the second representation of the second gesture are displayed within a tutorial interface, the tutorial interface comprising a canvas and a set of drawing tools.

13. The method according to claim 12, further comprising: moving a focus to a specific drawing tool in the set of drawing tools based on determining that the finger manipulation data corresponds to a first type; and Based on determining that the finger manipulation data corresponds to the second type, a property of a drawing tool currently having focus is changed.

14. The method according to claim 12, further comprising: In response to detecting a drawing input corresponding to the canvas, a corresponding indicator is displayed in the canvas according to a specific drawing tool having focus in the set of tools.

15. The method of claim 1, wherein in response to determining that the finger manipulation input satisfies gesture criteria, the second representation of the second gesture is displayed.

16. An electronic device, comprising: Touch-sensitive surfaces; monitor; one or more processors; non-transitory memory; a communication interface provided to communicate with the stylus; and one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: In response to detecting that the stylus is in proximity to the electronic device, pairing the electronic device with the stylus; and In response to pairing the stylus with the electronic device: displaying, on the display, a first representation of a first gesture performed on the stylus; obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data indicates finger manipulation input received by the stylus; and In response to obtaining the finger manipulation data, displaying on the display a second representation of a second gesture performed on the stylus corresponding to the finger manipulation input received by the stylus.

17. The electronic device according to claim 16, wherein: The one or more programs include additional instructions for: obtaining additional finger manipulation data from the stylus via the communication interface, wherein the additional finger manipulation data indicates a second finger manipulation input received by the stylus corresponding to movement of a finger on the stylus; and In response to obtaining the additional finger manipulation data, the second representation of the second gesture performed on the stylus is changed on the display according to the second finger manipulation input.

18. The electronic device according to claim 16, wherein: The one or more programs include additional instructions for: obtaining, from the stylus via the communication interface, state information regarding one or more states of the stylus; and In response to obtaining the status information, one or more status indicators indicating the one or more statuses of the stylus are displayed on the display.

19. The electronic device of claim 16, wherein the first representation of the first gesture and the second representation of the second gesture are displayed within a tutorial interface, the tutorial interface comprising a canvas and a set of drawing tools.

20. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device having one or more processors, a touch-sensitive surface, a display, and a communication interface provided to communicate with a stylus, cause the electronic device to: In response to detecting that the stylus is in proximity to the electronic device, pairing the electronic device with the stylus; and In response to pairing the stylus with the electronic device: displaying, on the display, a first representation of a first gesture performed on the stylus; obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data indicates finger manipulation input received by the stylus; and In response to obtaining the finger manipulation data, displaying on the display a second representation of a second gesture performed on the stylus corresponding to the finger manipulation input received by the stylus.

Images