Device, method and graphical user interface for electronic device interacting with stylus

Abstract

Devices, methods, and graphical user interfaces are provided for an electronic device that interacts with a stylus. In some embodiments, the application mode is changed in response to detecting that the stylus is being held. In some embodiments, the functionality of the stylus is changed based on the stylus grip arrangement. In some embodiments, the touch input function is changed when the stylus is being held. In some embodiments, the displayed identification is changed based on a finger manipulation input received by the stylus. In some embodiments, a finger manipulation input received by a stylus enables navigation within a menu. In some embodiments, an identification is created based on a hover distance of a stylus. In some embodiments, an interactive stylus tutorial is displayed. In some embodiments, stylus settings on an electronic device are maintained. In some embodiments, the stylus is picked up and then gestures are made thereon to change tool properties. In some embodiments, the stylus touch changes the active color.

Description

[0001] This application is a divisional application based on the Chinese invention patent application with application date of May 30, 2019, application number 2019800363133, and invention name “Device, method and graphical user interface for electronic device for interacting with stylus”. Technical Field

[0002] The present invention generally relates to electronic devices that interact with a stylus, including but not limited to a user interface on a display of the electronic device that is influenced by sensor data received from the stylus. Background Art

[0003] The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Examples of touch-sensitive surfaces include touchpads and touchscreen displays. These surfaces are widely used to manipulate user interfaces on displays.

[0004] However, touch input (including gesture input) provides limited and inefficient control for manipulating the user interface. Therefore, in order to achieve a specific goal, repeated, complex and / or cumbersome touch input may be required to manipulate the user interface. Summary of the Invention

[0005] Therefore, there is a need for a robust mechanism for manipulating a user interface of a display on an electronic device. Specifically, electronic devices need to have faster, more efficient methods and interfaces for manipulating user interfaces. Such methods and interfaces optionally supplement or replace conventional methods for manipulating user interfaces. Such methods and interfaces reduce the amount, extent, and / or nature of input from the user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0006] The disclosed devices and methods can reduce or eliminate the aforementioned drawbacks and other problems associated with user interfaces for electronic devices with touch-sensitive surfaces. In some embodiments, the electronic device is a desktop computer. In some embodiments, the electronic device is portable (e.g., a laptop, tablet, or handheld device). In some embodiments, the electronic device is a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the electronic device has a touchpad. In some embodiments, the electronic device has a touch-sensitive display (also known as a "touch screen" or "touchscreen display"). In some embodiments, the electronic device has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs or instruction sets stored in the memory for performing multiple functions. In some embodiments, a user interacts with the GUI primarily through stylus and / or finger contacts and gestures on the touch-sensitive surface. In some embodiments, when the stylus is not in physical contact with the touch-sensitive surface, the user interacts with the GUI primarily through user interaction with the stylus. In some embodiments, when the user is holding the stylus, the user interacts with the GUI primarily through finger and / or hand contacts and gestures on the stylus. In some embodiments, these functions optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] According to some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes obtaining, at the electronic device via the communication interface, information about a current state of the stylus when the electronic device is in a first state. The method also includes displaying a visual indication on the display that the electronic device is in a second state different from the first state based on a determination that a user is holding the stylus based on the information about the current state of the stylus. The method also includes maintaining the electronic device in the first state based on a determination that the user is not holding the stylus.

[0008] In accordance with some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes detecting input from a stylus on the touch-sensitive surface of the electronic device. The method also includes, in response to detecting the input and based on a determination that the stylus is being held in a first grip arrangement, making a first change to content displayed on the display, wherein the first grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus. The method also includes, in response to detecting the input and based on a determination that the stylus is being held in a second grip arrangement different from the first grip arrangement, making a second change to the content displayed on the display, wherein the second grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus, wherein the second change to the content displayed on the display is different from the first change to the content displayed on the display.

[0009] According to some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes detecting a touch input on the touch-sensitive surface. The method also includes, in response to detecting the touch input on the touch-sensitive surface and based on determining that sensor data obtained from the stylus via the communication interface indicates that the stylus is being held by a user, performing a first operation in response to the touch input. The method also includes, in response to detecting the touch input on the touch-sensitive surface and based on determining that the stylus is not being held by the user, performing a second operation in response to the touch input, wherein the second operation is different from the first operation.

[0010] According to some embodiments, a method is performed at an electronic device having one or more processors, a non-volatile memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes: when displaying multiple user interface elements on the display, obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data includes information about one or more finger manipulation inputs received by the stylus. The method also includes: in response to obtaining the finger manipulation data and based on determining that the finger manipulation data indicates a first finger manipulation input on the stylus, performing a first operation on at least a subset of the multiple user interface elements. The method also includes: in response to obtaining the finger manipulation data and based on determining that the finger manipulation data indicates a second finger manipulation input on the stylus that is different from the first finger manipulation input, performing a second operation on at least a subset of the multiple user interface elements, wherein the second operation is different from the first operation.

[0011] According to some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes displaying a selection user interface on the display including a plurality of selectable items, wherein a first item of the plurality of selectable items is currently selected within the selection user interface. The method also includes obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data includes information about one or more finger manipulation inputs received at the stylus. The method also includes: in response to obtaining the finger manipulation data and based on determining that the finger manipulation data meets a navigation criterion, changing the display of the selection user interface to indicate that the focus moves to a second item of the plurality of selectable items. The method also includes: in response to obtaining the finger manipulation data and based on determining that the finger manipulation data does not meet the navigation criterion, maintaining the display of the selection user interface, wherein the first item of the plurality of selectable items currently has focus within the selection user interface.

[0012] According to some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes obtaining input data corresponding to an input detected at the stylus from the stylus via the communication interface. The method also includes: in response to obtaining the input data from the stylus and based on determining that a distance between the stylus and the touch-sensitive display when the input was detected at the stylus satisfied a first distance threshold, displaying a first user interface element corresponding to the input. The method also includes: in response to obtaining the input data from the stylus and based on determining that the distance between the stylus and the touch-sensitive display when the input was detected at the stylus satisfied a second distance threshold, forgoing displaying the first user interface element corresponding to the input.

[0013] According to some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a display, a touch-sensitive surface, and a communication interface provided for communicating with a stylus. The method includes: in response to detecting that the stylus is in proximity to the electronic device, pairing the electronic device with the stylus. The method includes: in response to pairing the stylus with the electronic device: displaying a first representation of a first gesture performed on the stylus on the display; obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data indicates finger manipulation input received by the stylus; and in response to obtaining the finger manipulation data, displaying a second representation of a second gesture performed on the stylus on the display that corresponds to the finger manipulation input received by the stylus.

[0014] According to some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus. The method includes detecting movement of the stylus on the touch-sensitive surface. The method includes: in response to detecting the movement of the stylus, performing a stylus operation in a user interface displayed on the display based on the movement of the stylus. The method includes: after performing the stylus operation in the user interface, obtaining finger manipulation data indicating finger manipulation input received at the stylus via the communication interface. The method includes: in response to obtaining the finger manipulation data from the stylus: changing an attribute of the stylus operation in the user interface based on the finger manipulation input; and displaying a visual indication of the change in the attribute of the stylus operation on the display of the electronic device.

[0015] According to some embodiments, a method is performed at a first electronic device having one or more processors, non-volatile memory, a display, and a communication interface provided for communicating with a stylus. The method includes detecting input corresponding to the stylus, the stylus communicating with the first electronic device via the communication interface. The method includes: in response to detecting the input corresponding to the stylus: performing a first operation at the first electronic device based on determining that a first setting of the stylus has a first value; and performing a second operation at the first electronic device that is different from the first operation based on determining that the first setting of the stylus has a second value different from the first value, wherein the value of the first setting is determined based on input at a second electronic device with which the stylus previously communicated.

[0016] In accordance with some embodiments, a method is performed at an electronic device having one or more processors, non-volatile memory, a touch-sensitive surface, and a display. The method includes detecting a first input on the touch-sensitive surface corresponding to a color selection affordance selected by a user. The method includes: in response to detecting the first input, displaying a color picker user interface on the display, wherein the color picker user interface includes a plurality of options for selecting a user-selected color. The method includes detecting a second input on the touch-sensitive surface corresponding to a particular option in the plurality of options for selecting the user-selected color. The method includes, in response to detecting the second input: designating a first color selected based on the particular option in the plurality of options for selecting the user-selected color as an active color; upon detecting an end of the second input, upon determining that the second input is a continuation of the first input, ceasing to display the color picker user interface; and upon determining that the second input is detected after the end of the first input and while the color picker user interface continues to be displayed on the display, maintaining display of the color picker user interface after detecting the end of the second input.

[0017] In accordance with some embodiments, an electronic device includes a touch-sensitive surface, a display, a communication interface provided for communicating with a stylus, one or more processors, a memory, and one or more programs; the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. In accordance with some embodiments, a computer-readable storage medium stores therein instructions that, when executed by an electronic device having a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus, cause the electronic device to perform or cause the performance of any of the methods described herein. In accordance with some embodiments, a graphical user interface on an electronic device having a touch-sensitive surface, a display, a communication interface provided for communicating with a stylus, a memory, and one or more processors for executing the one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, the elements being updated in response to input, as described in any of the methods described herein. In accordance with some embodiments, an electronic device includes a touch-sensitive surface, a display, a communication interface provided for communicating with a stylus, and means for performing or causing the performance of any of the methods described herein. According to some embodiments, an information processing apparatus for use in an electronic device having a touch-sensitive surface, a display, and a communication interface provided for communicating with a stylus comprises means for performing or causing performance of the operations of any of the methods described herein.

[0018] Thus, an electronic device having a touch-sensitive surface and a communication interface provided for communicating with a stylus utilizes data received from the stylus. The received data indicates user input detected at the stylus's sensor. The stylus's sensor can detect various user inputs and provide data indicating these inputs to the electronic device. Based on the received data, the electronic device performs various operations, such as drawing and navigation operations. Thus, the electronic device can perform various operations without receiving input at the touch-sensitive surface of the electronic device. This improves the functionality of the electronic device in many ways, including longer battery life and less wear and tear. In addition, the improved user interface enables more efficient and accurate user interaction with the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0020] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.

[0021] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments.

[0022] Figure 2 A portable multifunction device with a touch screen is shown in accordance with some embodiments.

[0023] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.

[0024] Figure 4 is a block diagram of an exemplary electronic stylus in accordance with some embodiments.

[0025] Figures 5A to 5B The positional state of a stylus relative to a touch-sensitive surface is shown in accordance with some embodiments.

[0026] Figure 6A An exemplary user interface for a menu of applications on a portable multifunction device is shown in accordance with some embodiments.

[0027] Figure 6B An exemplary user interface for a multifunction device having a touch-sensitive surface that is separate from the display is shown in accordance with some embodiments.

[0028] Figures 7A to 7Y An exemplary user interface for changing application state is shown in accordance with some embodiments.

[0029] Figures 8A to 8H An exemplary user interface for changing stylus functionality is shown in accordance with some embodiments.

[0030] Figures 9A to 9P An exemplary user interface for modifying touch input functionality is shown in accordance with some embodiments.

[0031] 10A to 10I An exemplary user interface for performing operations on existing identifiers based on finger manipulation input is shown in accordance with some embodiments.

[0032] 11A to 11O An exemplary user interface for performing finger manipulations of a stylus to navigate within a menu is shown in accordance with some embodiments.

[0033] 12A to 12O An exemplary user interface for displaying user interface elements based on a hover distance of a stylus in accordance with some embodiments is shown.

[0034] Figure 13A is a flow chart illustrating a method of processing sensor data collected at a stylus according to some embodiments.

[0035] Figure 13B is a flow chart illustrating another method of processing sensor data collected at a stylus in accordance with some embodiments.

[0036] 14A to 14C is a flow chart illustrating a method of changing application state according to some embodiments.

[0037] Figures 15A to 15B is a flow chart illustrating a method of changing stylus functionality according to some embodiments.

[0038] Figures 16A to 16B is a flow chart illustrating a method of modifying touch input functionality according to some embodiments.

[0039] 17A to 17C is a flow chart illustrating a method of performing an operation on an existing identifier based on finger manipulation input according to some embodiments.

[0040] 18A to 18B is a flow chart illustrating a method of performing finger manipulations to a stylus to navigate within a menu in accordance with some embodiments.

[0041] 19A to 19C is a flow chart illustrating a method of displaying a user interface element based on a hover distance of a stylus in accordance with some embodiments.

[0042] Figures 20A to 20W is an illustration of an exemplary user interface that provides an interactive stylus tutorial in accordance with some embodiments.

[0043] Figures 21A to 21AB is an illustration of an exemplary user interface for selecting stylus settings and drawing a mark based on the stylus settings in accordance with some embodiments.

[0044] Figures 22A to 22G is an illustration of an exemplary user interface for maintaining stylus settings on an electronic device in accordance with some embodiments.

[0045] Figures 23A to 23Z is an illustration of an exemplary user interface including a color picker user interface for assigning an active color, in accordance with some embodiments.

[0046] Figures 24A to 24C is a flow chart illustrating a method of displaying an exemplary user interface that provides an interactive stylus tutorial in accordance with some embodiments.

[0047] FIG. 25A to FIG. 25B is a flow chart illustrating a method of displaying an exemplary user interface for selecting stylus settings and drawing a mark based on the stylus settings in accordance with some embodiments.

[0048] Figures 26A to 26Bis a flow chart illustrating a method of maintaining stylus settings on an electronic device in accordance with some embodiments.

[0049] Figures 27A to 27C is a flow chart illustrating a method of displaying an exemplary user interface including a color picker user interface for assigning an active color in accordance with some embodiments. DETAILED DESCRIPTION

[0050] Many electronic devices include touch-sensitive surfaces that allow users to manipulate user interfaces. For example, finger strokes on a touch-sensitive surface draw lines on a canvas in a drawing application. However, existing methods for manipulating user interfaces are slow, cumbersome, and inefficient. For example, the number of types of input that can occur at an electronic device in response to a surface touch is limited. Even multi-step surface touches (e.g., gestures) provide relatively few types of input because they still require touch contact with the screen.

[0051] The following embodiments address these issues by providing a separate tool (e.g., a stylus) that leverages the user's numerous and complex hand and finger manipulations. The hand and finger manipulations occur at the stylus and are provided to the electronic device. As a result, the user does not need to constrain their hands and fingers to the screen to manipulate the user interface, and thus can utilize a wider range of hand and finger manipulations. These manipulations provide the electronic device with a richer and more robust set of instructions than can be implemented using touchscreen input alone.

[0052] under, Figure 1A to Figure 1B 、 Figures 2 to 4 、 Figures 5A to 5B as well as Figures 6A to 6B A description of an exemplary apparatus is provided. Figures 7A to 7Y An exemplary user interface for changing application state is shown in accordance with some embodiments. Figures 7A to 7Y The user interface in 14A to 14C in the process. Figures 8A to 8H An exemplary user interface for changing stylus functionality is shown in accordance with some embodiments. Figures 8A to 8H The user interface in FIG. 15A to FIG. 15B in the process. Figures 9A to 9P An exemplary user interface for modifying touch input functionality is shown in accordance with some embodiments. Figures 9A to 9P The user interface in Figures 16A to 16B in the process. 10A to 10I An exemplary user interface for performing operations on existing identifiers based on finger manipulation input is shown in accordance with some embodiments. 10A to 10I The user interface in 17A to 17C in the process. 11A to 11O An exemplary user interface for performing finger manipulations of a stylus to navigate within a menu is shown in accordance with some embodiments. 11A to 11O The user interface in 18A to 18B in the process. 12A to 12O An exemplary user interface for displaying user interface elements based on a hover distance of a stylus in accordance with some embodiments is shown. Figures 12A to 12O The user interface in 19A to 19C in the process. Figures 20A to 20W An exemplary user interface providing an interactive stylus tutorial is shown in accordance with some embodiments. Figures 20A to 20W The user interface in Figures 24A to 24C in the process. Figures 21A to 21AB An exemplary user interface for selecting stylus settings and drawing a marker based on the stylus settings is shown in accordance with some embodiments. Figures 21A to 21AB The user interface in FIG. 25A to FIG. 25B in the process. Figures 22A to 22G An exemplary user interface for maintaining stylus settings on an electronic device is shown in accordance with some embodiments. Figures 22A to 22G The user interface in FIG. 26A to FIG. 26B in the process. Figures 23A to 23Z An exemplary user interface including a color picker user interface for assigning an active color is shown in accordance with some embodiments. Figures 23A to 23Z The user interface in Figures 27A to 27C in the process.

[0053] Exemplary devices

[0054] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0055] It will also be understood that, although in some cases, the terms "first," "second," etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first contact can be named a second contact, and similarly, a second contact can be named a first contact without departing from the scope of the various described embodiments. Both a first contact and a second contact are contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0056] The terms used in the description of the various described embodiments herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various described embodiments and in the appended claims, the singular forms "a" and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "includes," "including," "comprises," and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0057] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining" or "in response to determining" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0058] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the electronic device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player. Example embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads), are optionally used. It should also be understood that in some embodiments, the electronic device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).

[0059] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0060] The electronic device typically supports a variety of applications, such as one or more of the following: a note-taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0061] Various applications executed on the electronic device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the corresponding application. In this way, the common physical architecture of the electronic device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0062] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 in accordance with some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and sometimes simply as a touch-sensitive display. The electronic device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripherals interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and external ports 124. The electronic device 100 optionally includes one or more optical sensors 164. The electronic device 100 optionally includes one or more intensity sensors 165 for detecting intensity of contacts on the electronic device 100 (e.g., a touch-sensitive surface such as the touch-sensitive display system 112 of the electronic device 100). The electronic device 100 optionally includes one or more tactile output generators 163 for generating tactile output on the electronic device 100 (e.g., generating tactile output on a touch-sensitive surface such as the touch-sensitive display system 112 of the electronic device 100 or the touchpad 355 of the device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0063] As used in this specification and claims, the term "tactile output" refers to a physical displacement of an electronic device relative to a previous position of the electronic device, a physical displacement of a component of an electronic device (e.g., a touch-sensitive surface) relative to another component of the electronic device (e.g., a housing), or a displacement of the component relative to the center of mass of the electronic device that will be detected by a user using their sense of touch. For example, when an electronic device or a component of the electronic device is in contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the electronic device or a component of the electronic device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "release click," "press click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the electronic device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0064] It should be understood that electronic device 100 is only one example of a portable multifunction device and that electronic device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0065] The memory 102 optionally includes high-speed random access memory and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory 102 by other components of the electronic device 100 (such as the CPU 120 and the peripheral device interface 118) is optionally controlled by a memory controller 122.

[0066] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the electronic device to the memory 102 and one or more CPUs 120. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the electronic device 100 and process data.

[0067] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0068] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols and / or technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11d), IEEE 802.11d, ... 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed on the date of filing of this document.

[0069] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the electronic device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0070] The I / O subsystem 106 couples input / output peripherals on the electronic device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. The one or more buttons (e.g., Figure 2 208) optionally includes up / down buttons for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in the ).

[0071] The touch-sensitive display system 112 provides an input interface and an output interface between the electronic device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0072] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from the user based on tactile and / or haptic contact. The touch-sensitive display system 112 and the display controller 156 (together with any associated modules and / or instruction sets in the memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary embodiment, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0073] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In one exemplary embodiment, projected mutual capacitance sensing technology is used, such as the ATmega 200 from Apple Inc. (Cupertino, California). iPod and Technology found in.

[0074] The touch-sensitive display system 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or greater). The user optionally uses any suitable object or appendage, such as a stylus, finger, or the like, to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the electronic device converts the rough finger-based input into a precise pointer / cursor position or command for performing the action desired by the user.

[0075] In some embodiments, in addition to the touch screen, the electronic device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the electronic device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0076] The electronic device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0077] The electronic device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to the optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The one or more optical sensors 164 receive light projected from the environment through one or more lenses and convert the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the one or more optical sensors 164 optionally capture still images and / or video. In some embodiments, the optical sensor is located on the rear portion of the electronic device 100 opposite the touch-sensitive display system 112 on the front of the electronic device, so that the touch screen can be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the electronic device to obtain an image of the user (e.g., for selfies, for video conferencing when the user views other video conference participants on the touch screen, etc.).

[0078] The electronic device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of electronic device 100 opposite touch screen display system 112 located on the front of electronic device 100.

[0079] The electronic device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0080] The electronic device 100 optionally also includes one or more tactile output generators 163 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 163 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile outputs on an electronic device). The tactile output generator 163 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile outputs on the electronic device 100 that can be felt by a user of the electronic device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the electronic device 100) or laterally (e.g., back and forth in the same plane as the surface of the electronic device 100). In some embodiments, at least one tactile output generator sensor is located on the back of electronic device 100 opposite touch-sensitive display system 112 located on the front of electronic device 100 .

[0081] The electronic device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (e.g., as part of an inertial measurement unit (IMU)) for obtaining information about the position (e.g., attitude) of the electronic device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripherals interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on analysis of data received from the one or more accelerometers. Electronic device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the position and orientation (e.g., portrait or landscape) of electronic device 100.

[0082] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a position module (or instruction set) 131, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in Figures 1A and 1B. Figure 3. The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications, if any, are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, which includes information obtained from various sensors and other input or control devices 116 of the electronic device; and position and / or orientation information regarding the position and / or posture of the electronic device.

[0083] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0084] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are suitable for coupling directly to other devices or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external ports are compatible with some of the Apple Inc. (Cupertino, California) iPod and In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in Apple devices. iPod and A Lightning connector that is the same as, or similar to, and / or compatible with, the Lightning connector used in the device.

[0085] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, contact / motion module 130 and display controller 156 detect contact on a touchpad.

[0086] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0087] In conjunction with the accelerometer 167, the gyroscope 168, and / or the magnetometer 169, the position module 131 optionally detects position information about the electronic device, such as the attitude (e.g., roll, pitch, and / or yaw) of the electronic device in a particular reference frame. The position module 130 includes software components for performing various operations related to detecting the position of the electronic device and detecting changes in the position of the electronic device. In some embodiments, the position module 131 uses information received from a stylus used with the electronic device to detect position information about the stylus, such as detecting the position state of the stylus relative to the electronic device and detecting changes in the position state of the stylus.

[0088] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0089] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program, etc., along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0090] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 163 to produce tactile output at one or more locations on electronic device 100 in response to user interaction with electronic device 100 .

[0091] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0092] The GPS module 135 determines the location of the electronic device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as a weather widget, a local yellow pages widget, and a map / navigation widget).

[0093] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0094] Contacts module 137 (sometimes called address book or contact list);

[0095] Telephone module 138;

[0096] Video conferencing module 139;

[0097] Email client module 140;

[0098] Instant messaging (IM) module 141;

[0099] Fitness support module 142;

[0100] A camera module 143 for still and / or video images;

[0101] Image management module 144;

[0102] Browser module 147;

[0103] Calendar module 148;

[0104] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, and user-created widgets 149-6;

[0105] A widget creator module 150 for forming user-created widgets 149-6;

[0106] Search module 151;

[0107] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0108] Memo module 153;

[0109] Map module 154;

[0110] Online video module 155; and / or

[0111] • An annotation application 195 for providing annotations to the user interface and optionally storing and / or accessing saved annotations 196 in memory 102 .

[0112] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0113] In combination with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and so on.

[0114] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing the corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0115] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and phone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0116] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0117] In conjunction with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, sending a corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving an instant message, and viewing a received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0118] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring a workout; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0119] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0120] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0121] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching for, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0122] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0123] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0124] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page into a desktop widget).

[0125] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0126] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, electronic device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0127] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notes module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0128] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving directions; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0129] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0130] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0131] In some embodiments, the electronic device 100 is an electronic device in which the operation of a predefined set of functions on the electronic device is performed exclusively through a touch screen and / or a touch pad. By using the touch screen and / or the touch pad as the primary input control device for operating the electronic device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the electronic device 100 is optionally reduced.

[0132] A predefined set of functions that are exclusively performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the electronic device 100 from any user interface displayed on the electronic device 100 to a main menu, home menu, or root menu. In such embodiments, the trackpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0133] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0134] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192 that indicates one or more current application views displayed on touch-sensitive display system 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0135] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user.

[0136] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (via audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0137] In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for a predetermined duration).

[0138] In some embodiments, event classifier 170 also includes a hit view determination module 172 and / or an active event recognizer determination module 173. When touch-sensitive display system 112 displays more than one view, hit view determination module 172 provides software procedures for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that a user can see on the display.

[0139] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0140] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0141] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to the area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0142] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0143] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0144] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 is part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0145] Respective event identifiers 180 receive event information (e.g., event data 179) from event classifier 170 and identify events from the event information. Event identifiers 180 include event receivers 182 and event comparators 184. In some embodiments, event identifiers 180 also include metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0146] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the electronic device rotating from one orientation to another orientation (for example, from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the electronic device (also referred to as the device posture).

[0147] Event comparator 184 compares event information with predefined event or sub-event definitions and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes definitions of events (e.g., predefined sub-event sequences), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0148] In some embodiments, event definitions 187 include definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0149] In some embodiments, the definition of the corresponding event 187 also includes delay actions that delay the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0150] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0151] In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0152] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, event recognizer 180 throws an identifier associated with the identified event, and the event handler 190 associated with the identifier obtains the identifier and performs a predefined process.

[0153] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined process.

[0154] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0155] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0156] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a trackpad, such as taps, drags, scrolls, etc.; stylus input; movement of an electronic device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0157] Figure 2 A device with a touch screen (e.g., Figure 1AThe portable multifunction device 100 includes a touch-sensitive display system 112 (e.g., a touchscreen). The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, as well as other embodiments described below, a user can select one or more of the graphics by, for example, gesturing on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up, and / or down), and / or a rolling motion of a finger (from right to left, from left to right, up, and / or down) that has made contact with the electronic device 100. In some embodiments or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon may not select the corresponding application when the gesture corresponding to selection is a tap.

[0158] Stylus 203 includes a first end 276 and a second end 277. In various embodiments, first end 276 corresponds to the tip of stylus 203 (e.g., the tip of a pencil), and second end 277 corresponds to the opposite end or base of stylus 203 (e.g., the eraser of a pencil).

[0159] Stylus 203 includes a touch-sensitive surface 275 for receiving touch input from a user. In some embodiments, touch-sensitive surface 275 corresponds to a capacitive touch element. Stylus 203 includes a sensor or sensor array that detects input from the user based on tactile and / or haptic contact with touch-sensitive surface 275. In some embodiments, stylus 203 includes any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as an array of other proximity sensors or other elements for determining one or more points of contact with touch-sensitive surface 275. Because stylus 203 includes a variety of sensors and sensor types, stylus 203 can detect a variety of different inputs from the user, including gestures disclosed herein with respect to the touch screen of portable multifunction device 100. In some embodiments, one or more sensors can detect a single touch input or a continuous touch input in response to a user tapping one or more times on touch-sensitive surface 275. In some embodiments, one or more sensors can detect a swipe input on stylus 203 in response to a user swiping one or more fingers along touch-sensitive surface 275. In some embodiments, if the speed at which the user swipes along touch-sensitive surface 275 violates a threshold, one or more sensors detect a flick input instead of a swipe input.

[0160] Stylus 203 also includes one or more sensors, such as an accelerometer, magnetometer, gyroscope, and the like, that detect the orientation (e.g., angular position relative to the electronic device) and / or movement of stylus 203. These one or more sensors can detect various rotational movements of stylus 203 by the user, including the type and direction of rotation. For example, these one or more sensors can detect when the user rolls and / or rotates stylus 203, and can detect the direction of the roll / rotation (e.g., clockwise or counterclockwise). In some embodiments, the detected input depends on the angular position of first end 276 and second end 277 of stylus 203 relative to the electronic device. For example, in some embodiments, if stylus 203 is substantially perpendicular to the electronic device and second end 277 (e.g., the eraser) is closer to the electronic device, contacting the surface of the electronic device with second end 277 results in an erase operation. On the other hand, if stylus 203 is substantially perpendicular to the electronic device and first end 276 (e.g., the tip) is closer to the electronic device, contacting the surface of the electronic device with first end 276 results in a marking operation.

[0161] The electronic device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the electronic device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0162] In some embodiments, the electronic device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the electronic device on / off and for locking the electronic device, a volume adjustment button 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the electronic device on / off by pressing the button and holding it in the depressed state for a predefined time interval; to lock the electronic device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the electronic device or initiate an unlocking process. In some embodiments, the electronic device 100 also accepts voice input for activating or deactivating certain functions via a microphone 113. The electronic device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112 and / or one or more tactile output generators 163 for generating tactile output for a user of the electronic device 100.

[0163] Figure 3is a block diagram of an exemplary multifunction device 300 with a display and a touch-sensitive surface according to some embodiments. The electronic device 300 does not have to be portable. In some embodiments, the electronic device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The electronic device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the electronic device 300 (e.g., similar to the above referenced device). Figure 1A The tactile output generator 163), sensor 359 (e.g., similar to the tactile output generator 163 described above), Figure 1A The memory 370 may include touch sensors, optical sensors, contact intensity sensors, proximity sensors, acceleration sensors, gesture sensors, and / or magnetic sensors of the sensors 112, 164, 165, 166, 167, 168, and 169. The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100( Figure 1A )'s memory 102 optionally does not store these modules.

[0164] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In certain embodiments, memory 370 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0165] Figure 4 is a block diagram of an exemplary electronic stylus 203 according to some embodiments. The electronic stylus 203 is sometimes simply referred to as a stylus. The stylus 203 includes a memory 402 (which optionally includes one or more computer-readable storage media), a memory controller 422, one or more processing units (CPUs) 420, a peripheral device interface 418, RF circuitry 408, an input / output (I / O) subsystem 406, and other input or control devices 416. The stylus 203 optionally includes an external port 424 and one or more optical sensors 464. The stylus 203 optionally includes one or more intensity sensors 465 for detecting the intensity of contact of the stylus 203 on the device 100 (e.g., when the stylus 203 is used with a touch-sensitive surface such as the touch-sensitive display system 112 of the electronic device 100) or on other surfaces (e.g., a table surface). The stylus 203 optionally includes one or more tactile output generators 463 for generating tactile output on the stylus 203. These components optionally communicate via one or more communication buses or signal lines 403.

[0166] In some embodiments, the term "tactile output" discussed above refers to a physical displacement of an accessory (e.g., stylus 203) of an electronic device (e.g., electronic device 100) relative to a previous position of the accessory, a physical displacement of a component of the accessory relative to another component of the accessory, or a displacement of the component relative to the center of mass of the accessory, which is detected by a user using their sense of touch. For example, when an accessory or a component of the accessory comes into contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the accessory or component of the accessory. For example, movement of a component (e.g., the housing of stylus 203) may be interpreted by the user as a "click" of a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "click," even when the physical actuation button associated with the stylus, which is physically pressed (e.g., displaced) by the user's movement, does not move. While such user interpretations of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., a “click”), unless otherwise stated, the tactile output generated corresponds to a physical displacement of the electronic device or a component thereof that would generate that sensory perception in a typical (or average) user.

[0167] It should be understood that stylus 203 is merely one example of an electronic stylus, and that stylus 203 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 4 The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0168] Memory 402 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more flash memory devices or other non-volatile solid-state memory devices. Access to memory 402 by other components of stylus 203 (such as one or more CPUs 420 and peripheral device interface 418) is optionally controlled by memory controller 422.

[0169] Peripheral interface 418 can be used to couple the input and output peripherals of the stylus to one or more CPUs 420 and memory 402. The one or more processors 420 run or execute various software programs and / or instruction sets stored in memory 402 to perform various functions of stylus 203 and process data.

[0170] In some embodiments, peripherals interface 418, CPU 420, and memory controller 422 are optionally implemented on a single chip, such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0171] RF (radio frequency) circuitry 408 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 408 converts electrical signals into / from electromagnetic signals and communicates with electronic device 100 or 300, a communication network, and / or other communication devices via the electromagnetic signals. RF circuitry 408 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 408 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols and / or technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed on the date of filing of this document.

[0172] The I / O subsystem 406 couples input / output peripherals on the stylus 203, such as other input or control devices 416, to a peripheral device interface 418. The I / O subsystem 406 optionally includes an optical sensor controller 458, an intensity sensor controller 459, a tactile feedback controller 461, and one or more input controllers 460 for other input or control devices. The one or more input controllers 460 receive / send electrical signals from / to other input or control devices 416. Other input or control devices 416 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, click wheels, etc. In some alternative embodiments, the one or more input controllers 460 are optionally coupled to any (or none) of the following: an infrared port and / or a USB port.

[0173] Stylus 203 also includes a power system 462 for powering various components. Power system 462 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device and / or portable accessory.

[0174] Stylus 203 optionally also includes one or more optical sensors 464 . Figure 4 An optical sensor is shown coupled to an optical sensor controller 458 in the I / O subsystem 406. The optical sensor 464 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 464 receives light from the environment projected through one or more lenses and converts the light into data representing an image.

[0175] Stylus 203 optionally also includes one or more contact intensity sensors 465 . Figure 4 A contact force sensor is shown coupled to force sensor controller 459 in I / O subsystem 406. One or more contact force sensors 465 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a surface). Contact force sensor 465 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to the tip of stylus 203.

[0176] Stylus 203 optionally also includes one or more proximity sensors 466 . Figure 4 A proximity sensor 466 is shown coupled to peripherals interface 418. Alternatively, proximity sensor 466 is coupled to input controller 460 in I / O subsystem 406. In some embodiments, the proximity sensor determines the proximity of stylus 203 to an electronic device (eg, electronic device 100).

[0177] Stylus 203 optionally also includes one or more tactile output generators 463 . Figure 4 A tactile output generator is shown coupled to a tactile feedback controller 461 in the I / O subsystem 406. The tactile output generator 463 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or an electromechanical device that converts energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on an electronic device). One or more tactile output generators 463 receive tactile feedback generation instructions from the tactile feedback module 433 and generate tactile outputs on the stylus 203 that can be felt by the user of the stylus 203. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to the length of the stylus 203 (e.g., the body or housing) and optionally generates tactile outputs by moving the stylus 203 vertically (e.g., in a direction parallel to the length of the stylus 203) or laterally (e.g., in a direction perpendicular to the length of the stylus 203).

[0178] Stylus 203 optionally also includes one or more accelerometers 467 , gyroscopes 468 , and / or magnetometers 469 for obtaining information about the position and positional state of stylus 203 (eg, as part of an inertial measurement unit (IMU)). Figure 4 Sensors 467, 468, and 469 are shown coupled to peripherals interface 418. Alternatively, sensors 467, 468, and 469 are optionally coupled to input controller 460 in I / O subsystem 406. Stylus 203 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of stylus 203.

[0179] Stylus 203 includes a touch-sensitive system 432. Touch-sensitive system 432 detects inputs received at touch-sensitive surface 275. These inputs include the inputs discussed herein with respect to touch-sensitive surface 275 of stylus 203. For example, touch-sensitive system 432 can detect tap, rotation, roll, flick, and swipe inputs. Touch-sensitive system 432 cooperates with touch interpretation module 477 to interpret specific types of touch inputs (e.g., rotation / roll / flick / swipe / etc.) received at touch-sensitive surface 275.

[0180] In some embodiments, the software components stored in memory 402 include an operating system 426, a communication module (or instruction set) 428, a contact / motion module (or instruction set) 430, a location module (or instruction set) 431, and a global positioning system (GPS) module (or instruction set) 435. In addition, in some embodiments, memory 402 stores device / global internal state 457, such as Figure 4 In addition, although not shown, the memory 402 includes a touch interpretation module 477. The device / global internal state 457 includes one or more of the following: sensor state, including information obtained from the various sensors and other input or control devices 416 of the stylus; position state, including information about the position of the stylus relative to the electronic device (e.g., electronic device 100) (e.g., position, orientation, tilt, roll, and / or distance, as shown); Figure 5A and Figure 5B and location information about the stylus location (eg, determined by GPS module 435).

[0181] The operating system 426 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.) and facilitating communication between various hardware components and software components.

[0182] The communication module 428 optionally facilitates communication with other devices via one or more external ports 424 and also includes various software components for processing data received by the RF circuit 408 and / or the external ports 424. The external ports 424 (e.g., Universal Serial Bus (USB), FireWire, etc.) are suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external ports are compatible with some of the Apple Inc. (Cupertino, California) iPod and A Lightning connector that is the same as, or similar to, and / or compatible with, the Lightning connector used in the device.

[0183] The contact / motion module 430 optionally detects contact with the stylus 203 and other touch-sensitive devices of the stylus 203 (e.g., buttons or other touch-sensitive components of the stylus 203). The contact / motion module 430 includes software components for performing various operations related to detecting contact (e.g., detecting contact of the tip of the stylus with a touch-sensitive display such as the touch screen 112 of the electronic device 100 or with another surface such as a table surface), such as determining whether contact has occurred (e.g., detecting a touch down event), determining the intensity of the contact (e.g., the force or pressure of the contact or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking that movement across (e.g., across the touch screen 112 of the electronic device 100), and determining whether the contact has ceased (e.g., detecting a lift-off event or contact break). In some embodiments, the contact / motion module 430 receives contact data from the I / O subsystem 406. Determining movement of the contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a sequence of contact data. As described above, in some embodiments, one or more of these operations related to detection of contact are performed by the electronic device using contact / motion module 130 (in addition to or in lieu of a stylus using contact / motion module 430).

[0184] Contact / motion module 430 optionally detects gesture input from stylus 203. Different gestures performed with stylus 203 have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single tap gesture includes detecting a touch down event, followed by detecting a lift-off event at the same location (or substantially the same location) as the touch down event (e.g., at the location of an icon). As another example, detecting a swipe gesture includes detecting a touch down event, followed by detecting one or more stylus drag events, and then followed by detecting a lift-off event. As described above, in some embodiments, gesture detection is performed by the electronic device using contact / motion module 130 (in addition to or in lieu of using contact / motion module 430 with a stylus).

[0185] In conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469, position module 431 optionally detects position information about the stylus, such as the stylus's attitude (roll, pitch, and / or yaw) in a particular reference frame. In conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469, position module 431 optionally detects stylus movement gestures, such as flicks, taps, and rotations of the stylus. Position module 431 includes software components for performing various operations related to detecting the position of the stylus and detecting changes in the position of the stylus in a particular reference frame. In some embodiments, position module 431 detects the positional state of the stylus relative to the electronic device and detects changes in the positional state of the stylus relative to the electronic device. As described above, in some embodiments, electronic device 100 or electronic device 300 uses position module 131 to determine the positional state of the stylus relative to the electronic device and changes in the positional state of the stylus (in addition to or in lieu of using position module 431 for the stylus).

[0186] Haptic feedback module 433 includes various software components for generating instructions used by one or more tactile output generators 463 to produce tactile output at one or more locations on stylus 203 in response to user interaction with stylus 203 .

[0187] GPS module 435 determines the location of the stylus and provides this information for use in various applications (e.g., to applications that provide location-based services, such as applications for finding lost devices and / or accessories).

[0188] Touch interpretation module 477 cooperates with touch-sensitive system 432 to determine (e.g., interpret or recognize) the type of touch input received at touch-sensitive surface 275 of stylus 203. For example, if a user swipes lightly across touch-sensitive surface 275 a sufficient distance in a sufficiently short amount of time, touch interpretation module 477 determines that the touch input corresponds to a swipe input (rather than a tap input). For another example, if a user swipes across touch-sensitive surface 275 at a speed sufficiently faster than a voice corresponding to a swipe input, touch interpretation module 477 determines that the touch input corresponds to a flick input (rather than a swipe input). The threshold speed for a swipe can be preset and can be changed. In various embodiments, the pressure and / or force of a touch received at the touch-sensitive surface determines the type of input. For example, a light touch can correspond to a first type of input, while a firmer touch can correspond to a second type of input.

[0189] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 402 optionally stores a subset of the above modules and data structures. In addition, memory 402 optionally stores additional modules and data structures not described above.

[0190] Figures 5A to 5B 1 shows a positional state of the stylus 203 relative to a touch-sensitive surface (e.g., touch screen 112 of electronic device 100) according to some embodiments. In some embodiments, the positional state of the stylus 203 corresponds to (or indicates): the projected position of the tip (or other representative portion) of the stylus on the touch-sensitive surface (e.g., Figure 5A (x,y) position 504 in FIG), the orientation of the stylus relative to the touch-sensitive surface (e.g., Figure 5A orientation 506 in the stylus), the tilt of the stylus relative to the touch-sensitive surface (e.g., Figure 5B 512) and / or the distance of the stylus relative to the touch-sensitive surface (e.g., Figure 5B In some embodiments, the position state of stylus 203 corresponds to (or indicates) the pitch, yaw, and / or roll of the stylus (e.g., the attitude of the stylus relative to a particular reference frame such as a touch-sensitive surface (e.g., touch screen 112) or the ground). In some embodiments, the position state includes a set of position parameters (e.g., one or more position parameters). In some embodiments, the position state is detected based on one or more measurements from stylus 203 sent to an electronic device (e.g., electronic device 100). For example, the stylus measures the tilt of the stylus (e.g., Figure 5B 512) and / or orientation (e.g., Figure 5A The position state is detected based on raw output from one or more electrodes in the stylus as sensed by a touch-sensitive surface (e.g., touch screen 112 of electronic device 100), instead of or in combination with the position state detected based on one or more measurements from stylus 203. For example, the touch-sensitive surface receives raw output from one or more electrodes in the stylus and calculates the tilt and / or orientation of the stylus based on the raw output (optionally in combination with position state information provided by the stylus based on sensor measurements generated by the stylus).

[0191] Figure 5A1 shows the stylus 203 relative to a touch-sensitive surface (e.g., touch screen 112 of electronic device 100) from a perspective directly above the touch-sensitive surface in accordance with some embodiments. Figure 5A , the z-axis 594 points out of the page (i.e., in a direction perpendicular to the plane of the touch screen 112), the x-axis 590 is parallel to a first edge of the touch screen 112 (e.g., length), the y-axis 592 is parallel to a second edge of the touch screen 112 (e.g., width), and the y-axis 592 is perpendicular to the x-axis 590.

[0192] Figure 5A The tip of stylus 203 is shown at (x,y) position 504. In some embodiments, the tip of stylus 203 is the end of the stylus that is configured to determine the proximity of the stylus to a touch-sensitive surface (e.g., touch screen 112). In some embodiments, the projection of the tip of the stylus onto the touch-sensitive surface is an orthographic projection. In other words, the projection of the tip of the stylus onto the touch-sensitive surface is the point at the end of a line from the tip of the stylus to the touch-sensitive surface that is perpendicular to the surface of the touch-sensitive surface (e.g., (x,y) position 504, where the tip of the stylus would touch the touch-sensitive surface if the stylus were moved directly along a path perpendicular to the touch-sensitive surface). In some embodiments, the (x,y) position at the lower left corner of touch screen 112 is position (0,0) (e.g., (0,0) position 502) and other (x,y) positions on touch screen 112 are relative to the lower left corner of touch screen 112. Alternatively, in some embodiments, the (0,0) position is located at another location of the touch screen 112 (eg, at the center of the touch screen 112 ) and the other (x,y) positions are relative to the (0,0) position of the touch screen 112 .

[0193] in addition, Figure 5A Stylus 203 is shown having orientation 506. In some embodiments, orientation 506 is the orientation of a projection of stylus 203 on touch screen 112 (e.g., an orthogonal projection of the length of stylus 203 or a line corresponding to a line between the projections of two different points of stylus 203 on touch screen 112). In some embodiments, orientation 506 is relative to at least one axis in a plane parallel to touch screen 112. In some embodiments, orientation 506 is relative to a single axis in a plane parallel to touch screen 112 (e.g., axis 508, with clockwise rotation angles ranging from 0 degrees to 360 degrees starting from axis 508, as shown in FIG. 1 ). Figure 5A Alternatively, in some embodiments, the orientation 506 is relative to a pair of axes in a plane parallel to the touch screen 112 (e.g., an x-axis 590 and a y-axis 592, as shown). Figure 5A as shown, or a pair of axes associated with an application displayed on the touch screen 112).

[0194] In some embodiments, an indication (e.g., indication 516) is displayed on a touch-sensitive display (e.g., touch screen 112 of electronic device 100). In some embodiments, indication 516 shows where the stylus will touch (or mark) the touch-sensitive display before the stylus touches the touch-sensitive display. In some embodiments, indication 516 is part of the mark drawn on the touch-sensitive display. In some embodiments, indication 516 is separate from the mark drawn on the touch-sensitive display and corresponds to a virtual "pen tip" or other element indicating where the mark will be drawn on the touch-sensitive display.

[0195] In some embodiments, indication 516 is displayed based on the position state of stylus 203. For example, in some cases, indication 516 is displaced from (x, y) position 504 (e.g., Figure 5A and Figure 5B In some embodiments, the indication 516 is displayed with varying colors, sizes (or radius or area), opacity, and / or other characteristics depending on the positional state of the stylus. In some embodiments, the displayed indication takes into account the thickness of the glass layer on the touch-sensitive display so that the indication is carried through to the "pixels" of the touch-sensitive display rather than being displayed on the "glass" covering the pixels.

[0196] Figure 5B 1 shows a stylus 203 relative to a touch-sensitive surface (e.g., touch screen 112 of electronic device 100) viewed from a side perspective of the touch-sensitive surface in accordance with some embodiments. Figure 5B , the z-axis 594 points in a direction perpendicular to the plane of the touch screen 112, the x-axis 590 is parallel to a first edge (e.g., length) of the touch screen 112, the y-axis 592 is parallel to a second edge (e.g., width) of the touch screen 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0197] Figure 5B Stylus 203 is shown with tilt 512. In some embodiments, tilt 512 is an angle relative to a normal to the surface of the touch-sensitive surface (also referred to simply as the normal to the touch-sensitive surface) (e.g., normal 510). Figure 5B As shown, tilt 512 is zero when the stylus is perpendicular / perpendicular to the touch-sensitive surface (eg, when stylus 203 is parallel to normal 510 ), and increases as the stylus tilts closer to being parallel to the touch-sensitive surface.

[0198] in addition, Figure 5BThe distance 514 of the stylus 203 relative to the touch-sensitive surface is shown. In some embodiments, the distance 514 is the distance from the tip of the stylus 203 to the touch-sensitive surface in a direction perpendicular to the touch-sensitive surface. For example, Figure 5B , distance 514 is the distance from the tip of stylus 203 to (x,y) position 504 .

[0199] Although the terms "x-axis," "y-axis," and "z-axis" are used herein to illustrate certain directions in certain drawings, it should be understood that these terms do not refer to absolute directions. In other words, the "x-axis" can be any corresponding axis, and the "y-axis" can be a specific axis different from the x-axis. Typically, the x-axis is perpendicular to the y-axis. Similarly, the "z-axis" is different from the "x-axis" and the "y-axis" and is typically perpendicular to both the "x-axis" and the "y-axis."

[0200] in addition, Figure 5B A rotation 518 is shown—a rotation about the length (major axis) of the stylus 203 .

[0201] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0202] Figure 6A An exemplary user interface for a menu of applications on portable multifunction device 100 is shown in accordance with some embodiments. A similar user interface is optionally implemented on electronic device 300. In some embodiments, user interface 600 includes the following elements, or a subset or superset thereof:

[0203] One or more signal strength indicators 602 of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0204] Time 604;

[0205] Bluetooth indicator 605;

[0206] Battery status indicator 606;

[0207] A tray 608 with icons for commonly used applications, such as:

[0208] o An icon 616 labeled "Phone" for the phone module 138, which optionally includes an indicator 614 of the number of missed calls or voice messages;

[0209] o An icon 618 of the email client module 140 labeled "Mail," which optionally includes an indicator 610 of the number of unread emails;

[0210] o An icon 620 labeled "Browser" for the browser module 147; and

[0211] ○ Video and music player module 152 (also known as (trademark of Apple Inc.) module 152) labeled “iPod” icon 622; and

[0212] Icons for other apps, such as:

[0213] o Icon 624 labeled “Message” of IM module 141;

[0214] o Icon 626 labeled “Calendar” of calendar module 148;

[0215] o Icon 628 labeled “Photos” of the image management module 144;

[0216] o An icon 630 labeled “Camera” of the camera module 143;

[0217] o Icon 632 labeled “Video Editing” for the online video module 155;

[0218] ○ Icon 634 labeled “Stock Market” of the Stock Market desktop applet 149-2;

[0219] o An icon 636 labeled “Map” of the map module 154;

[0220] ○ Icon 638 labeled “Weather” of weather widget 149-1;

[0221] ○ Icon 640 labeled “Clock” of the alarm clock widget 149-4;

[0222] o An icon 642 labeled “Fitness Support” of the fitness support module 142;

[0223] o An icon 644 labeled "Notepad" of the Notepad module 153; and

[0224] o An icon 646 for a settings application or module that provides access to settings for the electronic device 100 and its various applications 136 .

[0225] It should be pointed out that Figure 6AThe icon labels shown are exemplary only. For example, in some embodiments, icon 622 for video and music player module 152 is labeled "Music" or "Music Player." Other labels are optionally used for various application icons. In some embodiments, the label for a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label for a particular application icon is different from the name of the application corresponding to the particular application icon.

[0226] Figure 6B A touch-sensitive surface 651 (e.g., Figure 3 electronic device (e.g., a tablet or touchpad 355) Figure 3 Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 651 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0227] Figure 6B A touch-sensitive surface 651 (e.g., Figure 3 electronic device (e.g., a tablet or touchpad 355) Figure 3 Although many of the examples that follow will be given with reference to input on the touch screen display 112 (where the touch-sensitive surface and the display are combined), in some embodiments, the electronic device 100 detects input on a touch-sensitive surface that is separate from the display, such as Figure 6B In some embodiments, the touch-sensitive surface (e.g., Figure 6B 651) has a main axis (e.g., Figure 6B 653) corresponding to the main axis (for example, Figure 6B According to these embodiments, the electronic device 100 detects contact with the touch-sensitive surface 651 at a location that corresponds to a corresponding location on the display (e.g., Figure 6B 660 and 662 in (e.g., in Figure 6B , 660 corresponds to 668 and 662 corresponds to 670). Thus, when a touch-sensitive surface (e.g., Figure 6B 651) and a display of a multi-function device (e.g., Figure 6B650 in the figure) are separated, the user input detected by the electronic device on the touch-sensitive surface (e.g., contacts 660 and 662 and their movement) is used by the electronic device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0228] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger contact, finger tap gestures, finger swipe gestures, etc.) and / or stylus input, it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting the contact, followed by ceasing to detect the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or that a mouse and finger contact (or stylus contact) are optionally used simultaneously.

[0229] User interface and associated processes

[0230] Attention is now directed to an electronic device such as the portable multifunction device 100 or the like in FIG. Figure 3 An embodiment of a user interface ("UI") and associated processes implemented on an electronic device 300 in FIG. 1 having a touch-sensitive display and optionally one or more sensors for detecting signals from a stylus associated with the electronic device.

[0231] Figures 7A to 7Y Figures 1 and 2 illustrate exemplary user interfaces for changing application states according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including 14A to 14C Although some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and display are combined, such as on touch screen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 that is separate from display 650, such as Figure 6B In various embodiments, the electronic device 100 changes the application state based on data received from the stylus 203 .

[0232] In various embodiments, the touch-sensitive surface of stylus 203 (e.g., Figure 2 and Figures 5A to 5BThe stylus 203 detects touch input and gesture input, or the lack thereof, based on the touch-sensitive surface 275 in the electronic device 100. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0233] In various embodiments, an orientation and / or movement sensor (e.g., an accelerometer, a magnetometer, a gyroscope) of the stylus 203 detects orientation / movement input or the lack thereof. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, the rotation and / or orientation of the stylus shaft, the rotation and / or orientation thereof, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0234] 7A to 7C A sequence is shown in which the electronic device 100 transitions from a first state to a second state and displays a visual indication associated with the second state based on determining that the stylus 203 is being held by the user. Figure 7A 1 shows the electronic device 100 in a first state in which the stylus 203 is not held by the hand of the user 702. The stylus 203 includes a tip 276 and an end 277 opposite the tip 276. Figure 7A As shown, electronic device 100 displays navigation area 704, canvas area 706, and toolbar area 708. Navigation area 704, canvas area 706, and toolbar area 708 are associated with a stylus-compatible application, such as a drawing application (e.g., a notepad or drawing application).

[0235] like Figure 7B As shown, stylus 203 detects that it is being held by the hand of user 702. This may occur when the hand of user 702 holds stylus 203. In response to receiving data from stylus 203 indicating that it is being held by the hand of user 702, electronic device 100 transitions from the first state to the second state. Figure 7C As shown, in the second state, electronic device 100 stops displaying navigation area 704, canvas area 706, and toolbar area 708. Electronic device 100 displays an enlarged canvas area 710 and a visual indicator 712 to indicate that electronic device 100 is in the second state. Visual indicator 712 includes a marker icon 714 with a solid tip 716 to indicate that stylus 203 will form a solid marker mark on enlarged canvas area 710. Those skilled in the art will appreciate that visual indicator 712 can take various forms.

[0236] Figures 7C to 7H 2 shows various operations performed in the second state based on the manipulation input received at the stylus 203. As described above, Figure 7C FIG. 1 shows the electronic device 100 in the second state. Figure 7C As shown, the stylus 203 detects a downward swipe gesture 718. In response to receiving manipulation input data indicating the downward swipe gesture 718 from the stylus 203, Figure 7D , the electronic device 100 displays a color palette 720 adjacent to the visual indicator 712. The color palette 720 includes four color indicators, each corresponding to a different color or pattern of a marker marking tool. Those skilled in the art will appreciate that the color indicators in the color palette 720 can include a variety of styles and colors. Figure 7D A solid fill indicator 720a associated with the solid marker is shown as currently selected (eg, currently selected) within the palette 720. For example, the solid fill indicator 720a corresponds to the solid tip 716 of the marker icon 714.

[0237] like Figure 7D As shown, stylus 203 detects a rotation manipulation 722 in a counterclockwise (from above) direction. One of ordinary skill in the art will appreciate that stylus 203 may be rotated according to any number of angular manipulations. Figure 7D The manipulation input data of the rotation manipulation 722 in Figure 7E In FIG. 7 , the electronic device 100 displays a diagonally filled indicator 720 b having focus and a solid filled indicator 720 a having no focus. In addition, the electronic device 100 displays Figure 7E , and the corresponding diagonal striped tip 724 of the marker icon 714 in the image is pointed to indicate that the stylus 203 will form a diagonal striped mark on the enlarged canvas area 710.

[0238] like Figure 7E As shown, stylus 203 detects upward swipe gesture 726. In response to receiving an indication from stylus 203 Figure 7E The manipulation input data of the upward swipe gesture 726 in Figure 7F , the electronic device 100 stops displaying the color palette 720. In addition, the electronic device 100 keeps displaying the visual indicator 712 including the marker icon 714 having a diagonally striped tip 724.

[0239] like Figure 7F As shown, stylus 203 detects tap gesture 728. In response to receiving an indication from stylus 203 Figure 7F The electronic device 100 updates the manipulation input data of the tap gesture 728 in Figure 7GThe visual indicator 712 in the example is replaced by a pencil icon 730. Figures 7C to 7F . This indicates that the stylus 203 will form a pencil mark on the enlarged canvas area 710.

[0240] like Figure 7G As shown, the stylus detects a subsequent tap gesture 732. In response to receiving an indication from the stylus 203 Figure 7G The electronic device 100 updates the manipulation input data of the subsequent tap gesture 732 in Figure 7H The visual indicator 712 in FIG. 7A is now replaced with the pencil icon 730 by a ruler icon 734. This indicates that the stylus 203 will be used as a ruler with respect to the enlarged canvas area 710.

[0241] Figures 7H to 7J A sequence is shown in which the electronic device transitions from the second state to the first state according to determining that the stylus 203 is no longer held by the user. Figure 7H 1 shows the electronic device 100 in the second state, in which the stylus 203 is being held by the hand of the user 702. Figure 7H As shown, the electronic device 100 displays the visual indicator 712 including a ruler icon 734 .

[0242] like Figure 7I As shown, stylus 203 detects that it is not being held by the hand of user 702. This may occur when user 702 puts down stylus 203. In response to receiving data from stylus 203 indicating that it is not being held by the hand of user 702, electronic device 100 transitions from the second state to the first state. Figure 7J As shown, in the first state, the electronic device 100 stops displaying the enlarged canvas area 710 and the visual indicator 712. Figure 7J , the electronic device displays something like 7A to 7B The navigation area 704, canvas area 706 and toolbar area 708 are shown.

[0243] Figures 7K to 7M Another sequence is shown of the electronic device transitioning from a first state to a second state and displaying a visual indication associated with the second state based on determining that stylus 203 is being held by a user. Figure 7K 1 shows the electronic device 100 in a first state in which the stylus 203 is not held by the hand of the user 702. Figure 7K As shown, the electronic device 100 displays a lock screen 736 .

[0244] like Figure 7LAs shown, stylus 203 detects that it is being held by the hand of user 702. This may occur when the hand of user 702 holds stylus 203. In response to receiving data from stylus 203 indicating that it is being held by the hand of user 702, electronic device 100 transitions from the first state to the second state in which electronic device 100 is not in locked mode. Figure 7M As shown, in the second state, the electronic device 100 stops displaying the lock screen 736. The electronic device 100 displays a screen similar to Figure 7C 7 and a visual indicator 712. Although the visual indicator 712 corresponds to a marker icon 714 having a solid tip 716, one of ordinary skill in the art will appreciate that the visual indicator 712 may take various forms.

[0245] Figures 7M to 7O Another sequence is shown where the electronic device transitions from the second state to the first state and ceases displaying the visual indicator based on determining that the stylus 203 is no longer held by the user. Figure 7M 1 shows the electronic device 100 in the second state, in which the stylus 203 is being held by the hand of the user 702. Figure 7M As shown, the electronic device 100 displays a display similar to Figure 7C and Figure 7M An enlarged canvas area 710 and a visual indicator 712 associated with a stylus-compatible application, such as a drawing application (eg, a notepad application).

[0246] like Figure 7N As shown, stylus 203 detects that it is not held by the hand of user 702. In response to receiving data from stylus 203 indicating that it is not held by the hand of user 702, electronic device 100 transitions from the second state to the first state in which electronic device 100 is in locked mode. Figure 7O As shown, in the first state, the electronic device 100 stops displaying the enlarged canvas area 710 and the visual indicator 712. The electronic device 100 displays the lock screen 736.

[0247] Figures 7P to 7R Yet another sequence is shown of the electronic device transitioning from a first state to a second state and displaying a visual indication associated with the second state based on determining that the stylus 203 is being held by the user. Figure 7P 1 shows the electronic device 100 in a first state in which the stylus 203 is not held by the hand of the user 702. Figure 7P As shown, the electronic device 100 displays a lock screen 736 .

[0248] like Figure 7QAs shown, stylus 203 detects that it is being held by the hand of user 702. In response to receiving data from stylus 203 indicating that it is being held by the hand of user 702, Figure 7R In the embodiment of the present invention, electronic device 100 displays prompt interface 738 superimposed on lock screen 736. Prompt interface 738 includes a "yes" enable indication 740 and a "no" enable indication 742 to enable user 702 to enter the drawing application or cancel prompt interface 738 accordingly. The user can interact with enable indications 740 and 742 by pointing touch input at the touch-sensitive surface of electronic device 100 at positions corresponding to enable indications 740 and 742. Figures 7R to 7S These interactions are described in further detail. In some embodiments, the electronic device 100 stops displaying the prompt interface 738 in response to receiving data from the stylus 203 indicating that the stylus 203 is no longer held by the hand of the user 702. In some embodiments, the electronic device 100 continues displaying the prompt interface 738 in response to receiving data from the stylus 203 indicating that the stylus 203 is no longer held by the hand of the user 702.

[0249] Figures 7R to 7S The transition from the lock screen to the limited user interface associated with the drawing application is shown. Figure 7R As shown, the electronic device 100 displays a prompt interface 738 superimposed on the lock screen 736. Figure 7R In the “yes” indication, the touch input of 740 is indicated, the electronic device 100 stops displaying the lock screen 736 and the prompt interface 738, and then displays the restricted user interface 744 (e.g., associated with the drawing application) and the visual indicator 712, as shown in FIG. Figure 7S In response to detecting a touch input corresponding to “no” affordance 742 , electronic device 100 stops displaying prompt interface 738 and continues to display lock screen 736 (not shown).

[0250] Figures 7S to 7U Another sequence is shown where the electronic device transitions from the second state to the first state and ceases displaying the visual indication based on determining that the stylus 203 is no longer being held by the user. Figure 7S The electronic device 100 is shown in a second state in which the stylus 203 is held by the hand of the user 702 .

[0251] like Figure 7T As shown, the stylus 203 detects that it is not held by the hand of the user 702. In response to receiving data from the stylus 203 indicating that it is not held by the hand of the user 702, the electronic device 100 transitions from the second state to the first state. Figure 7U As shown, in the first state, the electronic device 100 stops displaying the visual indicator 712 and the limited user interface 744. Figure 7U As shown, the electronic device 100 displays a display similar to Figure 7J The navigation area 704, the canvas area 706 and the toolbar area 708 in the first state.

[0252] Figures 7V to 7X Yet another sequence is shown of the electronic device transitioning from a first state to a second state and displaying a visual indication associated with the second state based on determining that the stylus 203 is being held by the user. Figure 7V 1 shows the electronic device 100 in a first state in which the stylus 203 is not held by the hand of the user 702. Figure 7V As shown, the electronic device 100 displays a home screen 746. The home screen 746 includes a matrix of application icons (e.g., applications) arranged in a main area 748 of the display. The home screen 746 includes a taskbar 750 that includes a row of taskbar icons. One of ordinary skill in the art will appreciate that the number and arrangement of application icons and / or taskbar icons may vary.

[0253] like Figure 7W As shown, the stylus 203 detects that it is being held by the hand of the user 702. In response to receiving data from the stylus 203 indicating that it is being held by the hand of the user 702, the electronic device 100 transitions from the first state to the second state. Figure 7X As shown, in the second state, electronic device 100 displays prompt interface 752 superimposed on home screen 746. Prompt interface 752 includes a "yes" enable indication 754 and a "no" enable indication 756 to enable the user to enter the drawing application or cancel prompt interface 752 accordingly. The user can interact with enable indications 754 and 746 via touch input directed to the touch-sensitive surface of electronic device 100 at positions corresponding to enable indications 754 and 756.

[0254] Figure 7X to Figure 7Y 1 shows a transition from the home screen to a user interface associated with a drawing application. Figure 7X As shown, the electronic device 100 displays a prompt interface 752 superimposed on the home screen 746. In response to detecting the corresponding Figure 7X In response to the touch input of “yes” indicating 754, the electronic device 100 stops displaying the home screen 746 and the prompt interface 752, and then displays the limited user interface 744 (e.g., associated with the drawing application) and the visual indicator 712, as shown in FIG. Figure 7Y In response to detecting a touch input corresponding to “no” affordance 756 , electronic device 100 stops displaying prompt interface 752 and continues to display home screen 746 (not shown).

[0255] Figures 8A to 8H Figures 200 and 201 illustrate exemplary user interfaces for changing the functionality of the stylus 203 according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including FIG. 15A to FIG. 15B Although some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and display are combined, such as on touch screen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 that is separate from display 650, such as Figure 6B In various embodiments, the electronic device 100 changes the functionality of the stylus 203 based on data received from the stylus 203 .

[0256] In various embodiments, the touch-sensitive surface of stylus 203 (e.g., Figure 2 and Figures 5A to 5B The stylus 203 detects touch input and gesture input, or the lack thereof, based on the touch-sensitive surface 275 in the electronic device 100. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0257] In various embodiments, an orientation and / or movement sensor (e.g., an accelerometer, a magnetometer, a gyroscope) of the stylus 203 detects orientation / movement input or the lack thereof. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, the rotation and / or orientation of the stylus shaft, the rotation and / or orientation thereof, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0258] Figures 8A to 8B A first sequence of making a first change to displayed content based on determining that the stylus is being held in a first grip arrangement is shown. Figure 8A As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application including content 804 (e.g., a gray rectangle). Figure 8A , the electronic device 100 detects input 810 (e.g., drawing strokes or marks) from the stylus 203 while the user is holding the stylus 203 in his / her hand 802 in a first grip arrangement 815. The first grip arrangement 815 corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing toward the electronic device 100), with the fingers of the hand 802 proximate the tip 276 of the stylus 203.

[0259] In response to detecting that the stylus 203 is held in the first gripping arrangement 815, Figures 8A to 8B , the electronic device 100 displays an indicator 812 associated with a first marking tool (eg, a felt tip marker) within the user interface 800. Figure 8B As shown, the electronic device 100 is based on Figure 8A 8 and a first marking tool (eg, a felt tip marker) associated with a first grip arrangement 815 to display a first change 820 (eg, a stroke or mark) to the user interface 800 .

[0260] Figures 8C to 8D A second sequence of making a second identification change to displayed content based on determining that the stylus is being held in a second grip arrangement is shown. Figure 8C As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application including content 804 (e.g., a gray rectangle). Figure 8C , the electronic device 100 detects input 810 (e.g., drawing strokes or marks) from the stylus 203 while the user is holding the stylus 203 in his / her hand 802 in a second grip arrangement 835. The second grip arrangement 835 corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing toward the electronic device 100), with the fingers of the hand 802 proximate an end 277 of the stylus 203 opposite the tip 276 of the stylus 203.

[0261] In response to detecting that the stylus 203 is held in the second gripping arrangement 835, Figures 8C to 8D , the electronic device 100 displays an indicator 832 associated with the second marking tool (e.g., a watercolor paint brush) within the user interface 800. Figure 8D As shown, the electronic device 100 is based on Figure 8C 810 and a second marking tool (eg, a watercolor paint brush) associated with a second grip arrangement 835 to display a second change 840 (eg, a stroke or mark) to the user interface 800 .

[0262] Figures 8E to 8F A third sequence of making a third change to displayed content based on determining that the stylus is being held in a third grip arrangement is shown. Figure 8E As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application including content 804 (e.g., a gray rectangle). Figure 8E, the electronic device 100 detects input 810 (e.g., drawing strokes or marks) from the stylus 203 while the user is holding the stylus 203 in his / her hand 802 in a third grip arrangement 855. The third grip arrangement 855 corresponds to holding the stylus 203 in an inverted orientation (e.g., with the tip 276 of the stylus 203 pointing outward from the electronic device 100) and proximate an end 277 of the stylus 203 opposite the tip 276 of the stylus 203.

[0263] In response to detecting that the stylus 203 is held in the third gripping arrangement 855, Figures 8E to 8F , the electronic device 100 displays an indicator 852 associated with a third marking tool (eg, an eraser) within the user interface 800. Figure 8F As shown, the electronic device 100 is based on Figure 8E 810 and a third marking tool (eg, an eraser) associated with a third grip arrangement 855 to display a third change 860 (eg, a stroke or mark) to the user interface 800 .

[0264] Figures 8G to 8H A fourth sequence of making a fourth change to displayed content based on determining that the stylus is being held in a fourth grip arrangement is shown. Figure 8G As shown, the electronic device 100 displays a user interface 800 associated with a drawing or notepad application including content 804 (e.g., a gray rectangle). Figure 8G , the electronic device 100 detects input 810 (e.g., drawing strokes or marks) from the stylus 203 while the user is holding the stylus 203 in his / her hand 802 in a fourth grip arrangement 875. The fourth grip arrangement 875 corresponds to holding the stylus 203 in an inverted orientation (e.g., with the tip 276 of the stylus 203 pointing outward from the electronic device 100) and close to the tip 276 of the stylus 203.

[0265] In response to detecting that the stylus 203 is held in the fourth gripping arrangement 875, Figures 8G to 8H , the electronic device 100 displays an indicator 872 associated with a fourth marking tool (eg, a spray can) within the user interface 800. Figure 8H As shown, the electronic device 100 is based on Figure 8G 8 and a fourth marking tool (eg, a spray can) associated with a fourth grip arrangement 875 to display a fourth change 880 (eg, a stroke or logo) to the user interface 800 .

[0266] Those skilled in the art will understand that referring to Figures 8A to 8HThe specific depiction of the grip arrangement to the identification type in the sequence described is arbitrary and subject to change. One of ordinary skill in the art will appreciate that although reference is made to Figures 8A to 8H The same input 810 is shown in the depicted sequence, but in various other embodiments, other input vectors may be detected when the stylus is held in a grip arrangement.

[0267] Figures 9A to 9P Figure 1 shows an exemplary user interface for modifying stylus input functionality according to some embodiments. The user interfaces in these figures are used to illustrate the process described below, including 16A to 16B Although some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and display are combined, such as on touch screen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 that is separate from display 650, such as Figure 6B In various embodiments, the electronic device 100 modifies the touch input functionality based on data received from the stylus 203 .

[0268] In various embodiments, the touch-sensitive surface of stylus 203 (e.g., Figure 2 and Figures 5A to 5B The stylus 203 detects touch input and gesture input, or the lack thereof, based on the touch-sensitive surface 275 in the electronic device 100. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0269] In various embodiments, an orientation and / or movement sensor (e.g., an accelerometer, a magnetometer, a gyroscope) of the stylus 203 detects orientation / movement input or the lack thereof. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, the rotation and / or orientation of the stylus shaft, the rotation and / or orientation thereof, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0270] Figures 9A to 9C An example of performing a first operation based on determining that the stylus is being held is shown. Figure 9AAs shown, electronic device 100 displays a user interface 900 associated with a drawing or notepad application that includes content 904 (e.g., a logo) and a visual indicator 906 indicating that stylus 203 is being held by the hand of user 902. Visual indicator 906 corresponds to a solid tip marker icon to indicate that stylus 203 will form a solid marker logo on user interface 900. Those skilled in the art will appreciate that content 904 and / or visual indicator 906 can take various forms.

[0271] like Figure 9A As shown, electronic device 100 detects a left swipe gesture 908 on the touch-sensitive surface of electronic device 100. In response to detecting left swipe gesture 908 and in accordance with determining that stylus 203 is being held by the hand of user 902 based on data received from stylus 203, electronic device 100 performs an erase or undo operation with respect to content 904. As a result, electronic device 100 stops displaying content 904 on user interface 900 (e.g., Figure 9B as shown) and visual indicator 906 remains displayed.

[0272] like Figure 9B As shown, electronic device 100 detects a right swipe gesture 910 on the touch-sensitive surface of electronic device 100. In response to detecting right swipe gesture 910 and determining that stylus 203 is being held by the hand of user 902 based on data received from stylus 203, electronic device 100 performs a restore operation with respect to content 904. As a result, electronic device 100 redisplays content 904 on user interface 900 (e.g., Figure 9C as shown) and visual indicator 906 remains displayed.

[0273] Figures 9D to 9E An example of performing the second operation according to determining that the stylus pen is not held is shown. Figure 9D The electronic device 100 determines that the stylus 203 is not held by the hand of the user 902 based on the data received from the stylus 203 and / or based on the lack (e.g., absence) of data received from the stylus 203. Figures 9D to 9E As shown, the electronic device 100 does not display Figures 9A to 9C Visual indicator 906 is shown.

[0274] like Figure 9D As shown, the electronic device 100 detects a left swipe gesture 908 (e.g., a stroke) on the touch-sensitive surface of the electronic device 100. In response to detecting the left swipe gesture 908 (e.g., similar to Figure 9A 908) and based on determining that the stylus 203 is not held by the hand of the user 902, the electronic device 100 Figure 9EExecute a drawing operation on the user interface 900 in FIG. Figure 9E As shown, the electronic device 100 displays the corresponding Figure 9D 908 in the left swipe gesture 908, and keeps displaying the content 904. This drawing operation is different from the erase / undo operation performed in response to the left swipe gesture 908 made when the stylus 203 is held. 7A to 7B Shown in.

[0275] Figures 9F to 9H Another example of performing the first operation according to determining that the stylus is held is shown. Figure 9F As shown, electronic device 100 displays user interface 900 including content 904 (eg, a logo) and visual indicator 906 indicating that stylus 203 is being held by the hand of user 902 .

[0276] like Figure 9F As shown, electronic device 100 detects a looping gesture 916 (e.g., a lasso gesture) on the touch-sensitive surface of electronic device 100. Looping gesture 916 corresponds to enclosing (e.g., substantially enclosing) content 904. In other words, looping gesture 916 corresponds to circling and / or surrounding displayed content, including a portion of the displayed content (e.g., the top half of a circle, a segment of a line, a small piece of an image, half of a stanza, etc.). Those of ordinary skill in the art will appreciate that looping gesture 916 may include various lines (e.g., regular polygonal lines, irregular polygonal lines, circular lines, oval lines, lines with various curvatures, or combinations thereof) and / or may enclose multiple or more types of displayed content (e.g., lines, polygons, logos, images, text, etc.).

[0277] In response to detecting Figure 9F The looping gesture 916 in FIG. 1 and based on determining that the stylus 203 is being held by the hand of the user 902 based on the data received from the stylus 203, the electronic device 100 changes the content 904 surrounded by the looping gesture 916 to indicate that the content 904 has been Figure 9G is selected. That is, Figures 9F to 9G As shown, the electronic device 100 changes the content 904 from the solid line to the dotted line 920. One skilled in the art will appreciate that the electronic device 100 may change the content 904 in various ways to indicate detection of the ring gesture 916.

[0278] like Figure 9G As shown, the electronic device 100 detects a drag gesture 922 including a start point 924 and an end point 926. In response to detecting the drag gesture 922 and determining that the stylus 203 is being held by the hand of the user 902 based on data received from the stylus 203, the electronic device 100 moves the content 920 according to the drag gesture 922, as shown in FIG. Figure 9H That is, Figure 9H As shown, the electronic device 100 moves the content 920 (eg, changes its display position) to the end point 926 of the drag gesture 922 and restores the display of the content 904 to the solid line mark.

[0279] Figures 9I to 9J Another example of performing the second operation based on determining that the stylus is not held is shown. The electronic device 100 determines that the stylus 203 is not held by the hand of the user 902 based on data received from the stylus 203 and / or based on the lack (e.g., absence) of data received from the stylus 203. Figure 9I As shown, in response to determining that the stylus 203 is not held by the hand of the user 902, the electronic device 100 does not display Figures 9F to 9H The electronic device 100 is in the visual indicator 906 shown. Figure 9I 900 in the user interface 900. The navigation area 928, the canvas area 930, and the toolbar area 932 are displayed. The navigation area 928, the canvas area 930, and the toolbar area 932 are associated with a stylus-compatible application such as a drawing application (e.g., a notepad or a drawing application).

[0280] like Figure 9I As shown, the electronic device 100 detects a ring gesture 916 (e.g., similar to Figure 9F However, because the stylus 203 is not held by the hand of the user 902, the electronic device 100 performs the same gesture as that relative to the stylus 203. Figures 9F to 9H The first operation is different from the second operation. Figure 9J As shown, in response to detecting looping gesture 916 and determining that stylus 203 is not held by a hand of user 902 based on data received from stylus 203 and / or based on a lack of such data, electronic device 100 displays an indicator 934 corresponding to looping gesture 916.

[0281] Figures 9K to 9M Another example of performing the first operation according to determining that the stylus is held is shown. Figure 9K As shown, electronic device 100 displays user interface 900 including text 936 and visual indicator 906 indicating that stylus 203 is being held by the hand of user 902 .

[0282] like Figure 9K As further shown in FIG, the electronic device 100 detects a right swipe gesture 938 on the touch-sensitive surface of the electronic device 100. In response to detecting Figure 9K The right swipe gesture 938 in FIG. 1 and based on determining that the stylus 203 is being held by the hand of the user 902 based on the data received from the stylus 203, the electronic device 100 selects a portion of the text 936, such as Figure 9LThat is, Figure 9L As shown, the electronic device displays the selected text 940 with a selection indicator 941 indicating the selection.

[0283] like Figure 9L As further shown, the electronic device 100 detects a drag gesture 942 including a start point 944 and an end point 946. In response to detecting Figure 9L The drag gesture 942 in FIG. 1 and according to the data received from the stylus 203, it is determined that the stylus 203 is being held by the hand of the user 902, the electronic device 100 moves the selected text 940 according to the drag gesture 942, as shown in FIG. Figure 9M That is, Figure 9M As shown, the electronic device 100 moves the selected text 940 (eg, changes its display position) to the endpoint 946 of the drag gesture 942. Figure 9M As shown, the electronic device 100 displays modified text 948 corresponding to the text 936 without the moved selected text 940.

[0284] Figures 9N to 9P Another example of performing the second operation based on determining that the stylus is not held is shown. Figure 9N As shown, electronic device 100 displays user interface 900 including text 936. Electronic device 100 displays navigation area 928, canvas area 930, and toolbar area 932 on user interface 900. Navigation area 928, canvas area 930, and toolbar area 932 are associated with a stylus-compatible application such as a drawing application (e.g., a notepad or drawing application). In response to data received from stylus 203 indicating that stylus 203 is not held by the hand of user 902 and / or the lack of such data, different from Figures 9K to 9M , the electronic device 100 is Figure 9N No visual indicator 906 is displayed.

[0285] like Figure 9N As further shown in FIG, electronic device 100 detects a right swipe gesture 938 on the touch-sensitive surface of electronic device 100. In response to detecting right swipe gesture 938 and in accordance with a determination that stylus 203 is not being held by the hand of user 902 based on data received from stylus 203 and / or a lack of such data, electronic device 100 highlights a portion of text 936, as shown in FIG. Figure 9O That is, Figure 9O As shown, the electronic device 100 displays a highlighted text 950 with a highlight indicator 952 indicating highlighting. This highlight operation is different from when the stylus 203 is being held by the hand of the user 902. Figures 9K to 9L The selection operation that occurred.

[0286] like Figure 9OAs further shown, the electronic device 100 detects a drag gesture 942 including a start point 944 and an end point 946. In response to detecting Figure 9O and determining that the stylus 203 is not held by the hand of the user 902 based on the data received from the stylus 203 and / or the lack of such data, the electronic device 100 performs the drag gesture 942 in Figure 9P The icon 954 corresponding to the drag gesture 942 is displayed in the middle. This icon display operation is different from the operation when the stylus 203 is being held by the hand of the user 902. Figures 9L to 9M The move operation that occurred. Figure 9P As further shown in FIG. 9 , the electronic device 100 maintains display of text 936 , highlighted text 950 , and highlight indicator 952 .

[0287] 10A to 10I An exemplary user interface for performing an operation on an existing identifier based on finger manipulation input according to some embodiments is shown. The user interfaces in these figures are used to illustrate the process described below, including 17A to 17C Although some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and display are combined, such as on touch screen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 that is separate from display 650, such as Figure 6B In various embodiments, the electronic device 100 performs an operation on an existing identifier based on data received from the stylus 203 .

[0288] In various embodiments, the touch-sensitive surface of stylus 203 (e.g., Figure 2 and Figures 5A to 5B The stylus 203 detects touch input and gesture input, or the lack thereof, based on the touch-sensitive surface 275 in the electronic device 100. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0289] In various embodiments, an orientation and / or movement sensor (e.g., an accelerometer, a magnetometer, a gyroscope) of the stylus 203 detects orientation / movement input or the lack thereof. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, the rotation and / or orientation of the stylus shaft, the rotation and / or orientation thereof, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0290] FIG. 10A to FIG. 10B A sequence for selecting a user interface element within a user interface is shown. Figure 10A As shown, the electronic device 100 displays a user interface 1000 associated with a drawing or notepad application, which includes pre-existing content: a star 1004a and a lightning bolt 1004b. Figure 10A , when the user is holding the stylus 203 in his / her hand 1002 by clenching one or more fingers 202 of the hand 1002 into a fist, the electronic device 100 detects input 1010 of a substantially circular mark (e.g., a drawing stroke or mark) around the lightning bolt 1004b from the one or more fingers 202.

[0291] In response to detecting the selection Figure 10A The electronic device 100 receives the input 1010 of the lightning 1004b, and the electronic device 100 receives the input 1010 of the lightning 1004b. Figure 10B The selected state in FIG shows a lightning bolt 1004b' with a dashed outline to indicate that the lightning bolt 1004b' is currently selected. Figure 10B , star 1004a remains shown with a solid outline corresponding to the user not selecting star 1004a.

[0292] FIG. 10B to FIG. 10C A sequence is shown for performing a first operation (e.g., increasing size) on a user interface element based on determining that finger manipulation data from a stylus indicates a first finger manipulation input on the stylus (e.g., a counterclockwise roll of the stylus). Figure 10B In FIG. 1 , the electronic device 100 displays the lightning 1004b′ in the first size 1015a. Figure 10B As shown, when the user is holding the stylus 203 in his / her hand 1002 and rolling the stylus 203 in a counterclockwise direction, the stylus 203 detects input 1020a (eg, counterclockwise rolling of the stylus 203).

[0293] In response to obtaining instructions Figure 10B The finger manipulation data of input 1020a in Figure 10C , the electronic device 100 displays the lightning bolt 1004 b ′ in the user interface 1000 , which increases from the first size 1015 a to the lightning bolt 1004 c ′ at the second size 1015 b .

[0294] FIG. 10C to FIG. 10D A sequence is shown for re-performing a first operation (e.g., increasing size) on a user interface element based on determining that finger manipulation data from the stylus indicates a first finger manipulation input on the stylus (e.g., a counterclockwise roll of the stylus). Figure 10CAs shown, when the user is holding the stylus 203 in his / her hand 1002 and rolling the stylus 203 in a counterclockwise direction, the stylus 203 detects input 1020b (eg, counterclockwise rolling of the stylus 203). Figure 10C The finger manipulation data of input 1020b in Figure 10D , the electronic device 100 displays the lightning bolt 1004c′ in the user interface 1000, and the lightning bolt 1004c′ is further enlarged from the second size 1015b to the third size 1015c .

[0295] Figures 10D to 10E A sequence is shown for performing a second operation (e.g., reducing size) on a user interface element based on determining that finger manipulation data from the stylus indicates a second finger manipulation input on the stylus (e.g., a clockwise roll of the stylus). Figure 10D As shown, when the user is holding the stylus 203 in his / her hand 1002 and rolling the stylus 203 in a clockwise direction, the stylus 203 detects input 1020c (eg, clockwise rolling of the stylus 203). Figure 10D The finger manipulation data of input 1020c in Figure 10E , the electronic device 100 further reduces the size of the lightning bolt 1004d' displayed in the user interface 1000 from the third size 1015c to the lightning bolt 1004e' at the fourth size 1015d.

[0296] Figures 10E to 10F A sequence is shown in which a second operation (e.g., reducing size) is again performed on the user interface element based on determining that finger manipulation data from the stylus indicates a second finger manipulation input on the stylus (e.g., a clockwise roll of the stylus). Figure 10E As shown, when the user is holding the stylus 203 in his / her hand 1002 and rolling the stylus 203 in a clockwise direction, the stylus 203 detects an input 1020d (eg, a clockwise rolling of the stylus 203). Figure 10E The finger manipulation data of input 1020d is Figure 10F , the electronic device 100 further reduces the size of the lightning bolt 1004e' displayed in the user interface 1000, from the fourth size 1015d to the lightning bolt 1004e' at the fifth size 1015e.

[0297] Figures 10G to 10H Another sequence is shown for performing a first operation (e.g., a cut operation) on a user interface element based on determining that finger manipulation data from the stylus indicates a third finger manipulation input on the stylus (e.g., an upward swipe on the stylus). Figure 10GAs shown, the electronic device 100 displays a user interface 1000 associated with a drawing or notepad application, which includes pre-existing content: a triangle 1004d. Figure 10G As shown, while the user is holding stylus 203 in his / her hand 1002 , stylus 203 detects input 1040 (e.g., an upward swipe on stylus 203 ) at the position of stylus 203 relative to electronic device 100 , indicating that the user has selected to cut triangle 1004 d from user interface 1000 .

[0298] In response to obtaining instructions Figure 10G The finger manipulation data of input 1040 is Figure 10H In FIG. 1 , the electronic device 100 performs a first operation (eg, a cut operation) on the triangle 1004d within the user interface 1000. In some embodiments, the first operation corresponds to a copy operation. Figure 10H As shown, in response to detecting an upward swipe on the stylus 203 corresponding to the user cutting (or, in some embodiments, copying) the triangle 1004d, the electronic device 100 no longer displays the triangle 1004d on the user interface 1000.

[0299] Figures 10H to 10I A sequence is shown for performing a second operation (e.g., a paste operation) on a user interface element based on determining that finger manipulation data from the stylus indicates a fourth finger manipulation input on the stylus (e.g., a downward swipe gesture on the stylus). Figure 10H As shown, while the user is holding stylus 203 in his / her hand 1002 , stylus 203 detects input 1050 (eg, a downward swipe on stylus 203 ) at the position of stylus 203 relative to electronic device 100 .

[0300] In response to obtaining instructions Figure 10H The finger manipulation data of input 1050 is Figure 10I In FIG. 1 , the electronic device 100 performs a second operation (eg, a paste operation) on the triangle 1004d in the user interface 1000. Figure 10I As shown, in response to detecting a downward swipe on stylus 203 corresponding to the user pasting triangle 1004d to user interface 1000, the electronic device displays triangle 1004d on user interface 1000 at the position of stylus 203 relative to electronic device 100.

[0301] 11A to 11O FIGURE 2 shows an exemplary user interface for performing finger manipulation of the stylus 203 to navigate within a menu according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including 18A to 18BAlthough some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and display are combined, such as on touch screen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 that is separate from display 650, such as Figure 6B In various embodiments, the electronic device 100 navigates within the menu based on data received from the stylus 203 .

[0302] In various embodiments, the touch-sensitive surface of stylus 203 (e.g., Figure 2 and Figures 5A to 5B The stylus 203 detects touch input and gesture input, or the lack thereof, based on the touch-sensitive surface 275 in the electronic device 100. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0303] In various embodiments, an orientation and / or movement sensor (e.g., an accelerometer, a magnetometer, a gyroscope) of the stylus 203 detects orientation / movement input or the lack thereof. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, the rotation and / or orientation of the stylus shaft, the rotation and / or orientation thereof, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0304] Figures 11A to 11B A first sequence of making a first change to the displayed content is shown. Figure 11A As shown, the electronic device 100 displays a user interface 1100 associated with a drawing or notepad application. Figure 11A In FIG, when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1110 (eg, drawing strokes or marks) from the stylus 203. In response to the electronic device 100 detecting the input 1110, Figure 11B In the embodiment, the electronic device 100 is based on Figure 11A Input 1110 in is displayed to show a first change 1106 (e.g., a stroke or mark) to user interface 1100 to display user interface element 1104.

[0305] Figures 11C to 11DAnother sequence is shown for performing a first operation on a user interface element (e.g., an operation to open a menu) based on determining that finger manipulation data from a stylus indicates a first finger manipulation input on the stylus (e.g., an upward swipe on the stylus). Figure 11C As shown, when the user is holding the stylus 203 in his / her hand 1102 , the stylus 203 detects an input 1120 a (eg, an upward swipe on the stylus 203 ) at the position of the stylus 203 relative to the electronic device 100 .

[0306] In response to obtaining instructions Figure 11C The finger manipulation data of input 1120a in Figure 11D , the electronic device 100 displays a menu 1114 on the user interface 1100. The menu 1114 includes four visual indicators, namely a solid indicator 1114a, a striped indicator 1114b, a dotted indicator 1114c, and a blank indicator 1114d, wherein the solid indicator 1114a has the focus by default (as shown by the focus indicator 1114i). Figure 11D As shown, menu 1114 is a radial menu having four visual indicators arranged in a circle. In addition, focus indicator 1114i corresponds to a star or other icon near the selectable item with focus, a ring around the selectable item with focus, a magnification of the selectable item in focus, a change in the color or appearance of the selectable item with focus, etc. Those skilled in the art will understand that menu 1114 can include any number of visual indicator types with various characteristics, wherein any of these visual indicators has focus by default.

[0307] Figures 11D to 11E Another sequence is shown in which a second operation is performed based on determining that finger manipulation data from the stylus indicates a second finger manipulation input on the stylus (e.g., a clockwise roll of the stylus). In some embodiments, the electronic device 100 can change which indicator is in focus in response to the user's hand 1102 manipulating the stylus 203. For example, in response to obtaining finger manipulation data from the stylus 203 indicating a clockwise rotation 1130a of the stylus 203, the electronic device 100 moves through (e.g., changes display of) the menu 1114 clockwise, causing the focus to change from the solid indicator 1114a to the striped indicator 1114b.

[0308] Figures 11E to 11FThe sequence of performing a second operation again based on determining that finger manipulation data from the stylus indicates a second finger manipulation input on the stylus (e.g., a clockwise roll of the stylus) is shown. For example, in response to obtaining finger manipulation data indicating a clockwise rotation 1130b of the stylus 203 from the stylus 203, the electronic device 100 further moves clockwise through (e.g., changes display of) the menu 1114, causing the focus to change from the striped indicator 1114b to the dotted indicator 1114c.

[0309] Figures 11F to 11G Another sequence is shown in which a third operation is performed based on a determination that finger manipulation data from the stylus indicates a third finger manipulation input on the stylus (e.g., a counterclockwise roll of the stylus). For example, in response to obtaining finger manipulation data from the stylus 203 indicating a counterclockwise rotation 1130c of the stylus 203, the electronic device 100 moves counterclockwise through (e.g., changes display of) the menu 1114, causing the focus to change from the dot indicator 1114c back to the striped indicator 1114b.

[0310] Figures 11G to 11H Another sequence of performing an operation (e.g., a selection operation) on a user interface element based on determining that finger manipulation data from a stylus indicates a manipulation input on the stylus (e.g., a tap on the stylus) is shown. Figure 11G As shown, when the user is holding the stylus 203 in his / her hand 1102, the stylus 203 detects an input 1140a (eg, a tap on the stylus 203) at the position of the stylus 203 relative to the electronic device 100. Figure 11G The finger manipulation data of input 1140a in Figure 11H In the embodiment, the electronic device 100 removes the menu 1114 on the user interface 1110 from the display. Figure 11G Indicator 1112a associated with a first marking tool (eg, a felt tip marker) in a solid line is shown in FIG. Figure 11H The indicator 1112b is changed to the indicator 1112b associated with the first marking tool in the shape of a stripe.

[0311] Figures 11H to 11I Another sequence is shown in which a second change is made to the displayed content. Figure 11H In FIG. 1 , when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1150 (eg, a drawing stroke or mark) from the stylus 203. In response to the electronic device 100 detecting the input 1150, Figure 11I In the embodiment, the electronic device 100 is based on Figure 11H 1150 to display a second change 1116 (e.g., a stroke or mark) to the user interface 1100 to display the user interface element 1124. Figure 11IAs shown, user interface element 1124 is a striped line corresponding to tool 1112b.

[0312] Figures 11J to 11K Another sequence is shown in which a third change is made to the displayed content. Figure 11J In FIG, when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1160 (eg, drawing strokes or marks) from the stylus 203. In response to the electronic device 100 detecting the input 1160, Figure 11K In the embodiment, the electronic device 100 is based on Figure 11J 1160 to display a third change 1126 (e.g., a stroke or mark) to the user interface 1100 to display the user interface element 1134. Figure 11J As shown, user interface element 1134 is a solid line corresponding to tool 1112a.

[0313] Figures 11K to 11L Another sequence is shown for performing an operation on a user interface element (e.g., an operation to open a menu) based on determining that finger manipulation data from a stylus indicates finger manipulation input on the stylus (e.g., a tap on the stylus). Figure 11K As shown, when the user is holding the stylus 203 in his / her hand 1102 , the stylus 203 detects an input 1120 b (eg, a tap on the stylus 203 ) at the position of the stylus 203 relative to the electronic device 100 .

[0314] In response to obtaining instructions Figure 11K The finger manipulation data of input 1120b in Figure 11L , the electronic device 100 displays a menu 1144 on the user interface 1110. The menu 1144 includes five tool indicators, namely a felt-tip marker tool indicator 1144a, a brush tool indicator 1144b, an eraser tool indicator 1144c, a pencil tool indicator 1144d, and a chisel marker tool indicator 1144e, wherein the felt-tip marker tool indicator 1144a has focus by default (as indicated by focus indicator 1144i). Those skilled in the art will appreciate that the menu 1144 may include any number of tool indicator types with various characteristics, wherein any one of these tool indicators has focus by default.

[0315] Figures 11L to 11MAnother sequence of operations is shown in which finger manipulation data from the stylus indicates finger manipulation input on the stylus (e.g., a counterclockwise roll of the stylus). For example, in response to obtaining finger manipulation data from the stylus 203 indicating a counterclockwise rotation 1130d of the stylus 203, the electronic device 100 moves counterclockwise through (e.g., changes display of) the menu 1144, causing the focus to change from the felt tip marker tool indicator 1144a to the brush tool indicator 1144b.

[0316] Figures 11M to 11N Performing an operation (e.g., a selection operation) on a user interface element based on determining that finger manipulation data from the stylus indicates a manipulation input on the stylus (e.g., a tap on the stylus) is shown. Figure 11M As shown, when the user is holding the stylus 203 in his / her hand 1102, the stylus 203 detects an input 1140b (eg, a tap on the stylus 203) at the position of the stylus 203 relative to the electronic device 100. Figure 11M The finger manipulation data indicating the input 1140b and the selection of the brush tool indicator 1144b are obtained in Figure 11N In the embodiment, the electronic device 100 removes the menu 1144 on the user interface 1110 from the display. Figure 11M Indicator 1112a associated with a first marking tool (eg, a felt tip marker) is shown in FIG. Figure 11N , changes to an indicator 1112b associated with a second marking tool (e.g., a brush).

[0317] Figures 11N to 11O Another sequence is shown in which a fourth change is made to the displayed content. Figure 11N In FIG. 1 , when the user is holding the stylus 203 in his / her hand 1102, the electronic device 100 detects input 1170 (eg, a drawing stroke or mark) from the stylus 203. In response to the electronic device 100 detecting the input 1170, Figure 11O In the embodiment, the electronic device 100 is based on Figure 11N 1170 to display a fourth change 1136 (e.g., a stroke or mark) to the user interface 1100 to display the user interface element 1154. Figure 11O As shown, user interface element 1154 is a drawing stroke corresponding to tool 1112c.

[0318] 12A to 12O 2 shows exemplary user interfaces for displaying user interface elements based on the hovering distance of the stylus 203 according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including 19A to 19CAlthough some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and display are combined, such as on touch screen 112), in some embodiments, electronic device 100 detects input on a touch-sensitive surface 651 that is separate from display 650, such as Figure 6B In various embodiments, electronic device 100 displays a user interface element based on a hovering distance of stylus 203 , which is based on data received from stylus 203 .

[0319] In various embodiments, the touch-sensitive surface of stylus 203 (e.g., Figure 2 and Figures 5A to 5B The stylus 203 detects touch input and gesture input, or the lack thereof, based on the touch-sensitive surface 275 in the electronic device 100. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, flicked, swiped, tapped, double-clicked, etc.

[0320] In various embodiments, an orientation and / or movement sensor (e.g., an accelerometer, a magnetometer, a gyroscope) of the stylus 203 detects orientation / movement input or the lack thereof. Based on these detected inputs, the stylus 203 provides corresponding data to the electronic device 100. For example, in some embodiments, the stylus 203 provides data to the electronic device 100 indicating one or more of the following: whether the stylus is being held, the rotation and / or orientation of the stylus shaft, the rotation and / or orientation thereof, the orientation (e.g., position) of the tip 276 and / or end 277 of the stylus 203 relative to a reference plane, etc.

[0321] 12A to 12C An example is shown in which a marker is displayed according to a hovering distance of the stylus satisfying a first distance threshold. Figure 12A 12 includes a bird's-eye view 1202 of electronic device 100 and a side view 1204 of electronic device 100. As shown in bird's-eye view 1202, electronic device 100 displays a user interface 1206 (e.g., associated with a drawing or notepad application) that includes a visual indicator 1208 indicating that stylus 203 is being held by the hand of user 1210. Visual indicator 1208 corresponds to a solid tip marker icon to indicate that stylus 203 will form a solid mark on user interface 1206. Those skilled in the art will appreciate that visual indicator 1208 can take various forms.

[0322] Bird's-eye view 1202 and side view 1204 include a first position 1212 on the touch-sensitive surface of electronic device 100 that is located below tip 276 of stylus 203. In some embodiments, first position 1212 corresponds to the end of a straight vertical line starting from tip 276 of stylus 203. Those skilled in the art will appreciate that first position 1212 can vertically correspond to various points on stylus 203, such as end 277 of stylus 203, the midpoint of stylus 203, etc.

[0323] Figure 12A Also included is a rangefinder 1214. Rangefinder 1214 indicates a first hovering distance 1216. First hovering distance 1216 corresponds to the distance between stylus 203 and the touch-sensitive surface of electronic device 100 when stylus 203 is held above first position 1212 on the touch-sensitive surface of electronic device 100. Electronic device 100 determines first hovering distance 1216 based on data from stylus 203 (e.g., data indicating input detected at the stylus), data generated at electronic device 100 (e.g., sensor information at electronic device 100), or a combination thereof. Rangefinder 1214 also indicates a first distance threshold 1218 and a second distance threshold 1220.

[0324] like Figure 12A As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1222. In response to detecting the tap gesture 1222 and based on determining that the first hovering distance 1216 satisfies (e.g., reaches or exceeds) the first distance threshold 1218, the electronic device 100 displays a first cube 1224a associated with the first position 1212. Figure 12B , the electronic device 100 displays the first cube 1224a and keeps displaying the visual indicator 1208. For example, Figure 12B , a first cube 1224a is displayed at a position corresponding to the first position 1212 within the user interface 1206 (e.g., the first cube 1224a is centered on the first position 1212). Although the electronic device 100 displays a cube, those skilled in the art will appreciate that the electronic device 100 may display one or more of a variety of user interface elements, such as a logo, text, a menu, a bullet, an object, etc.

[0325] like FIG. 12A to FIG. 12B As shown, the stylus 203 is moved. Figure 12B As shown, bird's-eye view 1202 and side view 1204 illustrate second position 1226 on electronic device 100. Second hovering distance 1228 corresponds to the distance between stylus 203 and the touch-sensitive surface of electronic device 100 when stylus 203 is held over second position 1226 on electronic device 100.

[0326] like Figure 12B As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1230. In response to detecting the tap gesture 1230 and based on determining that the second hovering distance 1228 meets (e.g., reaches or exceeds) the first distance threshold 1218, the electronic device 100 displays a second cube 1224b associated with the second position 1226. Figure 12C , the electronic device 100 displays the second cube 1224b and keeps displaying the first cube 1224a and the visual indicator 1208. For example, in Figure 12C , a second cube 1224b is displayed at a location within the user interface 1206 that corresponds to the second location 1226 (eg, the second cube 1224b is centered on the second location 1226). Figure 12C As shown, because the first hovering distance 1216 and the second hovering distance 1228 satisfy the first distance threshold 1218 , the displayed first cube 1224 a and the second cube 1224 b share the same attributes (eg, are the same cube).

[0327] FIG. 12C to FIG. 12D An example of displaying a logo according to the hovering distance of the stylus meeting the second distance threshold is shown. Figures 12B to 12C As shown, the stylus 203 is moved to a position above the third position 1234. Figure 12C As shown, bird's-eye view 1202 and side view 1204 indicate a third position 1234 on electronic device 100. Third hovering distance 1236 corresponds to the distance between stylus 203 and the touch-sensitive surface of electronic device 100 when stylus 203 is held over third position 1234 on electronic device 100.

[0328] like Figure 12C As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1238. In response to detecting the tap gesture 1238 and based on determining that the third hovering distance 1236 satisfies (e.g., reaches or exceeds) the second distance threshold 1220 (e.g., exceeds), the electronic device 100 displays a third cube 1240 associated with the third position 1234. Figure 12D , the electronic device 100 displays the third cube 1240 and keeps displaying the first cube 1224a, the second cube 1224b, and the visual indicator 1208. For example, in Figure 12D , a third cube 1224c is displayed at a location within the user interface 1206 that corresponds to the third position 1234 (eg, the third cube 1224c is centered on the third position 1234).

[0329] It is worth noting that the electronic device 100 behaves differently depending on whether the hovering distance of the stylus meets the first distance threshold 1218 or the second distance threshold 1220. That is, depending on whether the first threshold 1218 is met, the electronic device 100 Figures 12B to 12C The first cube 1224a and the second cube 1224b are displayed; and according to the second distance threshold 1220 being met, the electronic device displays Figure 12D A third cube 1240 of a larger size in FIG. One of ordinary skill in the art will appreciate that the user interface elements corresponding to satisfying the first distance threshold 1218 may differ from the user interface elements corresponding to satisfying the second distance threshold 1220 in various ways.

[0330] 12E to 12F Another example of displaying a logo according to the hovering distance of the stylus meeting the first distance threshold is shown. Figure 12E As shown in the bird's-eye view 1202 in FIG, the electronic device 100 displays a user interface 1206 (e.g., associated with a drawing or notepad application) including a visual indicator 1208 indicating that the stylus 203 is being held by the hand of the user 1210. Figure 12D 1202 and side view 1204 indicate a fourth position 1242 on the touch-sensitive surface of the electronic device 100 below the tip 276 of the stylus 203. A fourth hovering distance 1244 corresponds to a distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the fourth position 1242 on the electronic device 100.

[0331] like Figure 12E As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1246. In response to detecting the tap gesture 1246 and based on determining that the fourth hovering distance 1244 meets (e.g., reaches or exceeds) the first distance threshold 1218, the electronic device 100 displays a solid oval 1248 associated with the fourth position 1242. Figure 12F , the electronic device 100 displays the solid oval 1248 and maintains displaying the visual indicator 1208. For example, in Figure 12F , a solid oval 1248 is displayed at a position corresponding to the fourth position 1242 within the user interface 1206 (e.g., the solid oval 1248 is centered on the fourth position 1242). Although the electronic device 100 displays the solid oval 1248, those skilled in the art will appreciate that the electronic device 100 may display one or more of a variety of user interface elements, such as a logo, a menu, a bullet, an object, etc.

[0332] Figures 12F to 12GAnother example of displaying a logo according to the hovering distance of the stylus meeting the second distance threshold is shown. 12E to 12F As shown, the stylus 203 is moved to a position above the fifth position 1250. Figure 12F , bird's-eye view 1202 and side view 1204 indicate a fifth position 1250 on electronic device 100. Fifth hovering distance 1252 corresponds to a distance between stylus 203 and the touch-sensitive surface of electronic device 100 when stylus 203 is held over fifth position 1250 on electronic device 100.

[0333] like Figure 12F As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1254. In response to detecting the tap gesture 1254 and based on determining that the fifth hovering distance 1252 meets (e.g., reaches or exceeds) the second distance threshold 1220, the electronic device 100 displays a splash mark 1256 associated with the fifth position 1250. Figure 12G , the electronic device 100 displays the splash mark 1256 and maintains the display of the solid oval 1248 and the visual indicator 1208. For example, in Figure 12G , the splash marker 1256 is displayed at a position within the user interface 1206 corresponding to the fifth position 1250 (eg, the splash marker 1256 is centered on the fifth position 1250).

[0334] It is worth noting that the electronic device 100 behaves differently depending on whether the hovering distance of the stylus meets the first distance threshold 1218 or the second distance threshold 1220. That is, depending on whether the first threshold 1218 is met, the electronic device 100 displays Figure 12F The solid oval 1248 in the figure; and according to satisfying the second distance threshold 1220, the electronic device 100 displays Figure 12G Sputtering mark 1256 in.

[0335] FIG. 12H to FIG. 12I Another example of displaying bullets according to the hovering distance of the stylus meeting the first distance threshold is shown. Figure 12H As shown, bird's-eye view 1202 and side view 1204 show sixth position 1258 on the touch-sensitive surface of electronic device 100 below tip 276 of stylus 203. Sixth hovering distance 1260 corresponds to the distance between stylus 203 and the touch-sensitive surface of electronic device 100 when stylus 203 is held above sixth position 1258 on electronic device 100.

[0336] like Figure 12HAs shown, electronic device 100 obtains data from stylus 203 indicating that stylus 203 detects tap gesture 1262. In response to detecting tap gesture 1262 and based on determining that sixth hovering distance 1260 satisfies (e.g., reaches or exceeds) first distance threshold 1218, electronic device 100 displays bullet 1264 associated with sixth position 1258 adjacent to text box 1266. Figure 12I , the electronic device 100 displays the bullet 1264 adjacent to the text box 1266 and maintains displaying the visual indicator 1208. For example, in Figure 12I , bullet 1264 and text box 1266 are displayed at a location within user interface 1206 that corresponds to sixth location 1258 (eg, bullet 1264 and text box 1266 are centered about sixth location 1258).

[0337] In some embodiments, when displaying text box 1266, electronic device 100 displays bullet 1266. In some embodiments, electronic device 100 simultaneously displays bullet 1264 and text box 1266. Those skilled in the art will appreciate that electronic device 100 may display one or more of a variety of user interface elements, such as a logo, menu, bullet, object, etc.

[0338] Figures 12J to 12K An example is shown in which the bullet is not displayed according to the hovering distance of the stylus satisfying the second distance threshold. Figure 12J As shown in the bird's-eye view 1202 in FIG, the electronic device 100 displays a user interface 1206 and a visual indicator 1208 indicating that the stylus 203 is being held by the hand of the user 1210. Figure 12J 1268 on the touch-sensitive surface of the electronic device 100. As further shown in FIG. 1269 , the bird's-eye view 1202 and the side view 1204 indicate a seventh position 1268 on the touch-sensitive surface of the electronic device 100 below the tip 276 of the stylus 203. A seventh hovering distance 1270 corresponds to a distance between the stylus 203 and the touch-sensitive surface of the electronic device 100 when the stylus 203 is held above the seventh position 1268 on the electronic device 100.

[0339] like Figure 12J As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1272. In response to detecting the tap gesture 1272 and based on determining that the seventh hovering distance 1270 meets (e.g., reaches or exceeds) the second distance threshold 1220, the electronic device 100 does not display the bullet or text box. Figure 12K , the electronic device 100 does not display the bullet or text box and keeps displaying the visual indicator 1208.

[0340] It is worth noting that the electronic device 100 behaves differently depending on whether the hovering distance of the stylus meets the first distance threshold 1218 or the second distance threshold 1220. That is, depending on whether the first threshold 1218 is met, Figure 12I , the electronic device 100 displays the bullet 1264 adjacent to the text box 1266; and according to satisfying the second distance threshold 1220, Figure 12K , the electronic device 100 does not display either the text box or the bullet points.

[0341] Figures 12L to 12M An example of displaying a menu based on the hovering distance of the stylus meeting the first distance threshold is shown. Figure 12L As shown in the bird's-eye view 1202 in FIG, the electronic device 100 displays a user interface 1206 and a visual indicator 1208 indicating that the stylus 203 is being held by the hand of the user 1210. Figure 12L 12. As further shown in FIG, bird's-eye view 1202 and side view 1204 indicate an eighth position 1274 on the touch-sensitive surface of electronic device 100 below tip 276 of stylus 203. An eighth hovering distance 1276 corresponds to a distance between stylus 203 and the touch-sensitive surface of electronic device 100 when stylus 203 is held above eighth position 1274 on electronic device 100.

[0342] like Figure 12L As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1278. In response to detecting the tap gesture 1278 and based on determining that the eighth hovering distance 1276 meets (e.g., reaches or exceeds) the first distance threshold 1218, the electronic device 100 displays a menu 1280 associated with the eighth position 1274. Figure 12M , the electronic device 100 displays the menu 1280 and keeps displaying the visual indicator 1208. For example, Figure 12M , menu 1280 is displayed at a location within user interface 1206 corresponding to eighth location 1274 (e.g., menu 1280 is centered around eighth location 1274). Menu 1280 includes four visual indicators, of which solid indicator 1280a has focus by default. One of ordinary skill in the art will appreciate that menu 1280 may include any number of visual indicator types with various characteristics, with any one of these visual indicators having focus by default.

[0343] In some embodiments, electronic device 100 may change which indicator is in focus in response to user 1210's hand manipulating stylus 203. For example, in response to obtaining data from stylus 203 indicating a clockwise rotation of stylus 203, electronic device 100 moves through (e.g., changes display of) menu 1280 clockwise, causing focus to change from solid indicator 1280a to dashed indicator 1280b (not shown).

[0344] Figures 12N to 12O An example is shown in which the menu is not displayed according to the hovering distance of the stylus satisfying the second distance threshold. Figure 12N As shown in the bird's-eye view 1202 in FIG, the electronic device 100 displays a user interface 1206 and a visual indicator 1208 indicating that the stylus 203 is being held by the hand of the user 1210. Figure 12N As shown, bird's-eye view 1202 and side view 1204 indicate ninth position 1282 on the touch-sensitive surface of electronic device 100 below tip 276 of stylus 203. Ninth hovering distance 1284 corresponds to the distance between stylus 203 and the touch-sensitive surface of electronic device 100 when stylus 203 is held above ninth position 1282 on electronic device 100.

[0345] like Figure 12N As shown, the electronic device 100 obtains data from the stylus 203 indicating that the stylus 203 detects a tap gesture 1286. In response to detecting the tap gesture 1286 and based on determining that the ninth hovering distance 1284 satisfies (e.g., reaches or exceeds) the second distance threshold 1220, the electronic device 100 does not display the menu. Figure 12O , the electronic device 100 does not display the menu and keeps displaying the visual indicator 1208.

[0346] It is worth noting that the electronic device 100 behaves differently depending on whether the hovering distance of the stylus meets the first distance threshold 1218 or the second distance threshold 1220. That is, depending on whether the first threshold 1218 is met, Figure 12M In the electronic device 100, a menu 1280 is displayed; and according to the second distance threshold 1220 being met, Figure 12O In the example, the electronic device 100 does not display a menu.

[0347] Figure 13A13 is a flow chart illustrating a method 1300 for processing sensor data collected at a stylus according to some embodiments. Method 1300 contemplates electronic device 100 processing sensor data obtained from stylus 203. As shown in block 1302, stylus 203 detects user input. The user input corresponds to one of the various user inputs described herein. As shown in step 1304, stylus 203 provides sensor information to electronic device 100. The sensor information indicates the user input detected by the stylus or the absence of user input detected by the stylus (e.g., when stylus 203 is not being held). Based on the sensor information, electronic device 100 determines (e.g., processes, interprets, translates, decodes, etc.) the input type, as shown in block 1306. The input type corresponds to one of the various input types described herein. As shown in block 1308, electronic device 100 performs an operation based on the input type. The operation corresponds to one of the various operations described herein.

[0348] Figure 13B 13 is a flow chart illustrating another method 1310 for processing sensor data collected at a stylus according to some embodiments. As shown in block 1312, stylus 203 detects user input. This user input corresponds to one of the various user inputs described herein. As shown in block 1314, stylus 203 determines an input type based on the detected user input. In various embodiments, stylus 203 determines (e.g., processes, interprets, translates, decodes, etc.) the input type. In some embodiments, touch interpretation module 477 of stylus 203 determines the input type. Although not shown, in various embodiments, stylus 203 and electronic device 100 jointly (e.g., collaboratively) determine the input type. In other words, stylus 203 and electronic device 100 share processing corresponding to determining the input type. The input type corresponds to one of the various input types described herein. As shown in step 1316, stylus 203 provides information indicating the input type to electronic device 100. In various embodiments, stylus 203 and electronic device 100 jointly determine the input type, and stylus 203 does not provide information indicating the input type. In other words, the method 1310 does not execute block 1316. As shown in block 1318, the electronic device 100 performs an operation based on the input type. The operation corresponds to one of the various operations described in this disclosure.

[0349] 14A to 14C 14 is a flow chart illustrating a method 1400 for changing an application state according to some embodiments. In an electronic device (e.g., Figure 3 The electronic device 300 or Figure 1AMethod 1400 is performed at a portable multifunction device 100 in a computer system. In some embodiments, the touch-sensitive surface and display are combined into a touch screen display (e.g., a mobile phone or tablet computer). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1400 are optionally combined, and / or the order of some operations is optionally changed.

[0350] Transitioning an electronic device from a first application state to a second application state based on sensor data from a stylus reduces the number of inputs required to perform the transition. This reduction in inputs enhances the operability of the electronic device and makes the electronic device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors when operating / interacting with the electronic device), which in turn reduces power usage and wear on the electronic device.

[0351] Method 1400 contemplates an electronic device utilizing data received from a stylus to utilize the myriad types of detectable input at the stylus. The stylus detects input from a user's hand when the user is holding the stylus and detects input when the user is not holding the stylus. Due to the user's complex and varied hand manipulation capabilities, the stylus can detect many types of user input. The stylus provides data indicative of these user inputs to the electronic device. Thus, method 1400 contemplates the electronic device receiving various types of data from the stylus indicative of various user inputs detected at the stylus.

[0352] This enhances the operability of the electronic device and makes the electronic device interface more efficient and robust. As described above, the user can provide various input types to the stylus (e.g., finger manipulation on the stylus, gestures on the stylus, rotational movement of the stylus, etc.). On the other hand, the touch-sensitive surface of the electronic device can receive a single input type (e.g., touch input). A single input type limits the user's ability to interact with the electronic device and may result in erroneous user input. Therefore, the transition of at least some user inputs from the touch-sensitive surface of the electronic device to the stylus provides a more efficient user interface with the electronic device and can reduce the number of erroneous inputs that occur at the electronic device. In addition, this transition to less touch input at the touch-sensitive surface of the electronic device reduces wear and tear on the electronic device and power usage. This extends the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently. For battery-powered electronic devices, enabling the user to input less input on the touch-sensitive surface of the electronic device saves power and increases the time interval between two battery charges of the electronic device.

[0353] Relative to Figure 14AWhen the electronic device is in the first state, the electronic device obtains (1402) information about the current state of the stylus via the communication interface. For example, the information corresponds to sensor data collected by a magnetometer of the stylus, an accelerometer of the stylus, a capacitive touch element or a touch-sensitive surface on the stylus shaft, etc. For example, the sensor data is transmitted / received via a Bluetooth connection, an IEEE802.11x connection, etc.

[0354] For example, reference Figure 7B , the electronic device 100 receives data from the stylus 203 indicating that the stylus is being held by the hand of the user 702. Figure 7I In FIG, the electronic device 100 receives data from the stylus 203 indicating that the stylus is not held by the hand of the user 702. Figure 7D , device 100 receives data from stylus 203 indicating that stylus 203 is undergoing rotational movement 722 by the hand of user 702. Figure 7F , the electronic device 100 receives data from the stylus 203 indicating that the stylus 203 is detecting a tap gesture 728 from the hand of the user 702 .

[0355] In some embodiments, when the electronic device is in the first state, the electronic device operates in an inactive mode (1404). Operating the electronic device in the inactive mode when in the first state enhances the operability of the electronic device and makes the electronic device more efficient, which extends the battery life of the electronic device. For example, the display of the electronic device is off in the first state and does not display a user interface. For example, Figure 7K As shown, the electronic device 100 displays the lock screen 736 and provides limited functions, thereby resulting in less power consumption. Figure 7V As shown, the electronic device 100 displays the home screen 746 and has no active foreground applications running, resulting in less power consumption.

[0356] In some embodiments, when the electronic device is in the first state, the electronic device displays (1406) a first interface on the display. For example, the first interface corresponds to a lock screen. For example, Figure 7K As shown, when the electronic device 100 is operated in the first state when the stylus 203 is not held by the user, the electronic device displays the lock screen 736 (for example, the first interface). For another example, the first interface corresponds to the home screen 746, such as Figure 7V As shown. For another example, the first interface corresponds to the drawing interface 706, as shown Figure 7A shown.

[0357] In some embodiments, at least a portion of the information about the current state of the stylus corresponds (1408) to touch sensor data from one or more touch sensors on the stylus. Having some of the information about the current state of the stylus correspond to the stylus touch sensor data enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, Figure 7A As shown, the electronic device 100 receives data (eg, information) from the stylus 203 indicating that the user is not holding the stylus 203 .

[0358] Based on determining that the user is holding the stylus based on information about the current state of the stylus, the electronic device displays (1410) on the display a visual indication that the electronic device is in a second state different from the first state. For example, one or more sensors on the stylus (such as a magnetometer, an accelerometer, and a capacitive touch element or touch-sensitive surface on the stylus shaft) are used to make this determination. For another example, in some embodiments, to avoid false positives, the sensor data indicates that the user is holding the stylus based on two or more inputs (e.g., accelerometer, capacitive touch) that indicate that the user is holding the stylus. For another example, the visual indication is a representation of the stylus, such as an icon, a pen tip, a picture of an icon, etc.

[0359] For example, 7A to 7C As shown, the electronic device 100 is Figure 7A The first state shown in the transition to Figure 7C The second state shown in Figure 7B The user is holding the stylus 203, Figure 7C , the electronic device 100 displays a visual indicator 712 ( Figure 7A not shown) to indicate that the electronic device is in the second state.

[0360] For example, Figures 7K to 7M As shown, the electronic device 100 is Figure 7K The first state shown in the transition to Figure 7M The second state shown in Figure 7L The user is holding the stylus 203, Figure 7M , the electronic device 100 displays a visual indicator 712 ( Figure 7K not shown) to indicate that the electronic device 100 is in the second state.

[0361] In some embodiments, when the electronic device is in the second state, the electronic device operates in the active mode (1412). For example, in the second state, the display of the electronic device is turned on and displays the interface. Figure 7CAs shown, when the electronic device 100 operates in the second state when the stylus 203 is held by the user, the electronic device displays an enlarged canvas 710 and a visual indicator 712 .

[0362] In some embodiments, when the electronic device is in the second state, the electronic device displays (1414) on the display a second interface that is different from the first interface and a visual indication that the electronic device is in the second state. Displaying different interfaces based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second interface corresponds to a home screen or an application interface. For example, Figures 7K to 7M As shown, the electronic device 100 changes from the first state to the second state and Figure 7M An application interface including an enlarged canvas 710 and a visual indicator 712 is shown in FIG.

[0363] In some embodiments, the visual indication corresponds to (1416) a drawing canvas associated with a drawing application. Displaying a visual indication indicating that the electronic device is in the second state based on data received from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, Figure 7M As shown, the electronic device 100 displays an application interface including an enlarged canvas 710 and a visual indicator 712 when operating in the second state.

[0364] In some embodiments, the visual indication corresponds to (1418) an application icon associated with a drawing application. Displaying a visual indication indicating that the electronic device is in the second state based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, when the application icon is selected, the electronic device runs (e.g., executes) and displays the drawing application in the foreground. For another example, when the stylus is no longer held, the electronic device stops displaying the application icon.

[0365] In some embodiments, the electronic device (1420): displays a drawing application interface on a display, and stops displaying one or more user interface elements associated with the drawing application interface on the display. Stopping displaying user interface elements based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, stopping displaying one or more user interfaces corresponds to removing a displayed tool group, such as a set of marking tools or a color palette. For example, in response to detecting that a drawing application interface is being drawn on the stylus, the user can stop displaying the user interface elements. Figure 7B The stylus 203 is being held. Figure 7C , the electronic device 100 stops displaying the navigation area 704 , the canvas area 706 and the toolbar area 708 .

[0366] In some embodiments, the visual indication corresponds to (1422) a first marking tool, wherein the first marking tool is the currently active marking tool. For example, the visual indication corresponds to an image, icon, text, etc. of the current marking tool. Displaying a visual indication indicating that the electronic device is in the second state based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, Figure 7C As shown, the electronic device 100 displays a visual indicator 712 including a marker pen icon 716 corresponding to a marking tool. Continuing with this example, Figure 7C The visual indicator 712 shown corresponds to a color (eg, hue, shade, etc.) to indicate that the mark made on the enlarged canvas 710 will have that color. One of ordinary skill in the art will appreciate that the visual indicator 712 may include any form of indicator.

[0367] Relative to Figure 14B : Based on determining that the user is not holding the stylus, the electronic device maintains (1424) the electronic device in the first state. For example, the electronic device receives from the stylus an absence or lack (e.g., below a threshold) of data indicating that the user is not holding the stylus. For another example, to conserve battery life of the stylus and / or the device, the stylus provides data to the electronic device in response to the stylus detecting a specific (e.g., significant) touch input (such as a gesture input (e.g., a tap, a swipe, etc.) to the stylus, a manipulation of the stylus itself (e.g., a roll, a rotation, etc.), etc.

[0368] For another example, the electronic device maintains the electronic device in the first state without displaying a visual indication indicating that the electronic device is in the second state. 7A to 7B As shown, the electronic device 100 maintains itself in the first state and does not display Figure 7CA visual indicator 712 in the diagram indicates that the electronic device 100 is in the second state.

[0369] In some embodiments, the electronic device (1426) displays a drawing application interface on a display, and displays one or more user interface elements associated with the drawing application interface on the display. For example, the one or more user interface elements correspond to a tool group including a drawing implementation, a drawing tool, a color palette, etc. For example, Figure 7A As shown, the electronic device 100 displays user interface elements in the toolbar area 708, including drawing tools (eg, marker, pencil, ruler) and a color palette.

[0370] In some embodiments, when the electronic device is in the second state, the electronic device (1428): obtains updated information about the current state of the stylus via the communication interface, wherein the updated information indicates that the user is no longer holding the stylus; in response to obtaining the updated information, the electronic device stops displaying the second interface and the visual indication on the display; and redisplays the first interface on the display. Stopping and redisplaying the interface based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first interface corresponds to a lock screen, and the second interface corresponds to a lock screen such as Figures 7M to 7O In this example, in response to detecting that the stylus 203 is no longer held, Figures 7M to 7N The electronic device 100 stops displaying the enlarged canvas 710 and the visual indicator 712, and Figure 7O Redisplay Figure 7K For example, the first interface corresponds to the lock screen displayed in Figure 7J The first drawing interface shown, and the second interface corresponds to Figure 7H The second drawing interface shown.

[0371] In some embodiments, when the electronic device is in the second state, the electronic device (1430): obtains first finger manipulation data from the stylus via the communication interface, wherein the first finger manipulation data represents one or more finger manipulation inputs received at the stylus; and in response to obtaining the first finger manipulation data: changes the currently active marking tool to a second marking tool; and updates the visual indicator to correspond to the second marking tool. Changing the active marking tool and updating the visual indicator based on the data received from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, one or more finger manipulation inputs correspond to gestures made on the touch-sensitive surface of the stylus, such as swiping down, swiping up, tapping, etc. For another example, one or more finger manipulation inputs correspond to manipulating the stylus, such as rolling the stylus shaft in a clockwise or counterclockwise manner, rotating the stylus in a clockwise or counterclockwise manner, etc. For another example, the finger manipulation data corresponds to data collected by the stylus's magnetometer, the stylus's accelerometer, and / or a capacitive touch element or touch-sensitive surface on the stylus shaft.

[0372] For example, finger manipulation data is transmitted / received via a Bluetooth connection, an IEEE 802.11x connection, etc. In some embodiments, the finger manipulation data includes information about the movement of the finger on the stylus or the movement of the stylus relative to the user's finger (e.g., data indicating how the finger and / or stylus moved). In some embodiments, the finger manipulation data includes a processed representation of the movement of the finger on the stylus or the movement of the stylus relative to the user's finger. The processed representation may indicate a gesture or manipulation performed at the stylus, such as a swipe or rotation gesture, optionally including information indicating the direction and / or magnitude of the gesture or movement.

[0373] For example, in response to receiving Figure 7F The electronic device 100 changes the visual indicator 712 from Figure 7F Marker 714 in the update is Figure 7G 730. In another example, the user can scan through a column of marking tools by rolling the stylus. For example, the column of marking tools corresponds to a customized column of tools, a default column of tools, recently and / or frequently used tools, etc.

[0374] refer to Figure 14CIn some embodiments, when the electronic device is in the second state, the electronic device (1432): obtains first finger manipulation data from the stylus via the communication interface, wherein the first finger manipulation data represents an upward swipe gesture received on the stylus; and in response to obtaining the first finger manipulation data, displays a color palette adjacent to the visual indication on the display. Displaying a color palette based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the color palette corresponds to a user interface area with a plurality of different colors to choose from, such as a color wheel, a grid with different color areas, a column of colors, etc. For example, in Figure 7C , the electronic device 100 receives data indicating a detected downward swipe gesture 718 from the stylus 203, and Figure 7D The electronic device 100 displays the color palette 720 adjacent to the visual indicator 712. Continuing with this example, in response to receiving Figure 7D In response to the data indicating the rotational manipulation (e.g., rolling) of the stylus 203, the electronic device 100 changes the visual indicator 712 from Figure 7D The marker 714 with solid tip 716 in the embodiment is updated to Figure 7E The marker 714 with the striped tip 724 is shown in FIG.

[0375] In some embodiments, when the electronic device is in the second state, the electronic device (1434): obtains second finger manipulation data from the stylus via the communication interface, wherein the second finger manipulation data represents a rotation gesture received at the stylus; and in response to obtaining the second finger manipulation data: changes the color associated with the currently active marking tool; and updates the visual indication to correspond to the color. Updating the color and visual indication associated with the active marking tool based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, to change the color and update the visual indication at the electronic device, the stylus is rotated by greater than a threshold angular distance and / or greater than a threshold angular velocity. For example, the electronic device displays a first indicator (e.g., a star) next to the selected color and / or displays a second indicator (e.g., a ring) around the selected color. For example, the electronic device displays a color icon that changes color. In this example, the electronic device increases the size of the icon corresponding to the currently selected color.

[0376] For example, in response to Figure 7D In the embodiment, data indicating a rotational manipulation 722 (e.g., a roll) of the stylus 203 is received, Figure 7E, the electronic device 100 stops displaying the solid fill indicator 720a having focus and displays the diagonally filled indicator 720b having focus. For another example, rolling the stylus 203 in one direction (e.g., clockwise) moves the focus downward (e.g., from 720a to 720b), while rolling the stylus 203 in another direction (e.g., counterclockwise) moves the focus downward (e.g., from 720b to 720a). One of ordinary skill in the art will appreciate that the nature of the stylus's rotation (e.g., direction) may have different effects on the user interface.

[0377] In some embodiments, when the electronic device is in the second state, the electronic device (1436): obtains third finger manipulation data from the stylus via the communication interface, wherein the third finger manipulation data represents a downward swipe gesture received at the stylus; and in response to obtaining the third finger manipulation data, removes the display of the palette on the display. Removing the palette based on data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, in response to Figure 7E In the case where data indicating an upward swipe gesture 726 is received at the stylus 203, Figure 7F , the electronic device 100 stops displaying the color palette 720.

[0378] It should be understood that 14A to 14C The specific order in which the operations are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations can be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0379] References 14A to 14C The operation is optionally performed by Figure 1A to Figure 1B1406, 1414, 1420, and 1426 are implemented by, for example, an event classifier 170, an event identifier 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects contact (or proximity) on the touch-sensitive display 112, and an event distributor module 174 transmits the event information to the application 136-1. The corresponding event identifier 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact (or proximity) at the first position on the touch-sensitive surface (or whether the rotation of the electronic device) corresponds to a predefined event or sub-event, such as a selection of an object on a user interface or a rotation of the electronic device from one orientation to another. When a corresponding predefined event or sub-event is detected, the event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update the application internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand that based on Figure 1A-1B How the components depicted in FIG. 1 can implement other processes.

[0380] Note that the details of the process described above with respect to method 1400 also apply in a similar manner to the other methods described herein (e.g., 1500, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700). For example, the stylus, stylus state, touch-sensitive surface, display, and communication interface described above with reference to method 1400 optionally have one or more of the properties of the stylus, stylus state, touch-sensitive surface, display, and communication interface described herein with reference to the other methods described herein (e.g., 1500, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700).

[0381] Figures 15A to 15B 15 is a flow chart illustrating a method 1500 of changing the functionality of a stylus according to some embodiments. In an electronic device (e.g., Figure 3 The electronic device 300 or Figure 1AMethod 1500 is performed at a portable multifunction device 100 in a computer system. In some embodiments, the touch-sensitive surface and display are combined into a touch screen display (e.g., a mobile phone or tablet computer). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1500 are optionally combined, and / or the order of some operations is optionally changed.

[0382] Changing the stylus function based on sensor data from the stylus reduces the number of inputs required to perform the change in stylus function. This reduction in inputs enhances the operability of the electronic device and makes the electronic device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors when operating / interacting with the electronic device), which in turn reduces power usage and wear on the electronic device.

[0383] Method 1500 contemplates an electronic device utilizing data received from a stylus in order to utilize the myriad detectable input types at the stylus and the orientation of the stylus relative to the electronic device. For example, the electronic device receives data from the stylus indicating how the stylus is being held (e.g., a grip arrangement). Due to the complex and varied hand manipulation capabilities of a user, the stylus can detect many types of user input. The stylus provides data indicative of these user inputs to the electronic device. Thus, method 1500 contemplates the electronic device receiving various types of data from the stylus indicating various user inputs detected at the stylus. Additionally, method 1500 contemplates the received data including information regarding the orientation of the stylus relative to the electronic device.

[0384] This enhances the operability of the electronic device and makes the electronic device interface more efficient and robust. As described above, the user can provide various input types to the stylus (e.g., gripping arrangements) and can change the orientation of the stylus (e.g., which end of the stylus is in contact with the electronic device). On the other hand, current systems envision that the touch-sensitive surface of the electronic device receives a single type of input: touch input from the user's finger. A single input type limits the user's ability to interact with the electronic device and can lead to erroneous user input. Therefore, converting at least some inputs from finger touch inputs to the aforementioned stylus inputs provides a more efficient user interface with the electronic device and can reduce the number of erroneous inputs that occur at the electronic device. In addition, this conversion to fewer finger touch inputs at the touch-sensitive surface of the electronic device reduces wear and tear on the electronic device and power usage. This extends the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently. For battery-powered electronic devices, enabling the user to input fewer inputs on the touch-sensitive surface of the electronic device saves power and increases the time interval between two battery charges of the electronic device.

[0385] refer to Figure 15A , the electronic device 100 detects (1502) input from a stylus on a touch-sensitive surface of the electronic device. Figures 8A to 8B , the electronic device 100 detects input 810 (eg, a drawing stroke or mark) corresponding to a contact vector between the stylus 203 and the touch-sensitive surface of the electronic device 100 .

[0386] In some embodiments, the electronic device (1504): obtains sensor data from the stylus; and determines a grip arrangement that characterizes a user's manipulation of the stylus based on the sensor data from the stylus, wherein the grip arrangement is determined during the detection of input. Obtaining grip arrangement data to affect operations performed by the electronic device enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the sensor data corresponds to a grip arrangement detected by a magnetometer of the stylus 203, an accelerometer of the stylus 203, a capacitive touch element on the pen shaft of the stylus 203, or a touch-sensitive surface (e.g., such as Figure 2 and Figures 5A to 5B In some embodiments, the sensor data is transmitted from the stylus 203 to the electronic device 100 via a Bluetooth connection, an IEEE 802.11x connection, or the like.

[0387] For example, the user holds the stylus 203 near the tip 276, with the tip 276 pointing toward the electronic device 100 to form an indicia associated with a pen marking tool (e.g., as Figures 8A to 8B As another example, the user holds the stylus 203 near the end 277 opposite the tip 276, with the tip 276 pointing toward the electronic device 100 to form a mark associated with a paint brush marking tool (e.g., as shown in FIG. Figures 8C to 8D As another example, the user holds the stylus 203 near the end 277 opposite the tip 276, where the tip 276 points outward from the electronic device 100 to form an indicia associated with an eraser marking tool (e.g., as shown in FIG. Figures 8E to 8F As another example, the user holds the stylus 203 near the tip 276, where the tip 276 points outward from the electronic device 100 to form an indicia associated with a smear or spray marking tool (e.g., as Figures 8G to 8H shown).

[0388] For example, reference Figures 8A to 8B, the electronic device 100 determines, based on sensor data from the stylus 203, that the user is holding the stylus 203 in his / her hand 802 in a first grip arrangement 815, which corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing toward the electronic device 100), with the fingers of the hand 802 proximate the tip of the stylus 203. In this example, reference Figures 8A to 8B , when input 810 (e.g., drawing strokes or marks) is detected via the touch-sensitive surface of the electronic device 100, the electronic device 100 determines that the user is holding the stylus 203 in his / her hand 802 in a first gripping arrangement 815. For another example, referring to Figures 8C to 8D , the electronic device 100 determines, based on sensor data from the stylus 203, that the user is holding the stylus 203 in his / her hand 802 in a second gripping arrangement 835, which corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing toward the electronic device 100), with the fingers of the hand 802 proximate the end 277 of the stylus 203 opposite the tip 276 of the stylus 203.

[0389] In some embodiments, a grip arrangement is determined (1506) based on at least one of a grip style, a grip position, or an orientation of the stylus relative to a reference frame. Obtaining grip arrangement data to influence operations performed by the electronic device enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, a grip style corresponds to the position of contact points on the stylus relative to each other (e.g., the position of different fingers). For example, a grip position corresponds to the position of contact points on the stylus relative to the stylus (e.g., the end of a finger near the stylus). For example, the orientation of the stylus corresponds to the position of the stylus relative to the electronic device, gravity, and / or the Earth's magnetic field.

[0390] In some embodiments, the grip style is determined based at least in part on the touch input in contact with the stylus, such as how many fingers are on the stylus, which fingers are on the stylus, a pinch grip, or a first grip. In some embodiments, the grip position is determined based at least in part on the position of the grip (e.g., the touch input) relative to the stylus. For example, the grip position includes the position of the touch input on the stylus (e.g., near the tip of the stylus or near the base of the stylus). In some embodiments, the grip style is determined based at least in part on the orientation of the stylus relative to the electronic device (e.g., upright or inverted).

[0391] For example, reference Figures 8A to 8B, the electronic device 100 determines a first grip arrangement 815. The first grip arrangement 815 corresponds to holding the stylus 203 in an upright orientation (e.g., the tip 276 of the stylus 203 points toward the electronic device 100), wherein the fingers of the hand 802 are close to the tip 276 of the stylus 203. Figures 8E to 8F , the electronic device 100 determines a third gripping arrangement 855. The third gripping arrangement 855 corresponds to holding the stylus 203 in an inverted orientation (e.g., with the tip 276 of the stylus 203 pointing outward from the electronic device 100) and near the end 277 of the stylus 203 opposite the tip 276 of the stylus 203. Figures 8G to 8H , the electronic device 100 determines a fourth grip arrangement 875 that corresponds to holding the stylus 203 in an inverted orientation (eg, with the tip 276 of the stylus 203 pointing outward from the electronic device 100 ) and proximate to the tip 276 of the stylus 203 .

[0392] In some embodiments, at least a portion of the sensor data corresponds to (1508) touch sensor data obtained from one or more touch sensors on the stylus. Obtaining sensor data from the stylus to affect operations performed by the electronic device enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, a grip style and / or grip position is determined based on a portion of the sensor data indicating the location of a touch detected on the touch-sensitive surface of the stylus. For example, with reference to Figures 8A to 8B , the electronic device 100 determines a first grip arrangement 815 based on sensor data received from the stylus 203 indicating that the stylus 203 is being held in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing toward the electronic device 100), wherein the fingers of the hand 802 hold the stylus 203 near the tip 276 of the stylus 203.

[0393] In response to detecting the input and based on determining that the stylus is being held in a first grip arrangement, the electronic device makes (1510) a first change to content displayed on the display, wherein the first grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus. For example, a sensor at the stylus (e.g., a capacitive touch sensor, an accelerometer, a magnetometer, or a gyroscope) detects the first grip arrangement. For example, the first change corresponds to drawing a line, spraying, erasing, etc. with a paint brush / pencil / spray gun, etc. For example, the first change is associated with a first marking tool corresponding to the first grip arrangement.

[0394] For example, reference Figures 8A to 8B, the electronic device 100 detects the input 810 and determines a first grip arrangement 815 based at least in part on the sensor data detected by the stylus 203. Figure 8B , the electronic device 100 makes a first change 820 (eg, a stroke or mark) to the user interface 800 based on the input 810 .

[0395] In some embodiments, making the first change includes (1512) displaying a first user element based on a first marking tool corresponding to the first grip arrangement. Displaying the user element based on the grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first grip arrangement (e.g., upright stylus orientation, grip position near the end of the stylus relative to the electronic device) invokes a writing marking tool (e.g., pencil, marker, etc.). For example, with reference to Figures 8A to 8B , the electronic device 100 makes a first change 820 based on a first marking tool (eg, a felt-tip marker) corresponding to the first gripping arrangement 815 .

[0396] In some embodiments, making the first change includes (1514) changing an existing markup displayed on the display based on a first marking tool corresponding to the first grip arrangement. Changing the existing markup based on the grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first grip arrangement (e.g., an inverted stylus orientation, a grip position near the bottom of the stylus relative to the electronic device) invokes an eraser marking tool.

[0397] For example, reference Figures 8E to 8F , the electronic device 100 detects the input 810 and determines a third grip arrangement 855 based at least in part on the sensor data detected by the stylus 203. The third grip arrangement 855 corresponds to a third marking tool (e.g., an eraser), as indicated by the eraser indicator 852. Figure 8F , the electronic device 100 makes a change to the existing mark 804 by displaying a white stroke / mark 860 (eg, an erase) corresponding to the input 810 at an appropriate location over a portion of the existing mark 804 .

[0398] In some embodiments, a first grip arrangement is detected (1516) based on detecting that the stylus is in an upright orientation of the stylus and detecting a touch input near a first end of the stylus, and making a first change includes displaying a stroke on the display based on a writing instrument corresponding to the first grip arrangement. Making changes to displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the upright orientation is based on a physical property of the stylus, such as the tip of the stylus pointing upward. For example, the first end of the stylus corresponds to the writing tip of the stylus. For example, the first grip arrangement corresponds to a pencil, pen, marker, etc.

[0399] For example, reference Figures 8A to 8B , the electronic device 100 determines based on the sensor data from the stylus 203 that the user is holding the stylus 203 in an upright orientation (e.g., the tip of the stylus 203 is pointing toward the electronic device 100), wherein the fingers of the hand 802 are close to the tip of the stylus 203. Therefore, in response to the input 810, Figure 8B , the electronic device 100 makes a first change 820 corresponding to displaying a stroke on the display according to a marking tool (eg, a felt tip marker) indicated by the indicator 812 .

[0400] refer to Figure 15B In response to detecting the input and based on determining that the stylus is being held in a second grip arrangement that is different from the first grip arrangement, the electronic device makes (1518) a second change to the content displayed on the display, wherein the second grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus, wherein the second change to the content displayed on the display is different from the first change to the content displayed on the display. This can reduce wear and battery consumption of the electronic device because the user interface is changed without requiring additional touches to the touch-sensitive surface of the electronic device. For example, a sensor at the stylus (e.g., a capacitive touch sensor, an accelerometer, a magnetometer, or a gyroscope) detects the second grip arrangement. For example, the second change corresponds to drawing a line, spraying, erasing, etc. with a paint brush / pencil / sprayer, etc. For example, the first second is associated with a second marking tool corresponding to the second grip arrangement.

[0401] For example, reference Figures 8C to 8D Based at least in part on the sensor data received from the stylus 203, the electronic device 100 determines that the stylus is held in a second gripping arrangement 835. The second gripping arrangement 835 is different from Figure 8B Thus, in response to detecting input 810, Figure 8D , the electronic device 100 makes a second change 840 (eg, a stroke or a mark) to the user interface 800. The second change 840 is different from Figure 8B The first change in 830.

[0402] In some embodiments, making the second change includes (1520) displaying a second user element based on a second marking tool corresponding to the second grip arrangement. Changing the displayed content based on the grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second grip arrangement (e.g., upright stylus orientation, grip position near the top of the stylus relative to the electronic device) invokes a painting marking tool (e.g., a paint brush, etc.). For example, with reference to Figures 8A to 8B , the electronic device 100 determines a first grip arrangement 815. The electronic device 100 determines that the first grip arrangement 815 corresponds to a felt tip marker marking tool and displays an indicator 812 indicating the marking tool. Therefore, in response to detecting the input 810, the electronic device 100 makes a first change 820 corresponding to a felt tip marker stroke. For another example, referring to Figures 8C to 8D , the electronic device 100 determines a second grip arrangement 835. The electronic device 100 determines that the second grip arrangement 835 corresponds to a paintbrush marking tool and displays an indicator 832 indicating the marking tool. Thus, in response to detecting the input 810, the electronic device 100 makes a second change 840 corresponding to a paintbrush stroke.

[0403] In some embodiments, making the second change includes (1522) changing an existing markup displayed on the display based on a second marking tool corresponding to the second grip arrangement. Changing the existing markup based on the grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second grip arrangement (e.g., an inverted stylus orientation, a grip position near the top of the stylus relative to the electronic device) invokes a scribble marking tool.

[0404] For example, reference Figures 8E to 8F , the electronic device 100 determines a third grip arrangement 855. The electronic device 100 determines that the third grip arrangement 855 corresponds to an eraser marking tool and displays an indicator 852 indicating the marking tool. Thus, in response to detecting the input 810, the electronic device 100 changes the existing mark 804 by displaying a white stroke 860 (e.g., erasing) over the existing mark 804.

[0405] In some embodiments, a second grip arrangement is detected (1524) based on detecting that the stylus is in an inverted orientation of the stylus and detecting a touch input near a second end of the stylus that is different from the first end of the stylus, and making a second change includes removing an existing indicia displayed on the display based on an eraser tool corresponding to the second grip arrangement. Changing displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the inverted orientation is based on a physical property of the stylus, such as the tip of the stylus pointing downwardly toward the electronic device. For example, the second end corresponds to an eraser tip of the stylus or an end opposite the writing tip of the stylus.

[0406] For example, reference Figures 8E to 8F , electronic device 100 determines, based at least in part on data received from stylus 203, that stylus 203 is being held in a third grip arrangement 855. Third grip arrangement 855 corresponds to stylus 203 being held in an inverted orientation (e.g., with tip 276 of stylus 203 pointing outward from electronic device 100) and proximate end 277 of stylus 203 opposite tip 276 of stylus 203. Electronic device 100 determines that third grip arrangement 855 corresponds to an eraser marking tool and displays indicator 852 indicating the marking tool. Thus, in response to detecting input 810, electronic device 100 changes existing marker 804 by displaying a white stroke 860 (e.g., erasing) over existing marker 804.

[0407] In some embodiments, a second grip arrangement is detected (1526) based on detecting that the stylus is in an upright orientation of the stylus and detecting a touch input near a second end of the stylus, and making a second change includes displaying a stroke based on a drawing tool corresponding to the first grip arrangement. Changing displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second end of the stylus corresponds to an eraser tip of the stylus or an end opposite the writing tip of the stylus. For example, the drawing tool corresponds to a pencil, pen, marker, etc.

[0408] For example, reference Figures 8C to 8D, the electronic device 100 determines a second grip arrangement 835. The second grip arrangement 835 corresponds to holding the stylus 203 in an upright orientation (e.g., with the tip 276 of the stylus 203 pointing toward the electronic device 100), with the fingers of the hand 802 proximate the end 277 of the stylus 203 opposite the tip 276 of the stylus 203. The electronic device 100 determines that the second grip arrangement 835 corresponds to a paint brush, as indicated by the indicator 832. Therefore, in response to detecting the input 810, the electronic device 100 makes a second change 840 corresponding to displaying a paint brush stroke 840.

[0409] In some embodiments, a second grip arrangement is detected (1528) based on detecting that the stylus is in an inverted orientation of the stylus and detecting a touch input near the first end of the stylus, and making a second change includes removing an existing logo displayed on the display based on a smudge tool corresponding to the second grip arrangement. Changing displayed content based on grip arrangement data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first end of the stylus corresponds to the writing tip of the stylus.

[0410] It should be understood that FIG. 15A to FIG. 15B The specific order in which the operations are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations can be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0411] References FIG. 15A to FIG. 15B The operation is optionally performed by Figure 1A to Figure 1B15. For example, the detecting operation 1502, the obtaining and determining operation 1504, and the making operations 1510 and 1518 are optionally implemented by the event classifier 170, the event recognizer 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects contact (or proximity) on the touch-sensitive display 112, and the event distributor module 174 transmits the event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact (or proximity) at the first position on the touch-sensitive surface (or whether the rotation of the electronic device) corresponds to a predefined event or sub-event, such as the selection of an object on the user interface, or the rotation of the electronic device from one orientation to another orientation. When the corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update the application internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand that based on Figure 1A-1B How the components depicted in FIG. 1 can implement other processes.

[0412] Note that the details of the process described above with respect to method 1500 also apply in a similar manner to the other methods described herein (e.g., 1400, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700). For example, the stylus, grip arrangement, display, touch-sensitive surface, and communication interface described above with reference to method 1500 optionally have one or more of the properties of the stylus, grip arrangement, display, touch-sensitive surface, and communication interface described herein with reference to the other methods described herein (e.g., 1400, 1600, 1700, 1800, 1900, 2400, 2500, 2600, 2700).

[0413] 16A to 16B 16 is a flow chart illustrating a method 1600 of modifying touch input functionality according to some embodiments. In an electronic device (e.g., Figure 3 The electronic device 300 or Figure 1AMethod 1600 is performed at a portable multifunction device 100 in a computer system. In some embodiments, the touch-sensitive surface and display are combined into a touch screen display (e.g., a mobile phone or tablet computer). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1600 are optionally combined, and / or the order of some operations is optionally changed.

[0414] Modifying touch input functionality based on sensor data from the stylus reduces the number of inputs required to perform changes to the stylus functionality. This reduction in inputs enhances the operability of the electronic device and makes the electronic device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors when operating / interacting with the electronic device), which in turn reduces power usage and wear on the electronic device.

[0415] Method 1600 contemplates an electronic device utilizing data received from a stylus and / or the lack of data received from the stylus. The stylus detects input from a user's hand when the user is holding the stylus and detects input when the user is not holding the stylus. This enhances the operability of the electronic device and makes the electronic device interface more efficient and robust. That is, the functionality of touch input to the electronic device depends on whether the stylus is being held by the user, as indicated by the data received from the stylus and / or the lack of data received from the stylus. In other words, the electronic device can perform multiple functions in response to detecting a particular touch input to the touch-sensitive surface of the electronic device.

[0416] Thus, method 1600 implements a richer set of functionality than current systems in which an electronic device performs a single operation in response to detecting a specific touch input to a touch-sensitive surface of the electronic device. The single operation envisioned in current systems limits the user's ability to interact with the electronic device and may result in erroneous user input. Therefore, extending the functionality generated by a specific touch input based on whether the stylus is held provides a more efficient user interface with the electronic device and can reduce the number of erroneous inputs that occur at the electronic device. Additionally, this reduces wear and tear on the electronic device and power usage. This extends the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently. For battery-powered electronic devices, enabling the user to input fewer inputs on the touch-sensitive surface of the electronic device saves power and increases the time interval between charges of the electronic device's battery.

[0417] refer to Figure 16A , the electronic device 100 detects (1602) a touch input on the touch-sensitive surface. For example, the touch input corresponds to a finger touch input, such as a tap, a swipe, a gesture, etc. For example, referring to Figure 9A, the electronic device 100 detects a left swipe gesture 908 made by a finger of the hand of the user 902. For another example, referring to Figure 9I , the electronic device 100 detects a looping gesture 916 (eg, a lasso gesture) surrounding the content 904 .

[0418] In some embodiments, the electronic device obtains (1604) sensor data from the stylus via a communication interface, and at least a portion of the sensor data corresponds to touch sensor data from one or more touch sensors on the stylus. Obtaining sensor data from the stylus to affect touch input functionality enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the sensor data corresponds to a capacitive touch element or touch-sensitive surface on the stylus pen. For example, the sensor data is transmitted / received via a Bluetooth connection, an IEEE 802.11x connection, etc. For example, reference Figure 9H , the electronic device 100 obtains sensor data from the stylus 203 indicating that the stylus 203 is not held by the hand of the user 902. Figure 9A , the electronic device 100 obtains sensor data from the stylus 203 indicating that the stylus 203 is being held by the hand of the user 902 .

[0419] In response to detecting a touch input on the touch-sensitive surface and based on determining that sensor data obtained from the stylus via the communication interface indicates that the stylus is being held by a user, the electronic device performs (1606) a first operation in response to the touch input. For example, the electronic device determines that the stylus is being held based on data obtained from the stylus indicating that the stylus is detecting one or more fingers in contact with the stylus. For example, the first operation is performed based on the directionality, velocity, acceleration, displacement, etc. of the touch input.

[0420] For example, reference Figures 9A to 9B , the electronic device 100 performs an undo / erase operation on the content 904 in response to detecting the left swipe gesture 908 and determining that the stylus 203 is being held by the hand of the user 902 based on data obtained from the stylus 203. For another example, referring to Figures 9F to 9H , electronic device 100 performs selection and movement operations on content 904. In this example, in response to detecting looping gesture 916 and subsequent drag gesture 922 relative to content 904 and in accordance with a determination that stylus 203 is being held by a hand of user 902 based on data obtained from stylus 203, electronic device 100 selects and moves content 904 in accordance with drag gesture 922.

[0421] In some embodiments, the electronic device performs (1608) a first operation that includes modifying one or more pre-existing user interface elements displayed on the display. Modifying pre-existing user interface elements based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, modifying the user elements includes copying and pasting identifiers / objects / text, cutting and pasting identifiers / objects / text, undoing and restoring identifiers / objects / text, erasing identifiers / objects / text, or a combination thereof. For example, the first operation corresponds to lassoing / selecting identifiers / objects / text in order to move them to a different location (e.g., the first operation corresponds to a cut and paste operation or a copy and paste operation) and / or in order to change their appearance.

[0422] For example, reference Figures 9A to 9C , the first operation corresponds to the electronic device 100 performing an erase / undo operation with respect to the content 904 in response to detecting the left swipe gesture 908. Continuing with this example, the first operation corresponds to the electronic device 100 performing a restore operation with respect to the content 904 in response to detecting the right swipe gesture 910. For another example, referring to Figures 9F to 9G , the first operation corresponds to the electronic device 100 performing a selection operation with respect to the content 904 in response to detecting the ring gesture 916. Figures 9K to 9L The first operation corresponds to the electronic device 100 responding to the detection Figure 9K The right swipe gesture 938 in Figure 9L A selection operation is shown being performed with respect to selected text 940 .

[0423] In some embodiments, based on determining that sensor data obtained from the stylus via the communication interface indicates that the stylus is being held by a user with a different hand than the hand corresponding to the touch input, the electronic device performs (1610) a second operation in response to the touch input. Performing the second operation based on the sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation corresponds to an airbrush can operation, an ink blot operation, a grid line placement operation, a ruler operation, etc. In some embodiments, the electronic device determines the difference between the hand holding the stylus and the hand making the touch input based on proximity (e.g., nearness). For example, the electronic device determines that the stylus is being held by the user with a different hand than the hand corresponding to the touch input based on determining that the stylus is at least a threshold distance from the electronic device. For example, the electronic device determines the proximity of the stylus to the electronic device based on data received from the stylus, sensor data generated at the electronic device, or a combination thereof.

[0424] In response to detecting a touch input on the touch-sensitive surface and based on determining that the stylus is not being held by the user, the electronic device performs (1612) a second operation in response to the touch input, and the second operation is different from the first operation. This can reduce wear and extend battery life because the electronic device does not need to detect the first touch input and the second touch input to perform the first operation and the second operation. For example, the second operation can be performed based on the directionality, velocity, acceleration, displacement, etc. of the touch input.

[0425] In some embodiments, the electronic device determines that the stylus is not being held based on the absence of sensor data, such as if the stylus has not been paired with the electronic device. In some embodiments, to extend battery life and reduce processing, the stylus provides sensor data when it experiences changes / events rather than continuously providing sensor data. In some embodiments, the electronic device determines that the stylus is not being held based on data received from the stylus. For example, the electronic device receives data from the stylus indicating that the stylus is not being held based on the stylus detecting that contact has been lifted from the stylus.

[0426] For example, reference Figures 9D to 9E , the second operation corresponds to the electronic device 100 performing a display operation. That is, the electronic device 100 displays the logo 914 in response to detecting the left swipe gesture 908. For another example, referring to Figures 9N to 9P , the second operation corresponds to the electronic device 100 performing a highlight operation with respect to the highlighted text 950 in response to detecting the right swipe gesture 938. Continuing with this example, the second operation corresponds to the electronic device 100 performing a display operation on the logo 954 in response to detecting the drag gesture 942.

[0427] In some embodiments, determining that the stylus is not being held by the user includes (1614) detecting an absence of sensor data from the stylus. Performing the second operation based on the absence of sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device.

[0428] In some embodiments, performing the second operation includes (1616) displaying one or more user interface elements on the display. Displaying user interface elements based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation includes displaying a new identifier / object / text, highlighting an existing identifier / object / text, identifying an existing identifier / object / text, etc. For example, referring to Figures 9D to 9E , the electronic device 100 performs a second operation, the second operation including responding to the detection Figure 9DThe corresponding left swipe gesture 908 in Figure 9E In the display mark 914. For example, refer to Figures 9I to 9J , the electronic device 100 performs a second operation, the second operation including responding to the detection Figure 9I The ring gesture 916 detected in Figure 9J 934 is displayed in FIG. Figures 9N to 9O , the electronic device 100 performs a second operation, the second operation including responding to the detection corresponding to Figure 9N The right swipe gesture 938 of the highlighted text 950 in Figure 9O Medium highlight highlights text 950 .

[0429] In some embodiments, performing the second operation includes (1618) navigating within a user interface displayed on the display. For example, the second operation includes panning and / or zooming the canvas. Navigating within the user interface based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation corresponds to navigating through user interface elements such as a marking tool (e.g., a pen, a marker, a pencil, a ruler, etc.). For example, referring to Figure 9O , the electronic device 100 performs a second operation including enlarging the highlighted text 950 or re-centering the text 936 (not shown).

[0430] refer to Figure 16B In some embodiments, the electronic device (1620): detects input from a stylus on a touch-sensitive surface of the electronic device and performs a second operation in response to detecting the input from the stylus. Performing the second operation based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the second operation corresponds to drawing a line. For example, with reference to Figures 9I to 9J , the electronic device 100 responds to the detection Figure 9I The loop input 916 is executed in Figure 9J Continuing with the example, the second operation of the indicator 904 is displayed. Figure 9I Unlike what is shown, the stylus 203 (rather than the hand of the user 902 ) makes the looped input 916 on the touch-sensitive surface of the electronic device 100 .

[0431] In some embodiments, the electronic device (1622) detects input from a stylus on a touch-sensitive surface of the electronic device and performs a third operation in response to detecting the input from the stylus, wherein the third operation is different from the first operation and the second operation. Performing the third operation based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the third operation corresponds to the electronic device displaying a paint brush mark, strokes, spray painting, ink blots, grid lines, etc. on a user interface (not shown).

[0432] It should be understood that 16A to 16B The specific order in which the operations are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations can be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0433] References 16A to 16B The operation is optionally performed by Figure 1A to Figure 1B 16. For example, detection operations 1602, 1620, and 1622, determination operations 1606 and 1612, and execution operations 1616 and 1618 are optionally implemented by event classifier 170, event recognizer 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact (or proximity) on touch-sensitive display 112, and event distributor module 174 transmits the event information to application 136-1. The corresponding event recognizer 180 of application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact (or proximity) at the first location on the touch-sensitive surface (or whether the rotation of the electronic device) corresponds to a predefined event or sub-event, such as the selection of an object on the user interface or the rotation of the electronic device from one orientation to another. When a corresponding predefined event or sub-event is detected, event recognizer 180 activates the event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update the application internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand that based on Figure 1A-1B How the components depicted in FIG. 1 can implement other processes.

[0434] Note that the details of the process described above with respect to method 1600 also apply in a similar manner to the other methods described herein (e.g., 1400, 1500, 1700, 1800, 1900, 2400, 2500, 2600, 2700). For example, the stylus, sensor data, display, touch-sensitive surface, input, and communication interface described above with reference to method 1600 optionally have one or more of the properties of the stylus, sensor data, display, touch-sensitive surface, input, and communication interface described herein with reference to the other methods described herein (e.g., 1400, 1500, 1700, 1800, 1900, 2400, 2500, 2600, 2700).

[0435] 17A to 17C 17 is a flow chart illustrating a method 1700 for performing an operation on an existing identifier displayed on an interface based on a finger manipulation input according to some embodiments. Figure 3 The electronic device 300 or Figure 1A Method 1700 is performed at a portable multifunction device 100 in a computer system. In some embodiments, the touch-sensitive surface and display are combined into a touch screen display (e.g., a mobile phone or tablet computer). In some embodiments, the touch-sensitive surface and display are separate (e.g., a laptop or desktop computer with a separate touchpad and display). Some operations in method 1700 are optionally combined, and / or the order of some operations is optionally changed.

[0436] Performing operations on existing identifiers displayed on the interface based on finger manipulation input data from the stylus reduces the number of inputs required to perform changes in the stylus's functionality. This reduction in inputs enhances the operability of the electronic device and makes the electronic device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the electronic device), which in turn reduces power usage and wear on the electronic device.

[0437] Method 1700 contemplates an electronic device utilizing data received from a stylus to perform operations on existing icons displayed on a tablet computer based on finger manipulation input received by the stylus. These operations include cut / paste operations, copy / paste operations, increase / decrease size operations, increase / decrease thickness operations, and the like. Many different types of finger manipulation inputs can be utilized for many different types of operations. For example, finger manipulation inputs received by the stylus include taps, flicks, swipes, rolls, rotations, and the like.

[0438] This enhances the operability of the electronic device and makes the electronic device interface more efficient and robust. As described above, the user can interact with the stylus in many different ways, while on the other hand, the touch-sensitive surface of the electronic device can receive a single input type (touch input). In addition, fewer transitions of touch input at the touch-sensitive surface of the electronic device reduce wear and tear on the electronic device and power usage. This extends the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently. For battery-powered electronic devices, enabling the user to input fewer inputs to the touch-sensitive surface of the electronic device saves power and increases the time interval between two charges of the battery of the electronic device.

[0439] Relative to Figure 17A When displaying a plurality of user interface elements on the display, the electronic device obtains (1702) finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data includes information about one or more finger manipulation inputs received by the stylus. For example, the plurality of user interface elements correspond to logos, objects, vector graphics and / or objects, etc. For example, referring to Figures 10G to 10I , the one or more finger manipulation inputs received by the stylus 203 include an upward swipe gesture or a downward swipe gesture on the pen shaft of the stylus 203 relative to the electronic device 100 (e.g., inputs 1040 and 1050). In another example, referring to Figures 10B to 10F , the one or more finger manipulation inputs received by stylus 203 include rolling the shaft of stylus 203 in a counterclockwise or clockwise manner (e.g., inputs 1020a to 1020d). In some embodiments, the finger manipulation data corresponds to data collected by the stylus's magnetometer, the stylus's accelerometer, and the capacitive touch element or touch-sensitive surface on the stylus' shaft. In some embodiments, the finger manipulation data is transmitted / received via a Bluetooth connection, an IEEE 802.11x connection, etc.

[0440] In some embodiments, the finger manipulation data includes information about the movement of the finger on the stylus or the movement of the stylus relative to the user's finger (e.g., data indicating how the finger and / or stylus moved). In some embodiments, the finger manipulation data includes a processed representation of the movement of the finger on the stylus or the movement of the stylus relative to the user's finger (e.g., data indicating a gesture or manipulation performed at the stylus, such as a swipe or rotation gesture that optionally includes information indicating the direction and / or magnitude of the gesture or movement). In one example, the finger manipulation data indicates a gesture or manipulation performed at the stylus, such as a swipe or rotation gesture that optionally includes information indicating the direction and / or magnitude of the gesture's movement.

[0441] In some embodiments, at least a portion of the finger manipulation data corresponds (1704) to touch sensor data from one or more touch sensors on the stylus. Obtaining finger manipulation data from the stylus to affect operations at the electronic device enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the sensor data corresponds to data collected by the stylus's magnetometer, the stylus's accelerometer, and the capacitive touch element or touch-sensitive surface on the stylus shaft (e.g., sensor data is transmitted and / or received via a Bluetooth connection, an IEEE 802.11x connection, etc.). For example, Figure 10G As shown, stylus 203 detects input 1040 (e.g., an upward swipe on stylus 203). In another example, as shown in FIG. Figure 10H As shown, stylus 203 detects input 1050 (eg, a downward swipe on stylus 203 ).

[0442] In some embodiments, when displaying multiple user interface elements on the display, the electronic device displays (1706) a drawing application interface including a canvas having multiple pre-existing identifiers displayed in response to previous input from a stylus detected on the touch-sensitive surface of the electronic device. Displaying the canvas based on sensor data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. In some embodiments, the multiple pre-existing identifiers are generated by writing or drawing strokes from the stylus. In some embodiments, the multiple pre-existing identifiers are generated by one or more vector graphics operations. Vector graphics include, for example, closed objects such as triangles, squares, or any polygons. For example, with reference to Figure 10A , the user interface 1000 associated with a drawing or notepad application includes pre-existing content: a star 1004a and a lightning bolt 1004b. In another example, referring to Figure 10G , the user interface 1000 associated with the drawing or notepad application includes pre-existing content: triangle 1004d.

[0443] In some embodiments, the plurality of user interface elements corresponds to a subset of a plurality of pre-existing identifications selected by the user (1708). Utilizing data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, in Figure 10A , pre-existing symbols (e.g., star 1004a and lightning 1004b) appear on user interface 1000 and lightning 1004b is selected, as shown in FIG. Figure 10B In another example, the user selects a pre-existing marker (eg, triangle 1004d) to perform a cut operation.

[0444] In some embodiments, the plurality of user interface elements corresponds to (1710) a subset of a plurality of pre-existing identifications selected based on the position of the stylus relative to the electronic device during detection of one or more finger manipulation inputs. Utilizing the finger manipulation data from the stylus enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, referring to Figure 10A , pre-existing identifiers (e.g., star 1004a and lightning bolt 1004b) appear on the user interface. In response to the user selecting lightning bolt 1004b (a subset of the plurality of pre-existing identifiers), electronic device 100 displays lightning bolt 1004b' in a selected state with a dashed outline to indicate that lightning bolt 1004b' is currently selected, while star 1004a remains unchanged.

[0445] In some embodiments, in response to detecting the finger manipulation data and based on determining that the finger manipulation data indicates a first finger manipulation input on the stylus, the electronic device performs (1712) a first operation on at least a subset of the plurality of user interface elements. In one example, in response to detecting the finger manipulation data, the electronic device performs (1712) a first operation on at least a subset of the plurality of user interface elements. Figure 10B By rolling the stylus 203 counterclockwise in FIG, the electronic device 100 enlarges a subset of the user interface elements (e.g., FIG. 10B to FIG. 10C In another example, in response to detecting an upward swipe on stylus 203, a subset of user interface elements (e.g., Figure 10G In another example, the subset of user interface elements includes objects, vector graphics or objects, etc.

[0446] In some embodiments, the first finger manipulation input corresponds to (1714) a first gesture type detected on the stylus. Obtaining finger manipulation data from the stylus to affect the performance of an operation enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, the first gesture type corresponds to a particular direction of the input gesture (e.g., a counterclockwise roll gesture of the stylus 203 versus a clockwise roll gesture, or an upward swipe gesture versus a downward swipe gesture on the stylus 203). For example, the first gesture type corresponds to a counterclockwise roll gesture of the stylus 203 (e.g., FIG. 10B to FIG. 10C1020a and 1020b in ), and the second gesture type corresponds to a clockwise rolling gesture of the stylus 203 (e.g., Figures 10D to 10E As another example, the first gesture type corresponds to an upward swipe on the stylus 203 (e.g., Figure 10G ), and the second gesture type corresponds to a downward swipe on stylus 203 (e.g., Figure 10H For example, the first gesture type corresponds to an input gesture in a specific manner (eg, a roll gesture of the stylus 203 versus a swipe gesture on the stylus 203 versus a tap gesture on the stylus 203).

[0447] In some embodiments, the first finger manipulation input corresponds to (1716) a first direction of movement of one or more fingers relative to the touch-sensitive surface of the stylus. Obtaining finger manipulation data from the stylus to affect the performance of an operation enhances the operability of the electronic device and reduces the number of inputs to the electronic device. Reducing the number of inputs makes the electronic device more efficient, which extends battery life and reduces wear and tear on the electronic device. For example, referring to Figure 10B , when the user holds the stylus 203 in his / her hand 1002, the movement in the first direction includes a counterclockwise roll of the stylus 203 (e.g., input 1020a). For example, the counterclockwise roll of the stylus 203 increases the size or thickness of the plurality of user interface elements. In such an example, referring to FIG. 10B to FIG. 10C , the size of lightning bolt 1004b' increases from first size 1015a to lightning bolt 1004c' at second size 1015b. For another example, a counterclockwise rotation of reference stylus 203 causes multiple user interface elements to rotate counterclockwise. For another example, the counterclockwise rotation navigates downward through objects at different layers (e.g., the object is hidden under another object, and counterclockwise scrolling can be used to navigate downward through objects at these layers).

[0448] refer to Figure 17B In some embodiments, the first operation increases (1718) the size of at least a subset of the plurality of user interface elements. Increasing the size of ...

Claims

1. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided to communicate with a stylus: detecting input from the stylus on a touch-sensitive surface of the electronic device; and In response to detecting the input: making a first change to content displayed on the display based on determining that the stylus is being held in a first grip arrangement, wherein the first grip arrangement of the stylus is determined based at least in part on sensor data detected at the stylus; and Based on determining that the stylus is being held in a second grip arrangement that is different from the first grip arrangement, making a second change to the content displayed on the display, wherein the second grip arrangement of the stylus is determined at least in part based on sensor data detected by the stylus, wherein the second change to the content displayed on the display is different from the first change to the content displayed on the display.

2. The method according to claim 1, further comprising: obtaining sensor data from the stylus; and A grip arrangement characterizing user manipulation of the stylus is determined based on the sensor data from the stylus, wherein the grip arrangement is determined during the detection of the input. 3 . The method of claim 1 , wherein the grip arrangement is determined based on at least one of a grip style, a grip position, or an orientation of the stylus relative to a reference frame. 4 . The method of claim 1 , wherein at least a portion of the sensor data corresponds to touch sensor data obtained from one or more touch sensors on the stylus.

5. A method according to claim 1, wherein making the first change includes displaying a first user element based on a first marking tool corresponding to the first grip arrangement, and wherein making the second change includes displaying a second user element based on a second marking tool corresponding to the second grip arrangement.

6. A method according to claim 1, wherein making the first change includes changing the existing mark displayed on the display based on a first marking tool corresponding to the first gripping arrangement, and wherein making the second change includes changing the existing mark displayed on the display based on a second marking tool corresponding to the second gripping arrangement.

7. The method of claim 1 , wherein the first grip arrangement is detected based on detecting that the stylus is in an upright orientation of the stylus and detecting a touch input near a first end of the stylus, and wherein making the first change comprises displaying a stroke on the display based on a writing instrument corresponding to the first grip arrangement.

8. The method of claim 1 , wherein the second grip arrangement is detected based on detecting that the stylus is in an inverted orientation of the stylus and detecting a touch input near a second end of the stylus that is different from the first end of the stylus, and wherein making the second change comprises removing an existing mark displayed on the display based on an eraser tool corresponding to the second grip arrangement.

9. The method of claim 1 , wherein the second grip arrangement is detected based on detecting the stylus is in an upright orientation of the stylus and detecting a touch input near a second end of the stylus, and wherein making the second change comprises displaying a stroke based on a drawing tool corresponding to the first grip arrangement.

10. The method of claim 1 , wherein the second grip arrangement is detected based on detecting the stylus being in an inverted orientation of the stylus and detecting a touch input near the first end of the stylus, and wherein making the second change comprises changing an existing indicia displayed on the display based on a smudge tool corresponding to the second grip arrangement.

11. An electronic device, comprising: Touch-sensitive surfaces; monitor; a communication interface provided to communicate with the stylus; non-transitory memory; one or more processors; and One or more programs, wherein the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1-10.

12. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by an electronic device having one or more processors, a display, a touch-sensitive surface, and a communication interface provided for communicating with a stylus, cause the electronic device to perform a method according to any one of claims 1-10.

13. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, and a display: detecting, on the touch-sensitive surface, a first input corresponding to a user-selected color selection affordance; in response to detecting the first input, displaying a color picker user interface on the display, wherein the color picker user interface includes a plurality of options for selecting a user-selected color; detecting, on the touch-sensitive surface, a second input corresponding to a particular option from the plurality of options for selecting a user-selected color; and In response to detecting the second input: designating a first color selected based on the particular option among the plurality of options for selecting a user-selected color as an active color; based on determining that the second input is a continuation of the first input, ceasing to display the color picker user interface upon detecting an end of the second input; and Based on determining that the second input is detected after the first input ends and while the color picker user interface continues to be displayed on the display, maintaining display of the color picker user interface after detecting the end of the second input.

14. A method according to claim 13, wherein the user-selected color selection indicator can represent a plurality of different colors.

15. The method of claim 13, wherein detecting the second input comprises detecting a lift-off of a contact at a location corresponding to the particular option among the plurality of options for selecting a user-selected color.

16. The method according to claim 13, further comprising: after detecting the second input and while the color picker user interface continues to be displayed on the display, detecting a third input corresponding to movement of the touch on the touch-sensitive surface at a location corresponding to a drawing area on the display; and In response to detecting a third input: drawing a marker in the drawing area based on the movement of the touch, wherein the logo has a color based on the active color; and Displaying the color picker user interface on the display is stopped.

17. The method according to claim 13, further comprising: In response to detecting the second input, changing the corresponding portion of the user-selected color selection enable representation to the first color, and displaying the user-selected color selection enable representation with focus, wherein before detecting the second input, the corresponding portion of the user-selected color selection enable representation includes one or more colors other than the first color.

18. A method according to claim 13, wherein before the second input is detected, the user-selected color has not been selected, and the corresponding portion of the user-selected color selection indicator includes multiple different colors.

19. A method according to claim 13, wherein before the second input is detected, a second color has been selected as the user-selected color, and the corresponding portion of the user-selected color selection indication includes the second color.

20. The method of claim 13, further comprising: detecting, on the touch-sensitive surface, a third input corresponding to a predefined color selection affordance; and In response to detecting a third input: designating a color associated with the predefined color selection affordance as the active color; and Display of the first color is maintained within the user-selected color selection affordance.In some embodiments, in response to detecting the third input, the device moves focus from the user-selected color selection affordance to a standard color selection affordance.

21. The method of claim 13, further comprising: detecting a fourth input on the touch-sensitive surface corresponding to the user-selected color selection affordance when the color associated with the predefined color selection affordance is the active color; In response to detecting the fourth input: based on determining that the fourth input corresponds to a first input type, designating the first color associated with the user-selected color selection affordance as the active color without displaying the color picker user interface; and Based on determining that the fourth input corresponds to a second input type different from the first input type, the color picker user interface is displayed on the display.

22. The method according to claim 13, comprising: detecting a third input corresponding to the user-selected color selection affordance; and In response to detecting a third input: designating the corresponding user-selected color as the active color without displaying the color picker user interface on the display based on determining that the corresponding user-selected color is associated with the user-selected color selection affordance; and Based on determining that no user-selected color is associated with the user-selected color selection affordance, displaying the color picker user interface on the display.

23. An electronic device, comprising: Touch-sensitive surfaces; monitor; one or more processors; non-transitory memory; and One or more programs, wherein the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 13-22.

24. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by an electronic device having one or more processors, a touch-sensitive surface, and a display, cause the electronic device to perform a method according to any one of claims 13-22.

25. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided to communicate with a stylus: When the electronic device is in the first state, obtaining information about the current state of the stylus at the electronic device via the communication interface; displaying, on the display, a visual indication that the electronic device is in a second state different from the first state in accordance with a determination that the user is holding the stylus based on the information about the current state of the stylus; and Based on determining that the user is not holding the stylus, the electronic device is maintained in the first state.

26. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided to communicate with a stylus: Detect touch input; and In response to detecting the touch input on the touch-sensitive surface: performing a first operation in response to the touch input based on determining that sensor data obtained from the stylus via the communication interface indicates that the stylus is being held by a user; and Based on determining that the stylus is not held by the user, a second operation is performed in response to the touch input, wherein the second operation is different from the first operation.

27. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided to communicate with a stylus: obtaining finger manipulation data from the stylus via the communication interface while displaying a plurality of user interface elements on the display, wherein the finger manipulation data includes information about one or more finger manipulation inputs received by the stylus; and In response to obtaining the finger manipulation data: performing a first operation on at least a subset of the plurality of the user interface elements based on determining that the finger manipulation data indicates a first finger manipulation input on the stylus; and Based on determining that the finger manipulation data indicates a second finger manipulation input on the stylus that is different from the first finger manipulation input, a second operation is performed on at least a subset of the plurality of the user interface elements, wherein the second operation is different from the first operation.

28. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided to communicate with a stylus: displaying a selection user interface including a plurality of selectable items on the display, wherein a first item of the plurality of selectable items in the selection user interface is currently selected; obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data includes information regarding one or more finger manipulation inputs received at the stylus; and In response to obtaining the finger manipulation data from the stylus: in response to determining that the finger manipulation data from the stylus satisfies a first navigation criterion, wherein the determination that the finger manipulation data from the stylus satisfies the first navigation criterion is based on a first contact pattern detected on the stylus corresponding to a first rotation of the stylus in a first direction, changing the display of the selection user interface to indicate that focus is moving from the first item to a second item of the plurality of selectable items in response to the first direction of the first rotation of the stylus; Based on determining that the finger manipulation data from the stylus satisfies a second navigation criterion, wherein the determination that the finger manipulation data from the stylus satisfies the second navigation criterion is based on a second contact pattern detected on the stylus corresponding to a second rotation of the stylus in a second direction different from the first direction, changing the display of the selection user interface to indicate that focus is moving from the first item to a third item of the plurality of selectable items based on the second direction of the second rotation of the stylus, wherein the third item is different from the second item, and wherein a determination as to whether to instruct the focus to move from the first item to the second item or to instruct the focus to move from the first item to the third item based on a difference between the first contact pattern detected on the stylus and the second contact pattern detected on the stylus; and Based on determining that the finger manipulation data from the stylus includes touch sensor data indicating a sliding gesture based on a third contact pattern detected on the stylus, performing a corresponding operation different from focus movement within the selection user interface.

29. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive display, and a communication interface provided to communicate with a stylus: obtaining, from the stylus via the communication interface, input data corresponding to input detected at the stylus; and In response to obtaining the input data from the stylus: displaying a first user interface element corresponding to the input based on determining that a distance between the stylus and the touch-sensitive display satisfies a first distance threshold when the input is detected at the stylus; and Based on determining that the distance between the stylus and the touch-sensitive display satisfies a second distance threshold when the input is detected at the stylus, displaying the first user interface element corresponding to the input is foregone.

30. A method for interacting with a stylus, comprising: At an electronic device having one or more processors, non-transitory memory, a display, a touch-sensitive surface, and a communication interface provided to communicate with a stylus: In response to detecting that the stylus is in proximity to the electronic device, pairing the electronic device with the stylus; and In response to pairing the stylus with the electronic device: displaying, on the display, a first representation of a first gesture performed on the stylus, wherein the first representation of the first gesture comprises a representation of a finger input displayed on the representation of the stylus displayed on the display; obtaining finger manipulation data from the stylus via the communication interface, wherein the finger manipulation data indicates finger manipulation input received by the stylus; and In response to obtaining the finger manipulation data, displaying on the display a second representation of a second gesture performed on the stylus corresponding to the finger manipulation input received by the stylus, wherein the second representation of the second gesture includes a representation of the finger input displayed on the representation of the stylus displayed on the display.

31. A method comprising: At an electronic device having one or more processors, non-transitory memory, a touch-sensitive surface, a display, and a communication interface provided to communicate with a stylus: detecting movement of the stylus over the touch-sensitive surface; In response to detecting the movement of the stylus, performing a stylus operation in a user interface displayed on the display according to the movement of the stylus; After performing the stylus operation in the user interface, obtaining finger manipulation data indicative of finger manipulation input received at the stylus via the communication interface; In response to obtaining the finger manipulation data from the stylus: changing a property of a stylus operation in the user interface based on the finger manipulation input; and A visual indication of the change in the property of the stylus operation is displayed on the display of the electronic device.

32. A method comprising: At a first electronic device having one or more processors, non-transitory memory, a display, and a communication interface provided to communicate with a stylus: detecting an input corresponding to the stylus, the stylus communicating with the first electronic device via the communication interface; and In response to detecting the input corresponding to the stylus: performing a first operation at the first electronic device in response to determining that the first setting of the stylus has a first value; and Based on determining that the first setting of the stylus has a second value different from the first value, performing a second operation different from the first operation at the first electronic device, wherein the value of the first setting is determined based on input at a second electronic device with which the stylus previously communicated.

Images