Device and method for processing touch input

Abstract

Embodiments of the present disclosure relate to devices and methods for processing touch inputs. The present invention discloses an electronic device having a touch-sensitive surface and a display, the electronic device displaying a user interface at a first display rate. When displaying the user interface, the device detects movement of touch inputs at a series of positions on the touch-sensitive surface at a first detection rate greater than the first display rate; and at each update time in a series of update times, the user interface is updated from a corresponding current state to a corresponding next state according to a selected subset of the series of positions of the touch input. Each selected subset of the series of positions includes multiple positions of the touch input.

Description

[0001] This application is a divisional application of the invention patent application with the application date of May 20, 2016, application number 201680028280.4, and invention name “Device and method for processing touch input”. Technical Field

[0002] This document generally relates to electronic devices with touch-sensitive surfaces, including but not limited to electronic devices with touch-sensitive surfaces that receive touch input. Background Art

[0003] The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Exemplary touch-sensitive surfaces include touch pads and touch screen displays. Such surfaces are widely used to manipulate user interface objects on displays.

[0004] Exemplary manipulations include adjusting the position and / or size of one or more user interface objects or activating a button or opening a file / application represented by a user interface object, as well as associating metadata with one or more user interface objects or otherwise manipulating the user interface. Exemplary user interface objects include digital images, videos, text, icons, control elements (such as buttons), and other graphics. In some cases, a user will need to perform such manipulations of user interface objects in: a file management program (e.g., Finder from Apple Inc. (Cupertino, California)); an image management application (e.g., Aperture, iPhoto, Photos from Apple Inc. (Cupertino, California)); a digital content (e.g., video and music) management application (e.g., iTunes from Apple Inc. (Cupertino, California)); a drawing application; a presentation application (e.g., Keynote from Apple Inc. (Cupertino, California)); a word processing application (e.g., Pages from Apple Inc. (Cupertino, California)); a website creation application (e.g., iWeb from Apple Inc. (Cupertino, California)); a disk editing application (e.g., iDVD from Apple Inc. (Cupertino, California)); or a spreadsheet application (e.g., Numbers from Apple Inc. (Cupertino, California)).

[0005] However, the rapid movement of the touch input results in inconsistencies between the actual location of the touch input and how the touch input is reflected in the user interface. For example, the user interface may be updated with a certain delay, making it impossible for the user interface to reflect the location of the touch input in a timely manner when the touch input moves rapidly. This creates a cognitive burden on the user. In addition, this may result in errors in manipulating user interface objects and may require repeated corrections, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the invention

[0006] Therefore, the present disclosure provides faster, more efficient and more accurate methods and interfaces for electronic devices to manipulate user interface objects. Such methods and interfaces optionally supplement or replace conventional methods for manipulating user interface objects. Such methods and interfaces reduce the amount, degree, and / or nature of input from the user and produce a more effective human-computer interface. In addition, such methods reduce the processing power consumed by processing touch input, save power, improve the accuracy of user input, reduce unnecessary / irrelevant / repetitive input, and may reduce memory usage. For battery-powered devices, such methods and interfaces save battery power and increase the time between battery charges.

[0007] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface are reduced or eliminated by means of the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the device is a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the device has a touch pad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, programs, or instruction sets for performing multiple functions stored in the memory. In some embodiments, the user interacts with the GUI mainly through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, display, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Alternatively or in addition thereto, executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0008] According to some embodiments, a method is performed at an electronic device having a display and a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the electronic device includes one or more sensor units for detecting signals from a stylus associated with the electronic device. The method includes: displaying a user interface at a first display rate; while displaying the user interface, detecting the movement of touch inputs at a series of positions on the touch-sensitive surface at a first detection rate greater than the first display rate; and at each of a series of update times, updating the user interface from a corresponding current state to a corresponding next state according to a selected subset of the series of positions of the touch input, each selected subset of the series of positions including multiple positions of the touch input.

[0009] According to some embodiments, a method is performed at an electronic device having a display and a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the electronic device includes one or more sensor units for detecting signals from a stylus associated with the electronic device. The method includes: displaying a user interface of a first software application updated at a first display rate; and when displaying a first frame of the user interface according to the first display rate: detecting a corresponding movement of a touch input on the touch-sensitive surface; and at a touch processing module independent of the application: based on the touch processing criteria of the first software application, selecting a corresponding touch position of the touch input detected during the corresponding movement to identify as a representative touch position of the corresponding movement; and sending touch position information of the touch input that identifies the corresponding touch position as a representative touch position of the corresponding movement to an application-specific portion of the first software application different from the touch processing module. The method also includes updating the user interface at the first software application based on the touch position information.

[0010] According to some embodiments, a method is performed at an electronic device having a display and a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the electronic device includes one or more sensor units for detecting signals from a stylus associated with the electronic device. The method includes: displaying a user interface at a first display rate; and while displaying the user interface according to the first display rate, detecting movement of a touch input, including detecting the touch input at a first set of sequential positions on the touch-sensitive surface. The first set of sequential positions includes a plurality of positions on the touch-sensitive surface. The method also includes predicting a first set of one or more predicted positions on the touch-sensitive surface for the touch input based on a plurality of positions in the first set of sequential positions. The method also includes updating the user interface based on the first set of one or more predicted positions of the touch input on the touch-sensitive surface.

[0011] According to some embodiments, a method is performed at an electronic device having a display and a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the electronic device includes one or more sensor units for detecting signals from a stylus associated with the electronic device. The method includes: displaying a user interface of a first software application updated at a first display rate; and detecting corresponding movements of a touch input on the touch-sensitive surface at a first detection rate higher than the first display rate. The method also includes sending touch position information of the touch input to an application-specific portion of the first software application at a touch processing module independent of the application, the touch position information identifying: one or more predicted positions of the touch input on the touch-sensitive surface; and one or more predicted intensity values ​​of the touch input at one or more intensity positions of the touch input on the touch-sensitive surface, the one or more intensity positions including at least a subset of the one or more predicted positions. The method also includes processing the touch position information at the first software application.

[0012] According to some embodiments, an electronic device includes a display unit configured to display a user interface; a touch-sensitive surface unit for receiving contact; and a processing unit coupled to the display unit and the touch-sensitive surface unit. The processing unit is configured to: enable display of the user interface at a first display rate; when the user interface is displayed, detect the movement of touch inputs at a series of positions on the touch-sensitive surface unit at a first detection rate higher than the first display rate; and at each of a series of update times, update the user interface from a corresponding current state to a corresponding next state according to a selected subset of the series of positions of the touch input, each selected subset of the series of positions including multiple positions of the touch input. In some embodiments, the electronic device includes one or more sensor units for detecting the intensity of contact with the touch-sensitive surface, and the processing unit is coupled to the one or more sensor units. In some embodiments, the electronic device includes one or more sensor units for detecting signals from a stylus associated with the electronic device, and the processing unit is coupled to the one or more sensor units.

[0013] According to some embodiments, an electronic device includes a display unit configured to display a user interface; a touch-sensitive surface unit for receiving contact; and a processing unit coupled to the display unit and the touch-sensitive surface unit. The processing unit is configured to: enable display of a user interface of a first software application updated at a first display rate; when a first frame of the user interface according to the first display rate is displayed: detect a corresponding movement of a touch input on the touch-sensitive surface unit; and at a touch processing module independent of the application: based on the touch processing criteria of the first software application, select a corresponding touch position of the touch input detected during the corresponding movement to identify it as a representative touch position of the corresponding movement; and send touch position information of the touch input that identifies the corresponding touch position as a representative touch position of the corresponding movement to an application-specific portion of the first software application different from the touch processing module; and at the first software application, update the user interface according to the touch position information. In some embodiments, the electronic device includes one or more sensor units for detecting the intensity of contact with the touch-sensitive surface, and the processing unit is coupled to the one or more sensor units. In some embodiments, the electronic device includes one or more sensor units for detecting a signal from a stylus associated with the electronic device, and the processing unit is coupled to the one or more sensor units.

[0014] According to some embodiments, an electronic device includes a display unit configured to display a user interface; a touch-sensitive surface unit for receiving contact; and a processing unit coupled to the display unit and the touch-sensitive surface unit. The processing unit is configured to: enable display of the user interface at a first display rate; while displaying the user interface according to the first display rate: detect movement of touch input, including detecting touch input at a first set of sequential positions on the touch-sensitive surface unit, wherein the first set of sequential positions includes multiple positions on the touch-sensitive surface unit; and predicting a first set of one or more predicted positions on the touch-sensitive surface unit based on multiple positions in the first set of sequential positions; and updating the user interface according to the first set of one or more predicted positions of the touch input on the touch-sensitive surface unit. In some embodiments, the electronic device includes one or more sensor units for detecting the intensity of contact with the touch-sensitive surface, and the processing unit is coupled to the one or more sensor units. In some embodiments, the electronic device includes one or more sensor units for detecting signals from a stylus associated with the electronic device, and the processing unit is coupled to the one or more sensor units.

[0015] According to some embodiments, an electronic device includes a display unit configured to display a user interface; a touch-sensitive surface unit for receiving contact; and a processing unit coupled to the display unit and the touch-sensitive surface unit. The processing unit is configured to: enable display of a user interface of a first software application updated at a first display rate; detect corresponding movement of a touch input on the touch-sensitive surface unit at a first detection rate higher than the first display rate; send touch position information of the touch input to an application-specific portion of the first software application at an application-independent touch processing module, the touch position information identifying: one or more predicted positions of the touch input on the touch-sensitive surface unit; and one or more predicted intensity values ​​of the touch input at one or more intensity positions of the touch input on the touch-sensitive surface unit, the one or more intensity positions at least including a subset of the one or more predicted positions; and process the touch position information at the first software application. In some embodiments, the electronic device includes one or more sensor units for detecting the intensity of contact with the touch-sensitive surface, and the processing unit is coupled to the one or more sensor units. In some embodiments, the electronic device includes one or more sensor units for detecting signals from a stylus associated with the electronic device, and the processing unit is coupled to the one or more sensor units.

[0016] According to some embodiments, the electronic device includes a display, a touch-sensitive surface, one or more processors, a memory, one or more programs, one or more sensors for detecting the intensity of contact with the touch-sensitive surface, and one or more sensors for detecting signals from a stylus associated with the electronic device; the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of operations of any of the methods described herein. According to some embodiments, a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium, or alternatively, a transient computer-readable storage medium) has instructions stored therein, which, when executed by an electronic device having a display, a touch-sensitive surface, one or more sensors for detecting the intensity of contact with the touch-sensitive surface, and one or more sensors for detecting signals from a stylus associated with the electronic device, causes the electronic device to perform or cause the execution of operations of any of the methods described herein. According to some embodiments, a graphical user interface on an electronic device having a display, a touch-sensitive surface, a memory, one or more processors for executing one or more programs stored in the memory, one or more sensors for detecting the intensity of contact with the touch-sensitive surface, and one or more sensors for detecting signals from a stylus associated with the electronic device, includes one or more elements displayed in any of the methods described herein, and the elements are updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display, a touch-sensitive surface, and a device for performing or causing the operation of any of the methods described herein. The electronic device optionally includes one or more sensors for detecting signals from a stylus associated with the electronic device and / or one or more sensors for detecting the intensity of contact with the touch-sensitive surface. According to some embodiments, an information processing device for use in an electronic device having a display and a touch-sensitive surface includes a device for performing or causing the operation of any of the methods described herein. The electronic device optionally includes one or more sensors for detecting signals from a stylus associated with the electronic device and / or one or more sensors for detecting the intensity of contact with the touch-sensitive surface.

[0017] Thus, a faster, more efficient method and interface for manipulating user interface objects is provided for an electronic device having a display, a touch-sensitive surface, and optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, and optionally one or more sensors for detecting signals from a stylus associated with the electronic device, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for manipulating user interface objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

[0019] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.

[0020] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0021] Figure 1C is a block diagram illustrating the transmission of event objects according to some embodiments.

[0022] Figure 1D is a block diagram illustrating the structure of an event object according to some embodiments.

[0023] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0024] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0025] Figure 4 is a block diagram of an exemplary electronic stylus according to some embodiments.

[0026] FIG. 5A to FIG. 5B The posture state of a stylus relative to a touch-sensitive surface according to some embodiments is shown.

[0027] Fig. 6A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0028] Figure 6B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0029] 7A to 7OO An exemplary user interface is shown for updating a user interface based on consolidated and / or predicted touch locations and related actions according to some embodiments.

[0030] FIG. 8A to FIG. 8B is a flowchart illustrating a method for updating a user interface based on merged touch locations according to some embodiments.

[0031] 9A to 9Dis a flow chart illustrating a method of processing touch input with a touch processing module according to some embodiments.

[0032] FIG. 10A to FIG. 10C is a flowchart illustrating a method for updating a user interface based on predicted touch locations according to some embodiments.

[0033] Fig.11 is a flow chart illustrating a method of transmitting predicted touch information according to some embodiments.

[0034] Figures 12 to 15 is a functional block diagram of an electronic device according to some embodiments.

[0035] Unless otherwise noted, the drawings are not drawn to scale. DETAILED DESCRIPTION

[0036] Many electronic devices have a graphical user interface to receive user input for manipulating user interface objects (e.g., moving user interface objects or creating user interface objects, such as drawing lines). Due to the delay associated with hardware and / or software components in processing touch input, the rapid movement of touch input causes inconsistencies between the actual position of touch input and how the touch input is reflected in the user interface. For example, the user interface may be updated with a certain delay so that the user interface cannot reflect the position of the touch input in a timely manner when the touch input moves quickly. This creates a cognitive burden on the user and may cause errors in manipulating user interface objects. In the embodiments described below, an improved method for manipulating user interface objects is implemented by using a combined touch position, a predicted touch position, or a combination of a combined touch position and a predicted touch position. By detecting touch input at a higher rate (e.g., higher than the display rate), the position of the touch input can be detected between frames. This position information is used to reduce inconsistencies. In addition or alternatively, the position of the touch input is predicted, which is also used to reduce inconsistencies. This method streamlines the object manipulation process by using a combined and / or predicted touch position, thereby reducing the inconsistency between the touch input and the displayed user interface and allowing more accurate manipulation of user interface objects.

[0037] under, Figure 1A to Figure 1B , Figure 2 and Figure 3 A description of an exemplary apparatus is provided. Figure 4 A description of an exemplary electronic stylus is provided. FIG. 5A to FIG. 5B The posture state of the stylus relative to the touch-sensitive surface is shown. FIG. 6A to FIG. 6B as well as 7A to 7OO Exemplary user interfaces for updating a user interface based on consolidated and / or predicted touch locations and related operations are shown. FIG. 8A to FIG. 8BA flow chart illustrating a method of updating a user interface based on merged touch locations. 9A to 9D A flow chart showing a method for processing touch input with a touch processing module. FIG. 10A to FIG. 10C A flow chart illustrating a method of updating a user interface based on predicted touch locations. Fig.11 A flow chart illustrating a method of communicating predicted touch information. 7A to 7OO The user interface in FIG. 8A to FIG. 8B , 9A to 9D , FIG. 10A to FIG. 10C and Fig.11 process.

[0038] Exemplary Devices

[0039] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are provided in the following detailed description to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be put into practice without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail to unnecessarily obscure the various aspects of the embodiments.

[0040] It will also be understood that, although in some cases, the expressions "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these expressions. These expressions are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0041] The terms used in the description of various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various embodiments described and in the appended claims, the singular forms "a" "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the expression "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the items listed in association. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0042] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0043] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0044] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0045] The device typically supports a variety of applications, such as one or more of the following applications: a notepad application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a game application, a phone application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0046] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports various applications with a user interface that is intuitive and clear to the user.

[0047] Attention is now directed to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0048] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, movement of a touch-sensitive surface may optionally be interpreted or sensed by a user as "roughness" of the touch-sensitive surface even when there is no change in the smoothness of the touch-sensitive surface. While such interpretation of touch by a user will be limited by the user's individualized sensory perceptions, there are many sensory perceptions of touch that are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "release click," "press click," "roughness"), unless otherwise stated, the tactile output generated corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0049] It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0050] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The access of other components of the device 100 (such as the CPU 120 and the peripheral device interface 118) to the memory 102 is optionally controlled by the memory controller 122.

[0051] Peripheral interface 118 may be used to couple the device's input and output peripherals to memory 102 and one or more CPUs 120. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0052] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0053] RF (radio frequency) circuit 108 receives and transmits RF signals, also called electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with a network and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence using Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0054] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headphone jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0055] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from other input or control devices 116. Other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, and the like. In some alternative embodiments, the input controller 160 is optionally coupled to any of the following (or none of the following): a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0056] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object.

[0057] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on haptic / tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some exemplary embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0058] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies. In some exemplary embodiments, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0059] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device translates the rough finger-based input into a precise pointer / cursor position or command for performing the action desired by the user.

[0060] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0061] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0062] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures static images and / or videos. In some embodiments, the optical sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0063] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to a force sensor controller 159 in the I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite to touch screen display system 112 located on the front of device 100.

[0064] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0065] Device 100 optionally also includes one or more tactile output generators 163. Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. The tactile output generator 163 optionally includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, a solenoid, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). In some embodiments, the tactile output generator 163 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch-sensitive display system 112 located on the front of the device 100.

[0066] The device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripheral device interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of device 100.

[0067] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a gesture module (or instruction set) 131, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state indicating which applications, if any, are currently active; a display state indicating what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state including information obtained from the device's various sensors and other input or control devices 116; and position and / or posture information regarding the device's position and / or posture.

[0068] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0069] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device compatible with some of Apple Inc. (Cupertino, California). Devices, iPod In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in devices and iPod devices. In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with some of Apple Inc. (Cupertino, California) Devices, iPod devices, and a Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in iPod devices.

[0070] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0071] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0072] Position module 131 optionally detects posture information about the device, such as the attitude (roll, pitch, and / or yaw) of the device in a particular reference frame, in conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169. Posture module 130 includes software components for performing various operations related to detecting device posture and detecting changes in device posture. In some embodiments, posture module 131 uses information received from a stylus being used with the device to detect posture information about the stylus, such as detecting a posture state of the stylus relative to the device and detecting a change in the posture state of the stylus.

[0073] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0074] In some embodiments, the graphics module 132 stores data to be used to represent graphics. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0075] Haptic feedback module 133 includes various software components for generating instructions (eg, used by haptic feedback controller 161 ) for generating tactile output at one or more locations on device 100 using tactile output generator 163 in response to user interaction with device 100 .

[0076] The text input module 134, which is optionally a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, -email 140, IM 141, browser 147, and any other application requiring text input).

[0077] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing, to the camera 143 as picture / video metadata, and to applications that provide location-based services such as a weather widget, a local yellow pages widget, and a map / navigation widget).

[0078] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0079] Contacts module 137 (sometimes called address book or contact list);

[0080] Telephone module 138;

[0081] Video conferencing module 139;

[0082] Email client module 140;

[0083] Instant messaging (IM) module 141;

[0084] Fitness support module 142;

[0085] A camera module 143 for still images and / or video images;

[0086] Image management module 144;

[0087] Browser module 147;

[0088] Calendar module 148;

[0089] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0090] A widget creator module 150 for forming a user-created widget 149-6;

[0091] Search module 151;

[0092] Video and music player module 152, which optionally consists of a video player module and a music player module;

[0093] Notepad module 153;

[0094] Map module 154; and / or

[0095] Online video module 155.

[0096] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0097] In combination with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the contact module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in the memory 102 or the memory 370), including: adding a name to the address book; deleting a name from the address book; associating a phone number, email address, physical address, or other information with a name; associating an image with a name; categorizing and classifying names; providing a phone number and / or email address to initiate and / or facilitate communication via telephone 138, video conferencing 139, email 140, or IM 141, and the like.

[0098] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a sequence of characters corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0099] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0100] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0101] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0102] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for: creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting fitness music and playing it; and displaying, storing, and transmitting fitness data.

[0103] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0104] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0105] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0106] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0107] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0108] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0109] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0110] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as, MP3 or AAC files), and to display, present, or otherwise play back video (e.g., on touch-sensitive display system 112 or on an external display connected wirelessly or via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0111] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, etc. according to user instructions.

[0112] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data relating to stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0113] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0114] Each module and application in the above-mentioned modules and applications corresponds to a set of executable instructions for performing one or more functions and the methods described in this patent application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 102 optionally stores additional modules and data structures not described above.

[0115] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, etc.) on device 100 is optionally reduced.

[0116] The predefined set of functions performed exclusively through the touch screen and / or touchpad optionally includes navigating between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, a home menu, or a root menu. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0117] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0118] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192, which indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0119] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by the application 136-1 or information ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0120] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0121] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving input for more than a predetermined duration).

[0122] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0123] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0124] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events that are considered to be correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0125] Hit view determination module 172 receives information related to sub-events of contact-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0126] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0127] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0128] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In other embodiments, event classifier 170 is a standalone module or is part of another module (such as contact / motion module 130) stored in memory 102.

[0129] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, such as a user interface toolkit (not shown) or a higher level object from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178 and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0130] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0131] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0132] Event comparator 184 compares event information with predefined event or sub-event definitions, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition for event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition for event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0133] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0134] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0135] When a corresponding event recognizer 180 determines that a series of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0136] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0137] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag receives the tag and performs a predefined process.

[0138] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event series or to an actively participating view. The event handler associated with the sub-event series or with an actively participating view receives the event information and performs a predetermined process.

[0139] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0140] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0141] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in combination with single or multiple keyboard presses or holds; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to a sub-event for defining an event to be recognized.

[0142] Figure 1C is a diagram showing an event object (e.g. Figure 1D Block diagram of the transmission of event object 194).

[0143] As mentioned above Figure 1AAs described, the contact / motion module 130 determines the state and / or state change of the touch input. In some embodiments, the device generates a signal or data (e.g., in the form of a data object) to transmit the determined state of the touch input and / or the determined state change of the touch input to one or more software components. In some embodiments, the data object is called an event object (e.g., event object 194). The event object includes data representing the state of the corresponding touch input. In some embodiments, the event object 194 is a mouse event object (e.g., an event object including information of one or more mouse events (such as mouse movement, button clicks, etc.)). In some other embodiments, the event object 194 is a touch event object different from the mouse event object. In some embodiments, the touch event object includes data representing the touch-specific attributes of the corresponding touch input (e.g., the number of simultaneous touches, the orientation of the finger contact or stylus, etc.). In some embodiments, the event object 194 includes data representing the force event-specific attributes of the corresponding touch input (e.g., the strength applied by the touch input, the force phase / period of the touch input, etc.). For example, a mouse event object including information of one or more force events (e.g., the strength applied by the touch input, the force phase / period of the touch input, etc.) can be used.

[0144] In some embodiments, contact / motion module 130 generates (or updates) an event object and sends the event object to one or more applications (e.g., application 136-1, such as Figure 3 380 in the drawing module, and / or application 136-2, such as browser module 147). Alternatively, the contact / information module 130 sends information about the contact (e.g., the original coordinates of the contact) to one or more applications (e.g., application 1 (136-1) and / or application 2 (136-2)), and the application receiving the information generates (or updates) one or more event objects. In some embodiments, the application includes a touch processing module 220, which generates (or updates) one or more event objects and sends the one or more event objects to a portion of the application other than the touch processing module 220. In some embodiments, the touch processing module 220 is application-independent (e.g., the same touch processing module is included in each of a plurality of different applications (such as a drawing application, a browser application, etc.)). As used herein, the touch processing module 220 being application-independent means that the touch processing module 220 is not specifically designed for a particular software application. The touch processing module 220 being application-independent does not mean that the touch processing module 220 is located separately from its associated application. Although the touch processing module 220 is distinct and separate from its associated application in some embodiments, such as Figure 1CAs shown, the touch processing module 220 is included in its associated application in some embodiments. In some embodiments, the application also includes an application kernel specific to the application.

[0145] exist Figure 1C In the embodiment, each of application 1 (136-1, such as a drawing application) and application 2 (136-2, such as a browser application) includes a touch processing module 220. In addition, application 1 (136-1) includes application kernel 1 (230-1) specific to application 1 (136-1), and / or application 2 (136-2) includes application kernel 2 (230-2) specific to application 2 (136-2). For example, application kernel 1 (230-1) includes instructions for performing operations specific to application 1 (136-1) (e.g., drawing a stroke), and application kernel 2 (230-2) includes instructions for performing operations specific to application 2 (136-2) (e.g., bookmarking a webpage).

[0146] In some embodiments, event object 194 is sent directly to a target (e.g., a software component such as application kernel 1 (230-1)). Optionally, event object 194 is sent through application programming interface 222. In some embodiments, event object 194 is sent by posting event object 194 (e.g., in queue 218-1) for retrieval by application kernel 1 (230-1).

[0147] In some embodiments, event object 194 includes force information. In some embodiments, a mouse event object includes force information (e.g., the raw or normalized force applied by a touch input). In some embodiments, a touch event object includes force information. In some embodiments, a force event object includes force information.

[0148] Figure 1D is a block diagram illustrating the structure of event object 194 according to some embodiments.

[0149] The event object 194 includes detected touch information 242 corresponding to one or more detected touches. In some embodiments, the detected touch information 242 includes information of individual touches detected (e.g., information 246-1 for touch 1, information 246-2 for touch 2, and so on). The detected touch information 242 optionally includes touch identifiers (e.g., touch identifier 1 (246-1) for touch 1, touch identifier 2 (246-2) for touch 2, and so on). In some embodiments, the detected touch information 242 includes information identifying a representative touch (e.g., a touch identifier for a representative touch).

[0150] The information of an individual touch includes location information 250 of the corresponding touch, and optionally also includes information 252 identifying the strength applied by the corresponding touch, information 254 identifying the tilt and / or orientation of a stylus associated with the device, a timestamp 256 of the corresponding touch (e.g., timestamp 26 indicates the time when the corresponding touch was detected), and / or a type 258 of the corresponding touch (e.g., whether the corresponding touch was made with a finger or a stylus). In some embodiments, even for a detected touch, the values ​​of some of these fields are predicted (e.g., due to delays in obtaining force and / or tilt information about the touch, the force and / or tilt values ​​of the detected touch are predicted).

[0151] In some embodiments, the event object 194 also includes predicted touch information 244 corresponding to one or more predicted touches. The structure of the predicted touch information 244 is similar to the structure of the detected touch information 242 above. For the sake of brevity, such details are not repeated here.

[0152] Figure 2 A device having a touch screen (e.g., Figure 1A The portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In these embodiments and in other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, the selection of one or more graphics occurs when the user interrupts contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0153] The device 100 optionally also includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0154] In some embodiments, the device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the device on / off and for locking the device, a volume adjustment button 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to: power on / off the device by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for a user of the device 100.

[0155] Figure 3 300 is a block diagram of an exemplary multi-function device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuits (sometimes called chipsets) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 163), sensors 359 (e.g., similar to those described above) Figure 1AThe memory 370 may include touch sensors, optical sensors, contact intensity sensors, proximity sensors, acceleration sensors, gesture and / or magnetic sensors of the sensors 112, 164, 165, 166, 167, 168 and 169 described above. The memory 370 includes a high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0156] Figure 3 Each of the above-mentioned identified elements is optionally stored in one or more previously mentioned memory devices. Each module in the above-mentioned modules corresponds to a group of instructions for performing the above-mentioned functions. The above-mentioned modules or programs (that is, instruction sets) do not have to be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0157] Figure 4is a block diagram of an exemplary electronic stylus 203 according to some embodiments. Electronic stylus 203 is sometimes referred to simply as a stylus. Stylus 203 includes memory 402 (which optionally includes one or more computer-readable storage media), memory controller 422, one or more processing units (CPUs) 420, peripheral device interface 418, RF circuit 408, input / output (I / O) subsystem 406, and other input or control devices 416. Stylus 203 optionally includes external port 424 and one or more optical sensors 464. Stylus 203 optionally includes one or more intensity sensors 465 for detecting the intensity of contact of stylus 203 on device 100 (e.g., when stylus 203 is used with a touch-sensitive surface such as touch-sensitive display system 112 of device 100) or on other surfaces (e.g., table surfaces). Stylus 203 optionally includes one or more tactile output generators 463 for generating tactile output on stylus 203. These components optionally communicate via one or more communication buses or signal lines 403 .

[0158] In some embodiments, the term "tactile output" as described above refers to a physical displacement of an accessory (e.g., stylus 203) of a device (e.g., device 100) relative to a previous position of the accessory, a physical displacement of one component of the accessory relative to another component of the accessory, or a displacement of a component relative to the center of mass of the accessory, which will be detected by the user with the user's sense of touch. For example, in the case where an accessory or a component of an accessory is in contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the accessory or a component of the accessory. For example, movement of a component (e.g., the housing of stylus 203) is optionally interpreted by the user as a "click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "click", even when a physical actuation button associated with the stylus that is physically pressed (e.g., displaced) by the user's movement does not move. Although such interpretation of touch by the user will be limited by the user's individualized sensory perception, there are many sensory perceptions of touch that are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., a “click”), unless otherwise stated, the tactile output generated corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception in a typical (or average) user.

[0159] It should be understood that stylus 203 is only one example of an electronic stylus, and that stylus 203 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 4The various components shown in the drawings are implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0160] Memory 402 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more flash memory devices, or other non-volatile solid-state memory devices. Access to memory 402 by other components of stylus 203 (such as CPU 420 and peripheral device interface 418) is optionally controlled by memory controller 422.

[0161] Peripheral interface 418 may be used to couple input and output peripherals of the stylus to CPU 420 and memory 402. The one or more processors 420 run or execute various software programs and / or instruction sets stored in memory 402 to perform various functions of stylus 203 and process data.

[0162] In some embodiments, peripherals interface 418, CPU 420, and memory controller 422 are optionally implemented on a single chip, such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0163] RF (radio frequency) circuit 408 receives and transmits RF signals, also called electromagnetic signals. RF circuit 408 converts electrical signals into / from electromagnetic signals and communicates with device 100 or 300, a communication network, and / or other communication devices via electromagnetic signals. RF circuit 408 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuit 408 optionally communicates with a network and other devices through wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution Pure Data (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0164] The I / O subsystem 406 couples input / output peripherals on the stylus 203, such as other input or control devices 416, to a peripheral device interface 418. The I / O subsystem 406 optionally includes an optical sensor controller 458, an intensity sensor controller 459, a tactile feedback controller 461, and one or more input controllers 460 for other input or control devices. The one or more input controllers 460 receive / send electrical signals from / to other input or control devices 416. The other input or control devices 416 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, click wheels, and the like. In some alternative embodiments, the input controller 460 is optionally coupled or uncoupled to any of the following: an infrared port and / or a USB port.

[0165] Stylus 203 also includes a power system 462 for powering various components. Power system 462 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device and / or portable accessory.

[0166] Stylus 203 optionally also includes one or more optical sensors 464 . Figure 4 An optical sensor is shown coupled to an optical sensor controller 458 in the I / O subsystem 406. The optical sensor 464 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 464 receives light projected through one or more lenses from the environment and converts the light into data representing an image.

[0167] Stylus 203 optionally also includes one or more contact intensity sensors 465 . Figure 4 A contact force sensor is shown coupled to force sensor controller 459 in I / O subsystem 406. Contact force sensor 465 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a surface). Contact force sensor 465 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to the tip of stylus 203.

[0168] Stylus 203 optionally also includes one or more proximity sensors 466 . Figure 4A proximity sensor 466 is shown coupled to the peripherals interface 418. Alternatively, the proximity sensor 466 is coupled to the input controller 460 in the I / O subsystem 406. In some embodiments, the proximity sensor determines that the stylus 203 is in proximity to the electronic device (eg, device 100).

[0169] Stylus 203 optionally also includes one or more tactile output generators 463 . Figure 4 A tactile output generator coupled to a tactile feedback controller 461 in the I / O subsystem 406 is shown. The tactile output generator 463 optionally includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, a solenoid, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on a device). The tactile output generator 463 receives tactile feedback generation instructions from the tactile feedback module 433 and generates a tactile output on the stylus 203 that can be felt by the user of the stylus 203. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to the length (e.g., body or housing) of the stylus 203 and optionally generates a tactile output by moving the stylus 203 vertically (e.g., in a direction parallel to the length of the stylus 203) or laterally (e.g., in a direction orthogonal to the length of the stylus 203).

[0170] Stylus 203 optionally also includes one or more accelerometers 467 , gyroscopes 468 , and / or magnetometers 470 for acquiring information about the position and posture state of stylus 203 (eg, as part of an inertial measurement unit (IMU)). Figure 4 Sensors 467, 469, and 470 are shown coupled to peripherals interface 418. Alternatively, sensors 467, 469, and 470 are optionally coupled to input controller 460 in I / O subsystem 406. Stylus 203 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information about the location of stylus 203.

[0171] In some embodiments, the software components stored in memory 402 include an operating system 426, a communication module (or instruction set) 428, a contact / motion module (or instruction set) 430, a gesture module (or instruction set) 431, and a global positioning system (GPS) module (or instruction set) 435. In addition, in some embodiments, memory 402 stores device / global internal state 457, such as Figure 4Device / global internal state 457 includes one or more of the following: sensor state, which indicates information obtained from various sensors of the stylus and other input or control devices 416; posture state, which includes information about the posture of the stylus relative to the device (e.g., device 100) (e.g., posture, orientation, tilt, roll, and / or distance, such as Figure 5A and Figure 5B and location information about the stylus location (eg, determined by GPS module 435).

[0172] The operating system 426 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.), and facilitates communication between various hardware and software components.

[0173] The communication module 428 optionally facilitates communication with other devices through one or more external ports 424, and also includes various software components for processing data received by the RF circuit 408 and / or the external port 424. The external port 424 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device compatible with some of Apple Inc. (Cupertino, California). Devices, iPod Equipment, and A Lightning connector that is the same as or similar to and / or compatible with the Lightning connector used in the device.

[0174] The contact / motion module 430 optionally detects contact with the stylus 203 and other touch-sensitive devices of the stylus 203 (e.g., buttons or other touch-sensitive components of the stylus 203). The contact / motion module 430 includes software components for performing various operations related to contact detection (e.g., detecting contact of the tip of the stylus with a touch-sensitive display such as the touch screen 112 of the device 100 or with another surface such as a table surface), such as determining whether contact has occurred (e.g., detecting a touch down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement (e.g., on the touch screen 112 of the device 100), and determining whether the contact has ceased (e.g., detecting a lift-off event or contact disconnection). In some embodiments, the contact / motion module 430 receives contact data from the I / O subsystem 406. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. As described above, in some embodiments, one or more of these operations related to contact detection are performed by the device utilizing contact / motion module 130 (in addition to or in lieu of stylus utilizing contact / motion module 430).

[0175] The contact / motion module 430 optionally detects gesture input of the stylus 203. Different gestures of the stylus 203 have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-click gesture includes detecting a touch-down event and then detecting a lift-off event at the same location (or substantially the same location) as the touch-down event (e.g., at the location of an icon). As another example, detecting a swipe gesture includes detecting a touch-down event, then detecting one or more stylus drag events, and then detecting a lift-off event. As described above, in some embodiments, gesture detection is performed by the device utilizing the contact / motion module 130 (in addition to or instead of the stylus utilizing the contact / motion module 430).

[0176] Posture module 431 optionally detects posture information about the stylus, such as the attitude (roll, pitch, and / or yaw) of the device in a particular reference frame, in conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469. Posture module 431 optionally detects stylus movement gestures, such as flicks, taps, and rolls of the stylus, in conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469. Posture module 431 includes software components for performing various operations related to detecting stylus postures and detecting stylus posture changes in a particular reference frame. In some embodiments, posture module 431 detects the posture state of the stylus relative to the device and detects changes in the posture state of the stylus relative to the device. As described above, in some embodiments, (in addition to or in lieu of the stylus utilizing posture module 431) device 100 or 300 utilizes posture module 131 to determine the posture state of the stylus relative to the device and changes in the posture state of the stylus.

[0177] Haptic feedback module 433 includes various software components for generating instructions used by tactile output generator 463 to produce tactile output at one or more locations on stylus 203 in response to user interaction with stylus 203 .

[0178] GPS module 435 determines the location of the stylus and provides this information for use in various applications (eg, to applications that provide location-based services, such as applications for finding lost devices and / or accessories).

[0179] Each module and application program in the above-mentioned modules and applications corresponds to a group of executable instructions for performing the above-mentioned one or more functions and the method described in this patent application (e.g., the computer-implemented method and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 402 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 402 optionally stores additional modules and data structures not described above.

[0180] FIG. 5A to FIG. 5B The posture state of the stylus 203 relative to the touch-sensitive surface (e.g., the touch screen 112 of the device 100) according to some embodiments is shown. In some embodiments, the posture state of the stylus 203 corresponds to (or indicates): the projected position of the tip (or other representative portion) of the stylus on the touch-sensitive surface (e.g., (x, y) position 504, Figure 5A ), the orientation of the stylus relative to the touch-sensitive surface (e.g., orientation 506, Figure 5A ), the tilt of the stylus relative to the touch-sensitive surface (e.g., tilt 512, Figure 5B ), and / or the distance of the stylus relative to the touch-sensitive surface (e.g., distance 514, Figure 5B ). In some embodiments, the posture state of stylus 203 corresponds to (or indicates) the pitch, yaw, and / or roll of the stylus (e.g., the posture of the stylus relative to a particular reference frame, such as a touch-sensitive surface (e.g., touch screen 112) or the ground). In some embodiments, the posture state includes a set of posture parameters (e.g., one or more posture parameters). In some embodiments, the posture state is detected based on one or more measurements from stylus 203 that are sent to an electronic device (e.g., device 100). For example, the stylus measures the tilt of the stylus (e.g., tilt 512, Figure 5B ) and / or orientation (e.g., orientation 506, Figure 5A ), and sends the measurements to device 100. In some embodiments, the posture state is detected based on raw input from one or more electrodes in the stylus sensed by the touch-sensitive surface (e.g., touch screen 112 of device 100), instead of or in combination with posture states detected based on one or more measurements from stylus 203. For example, the touch-sensitive surface receives raw output from one or more electrodes in the stylus and calculates the tilt and / or orientation of the stylus based on the raw output (optionally in combination with posture state information provided by the stylus based on sensor measurements generated by the stylus).

[0181] Figure 5A According to some embodiments, stylus 203 is shown relative to a touch-sensitive surface (e.g., touch screen 112 of device 100) from a perspective directly above the touch-sensitive surface. Figure 5A , the z-axis 594 points out of the page (i.e., in a direction orthogonal to the plane of the touch screen 112), the x-axis 590 is parallel to a first edge of the touch screen 112 (e.g., length), the y-axis 592 is parallel to a second edge of the touch screen 112 (e.g., width), and the y-axis 592 is perpendicular to the x-axis 590.

[0182] Figure 5AThe tip of stylus 203 is shown at (x, y) position 504. In some embodiments, the tip of stylus 203 is the terminal end of the stylus that is configured to determine the proximity of the stylus to a touch-sensitive surface (e.g., touch screen 112). In some embodiments, the projection of the tip of the stylus onto the touch-sensitive surface is an orthogonal projection. In other words, the projection of the tip of the stylus onto the touch-sensitive surface is the point at the end of a line from the tip of the stylus to the touch-sensitive surface that is orthogonal to the surface of the touch-sensitive surface (e.g., if the stylus is moved directly along a path orthogonal to the touch-sensitive surface, the tip of the stylus will hit (x, y) position 504 of the touch-sensitive surface). In some embodiments, the (x, y) position at the lower left corner of touch screen 112 is position (0, 0) (e.g., (0, 0) position 502), and other (x, y) positions on touch screen 112 are relative to the lower left corner of touch screen 112. Alternatively, in some embodiments, the (0,0) position is located at another location of the touch screen 112 (e.g., at the center of the touch screen 112), and the other (x,y) positions are relative to the (0,0) position of the touch screen 112.

[0183] in addition, Figure 5A Stylus 203 is shown with orientation 506. In some embodiments, orientation 506 is the orientation of the projection of stylus 203 onto touch screen 112 (e.g., an orthogonal projection of the length of stylus 203, or a line corresponding to a line between the projections of two different points of stylus 203 onto touch screen 112). In some embodiments, orientation 506 is relative to at least one axis in a plane parallel to touch screen 112. In some embodiments, orientation 506 is relative to a single axis in a plane parallel to touch screen 112 (e.g., axis 508, in clockwise rotation angles from 0 degrees to 360 degrees relative to axis 508, as shown in FIG. 1 ). Figure 5A Alternatively, in some embodiments, orientation 506 is relative to a pair of axes in a plane parallel to touch screen 112 (e.g., x-axis 590 and y-axis 592, as shown in FIG. Figure 5A as shown, or a pair of axes associated with an application displayed on touch screen 112).

[0184] In some embodiments, an indication (e.g., indication 516) is displayed on a touch-sensitive display (e.g., touch screen 112 of device 100). In some embodiments, indication 516 shows where the stylus will touch (or mark) the touch-sensitive display before the stylus touches the touch-sensitive display. In some embodiments, indication 516 is part of the mark being drawn on the touch-sensitive display. In some embodiments, indication 516 is separate from the mark being drawn on the touch-sensitive display and corresponds to a virtual "pen tip" or other element that indicates where the mark will be drawn on the touch-sensitive display.

[0185] In some embodiments, indication 516 is displayed based on the posture state of stylus 203. For example, in some cases, indication 516 is displaced relative to (x, y) position 504 (e.g., Figure 5A and Figure 5B In some embodiments, the indication 516 is displayed with varying colors, sizes (or radii or areas), opacities, and / or other characteristics depending on the posture state of the stylus. In some embodiments, the displayed indication takes into account the thickness of the glass layer on the touch-sensitive display to pass the indication "onto" the pixels of the touch-sensitive display, rather than "displaying" the indication "on" the glass covering the pixels.

[0186] Figure 5B According to some embodiments, stylus 203 is shown relative to a touch-sensitive surface (e.g., touch screen 112 of device 100) from a side view of the touch-sensitive surface. Figure 5B , the z-axis 594 points in a direction orthogonal to the plane of the touch screen 112, the x-axis 590 is parallel to a first edge (e.g., length) of the touch screen 112, the y-axis 592 is parallel to a second edge (e.g., width) of the touch screen 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0187] Figure 5B Stylus 203 is shown with tilt 512. In some embodiments, tilt 512 is an angle relative to a normal to the surface of the touch-sensitive surface (e.g., normal 510) (also referred to simply as the normal to the touch-sensitive surface). Figure 5B As shown, tilt 512 is zero when the stylus is perpendicular / normal to the touch-sensitive surface (eg, when stylus 203 is parallel to normal 510), and the tilt increases as the stylus is tilted closer to being parallel to the touch-sensitive surface.

[0188] in addition, Figure 5B 514 is shown relative to the touch-sensitive surface of the stylus 203. In some embodiments, the distance 514 is the distance from the tip of the stylus 203 to the touch-sensitive surface in a direction orthogonal to the touch-sensitive surface. Figure 5B , distance 514 is the distance from the tip of stylus 203 to (x,y) position 504 .

[0189] Although the terms "x-axis", "y-axis", and "z-axis" are used herein to illustrate specific directions in specific drawings, it should be understood that these terms do not refer to absolute directions. In other words, the "x-axis" can be any corresponding axis, and the "y-axis" can be a specific axis different from the x-axis. Typically, the x-axis is perpendicular to the y-axis. Similarly, the "z-axis" is different from the "x-axis" and the "y-axis", and is typically perpendicular to both the "x-axis" and the "y-axis".

[0190] in addition, Figure 5B Roll 518 of stylus 203 is shown, a rotation about the length (major axis).

[0191] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0192] Fig. 6A An exemplary user interface of an application menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 600 includes the following elements, or a subset or superset thereof:

[0193] Signal strength indicators 602 for wireless communications (such as cellular signals and Wi-Fi signals);

[0194] Time 604;

[0195] Bluetooth indicator 605;

[0196] Battery status indicator 606;

[0197] A tray 608 with icons of commonly used applications, such as:

[0198] o an icon 616 of the phone module 138 labeled “Phone” which optionally includes an indicator 614 of the number of missed calls or voice messages;

[0199] o an icon 618 of the email client module 140 labeled “Mail”, which optionally includes an indicator 610 of the number of unread emails;

[0200] o An icon 620 of the browser module 147 labeled "Browser"; and

[0201] o an icon 622 labeled “iPod” for the video and music player module 152 (also called the iPod (trademark of Apple Inc.) module 152); and

[0202] Icons of other applications, such as:

[0203] o Icon 624 of the IM module 141 labeled “Message”;

[0204] o Icon 626 labeled “Calendar” of calendar module 148;

[0205] o Icon 628 labeled “Photos” of the image management module 144;

[0206] o An icon 630 labeled “Camera” of the camera module 143;

[0207] o Icon 632 labeled “Online Video” of the online video module 155;

[0208] o Icon 634 labeled "Stock Market" of the Stock Market widget 149-2;

[0209] o an icon 636 labeled “MAP” of the map module 154;

[0210] o Icon 638 labeled “Weather” of the Weather widget 149-1;

[0211] o Icon 640 labeled “Clock” of the alarm clock widget 169-6;

[0212] o an icon 642 labeled “Fitness Support” of the fitness support module 142;

[0213] o An icon 644 of the notepad module 153 labeled "Notepad"; and

[0214] o An icon 646 of a settings application or module that provides access to settings for the device 100 and its various applications 136.

[0215] It should be noted that Fig. 6A The icon labels shown in are exemplary only. For example, in some embodiments, icon 622 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0216] Figure 6B A touch-sensitive surface 651 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contact on touch-sensitive surface 651, and / or one or more tactile output generators 359 for generating tactile output for a user of device 300.

[0217] Figure 6B A touch-sensitive surface 651 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3 Although many of the embodiments that follow will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 6B In some embodiments, the touch-sensitive surface (e.g., Figure 6B 651) has a main axis (e.g., Figure 6B 653) corresponding to the principal axis (for example, Figure 6B According to these embodiments, the device detects contact with the touch-sensitive surface 651 at a location that corresponds to a corresponding location on the display (e.g., Figure 6B 660 and 662 in (e.g., in Figure 6B , 660 corresponds to 668 and 662 corresponds to 670). Thus, on a touch-sensitive surface (e.g., Figure 6B 651) and a display of a multi-function device (e.g., Figure 6B When the user input detected by the device on the touch-sensitive surface (e.g., contacts 660 and 662 and their movement) is separated, the device is used to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0218] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contact, finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact); or by stylus contact, followed by movement of the stylus along the path of the swipe. As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detection of contact and then ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple styluses are optionally used simultaneously, or finger and stylus contact or mouse and finger contact are optionally used simultaneously.

[0219] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functions that the user cannot easily access on a smaller device with limited real estate for displaying indications (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical controls / mechanical controls such as knobs or buttons).

[0220] In some embodiments, the contact / motion module 130 and / or 430 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some embodiments, a software setting is provided to the user of the device for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0221] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the median value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more strength thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding option or give up performing the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0222] In some embodiments, a portion of a gesture is identified for determining a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an end position (e.g., a drag gesture), where the strength of the contact increases. In this example, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the swipe contact portion at the end position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or depressions in the strength of the swipe contact for determining the characteristic strength.

[0223] The user interface drawings described below (e.g., 7A to 7OO ) optionally includes various intensity graphs showing the intensity of contacts on the touch-sensitive surface relative to one or more intensity thresholds (e.g., contact detection intensity threshold IT 0 , Light press intensity threshold IT L , Deep compression intensity threshold IT D and / or one or more other intensity thresholds). This intensity graph is typically not part of the displayed user interface, but is provided to help interpret the graph. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or a trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from that typically associated with clicking a button of a physical mouse or a trackpad. In some embodiments, when a characteristic intensity is detected that is below the light press intensity threshold (e.g., and above the nominal contact detection intensity threshold IT 0 , contacts lower than the nominal contact detection intensity threshold are no longer detected), the device moves a focus selector (e.g., a cursor) based on the movement of the contact on the touch-sensitive surface without performing actions associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.

[0224] In some embodiments, the response of the device to the input detected by the device depends on a criterion based on the contact strength during the input. For example, for some "light press" inputs, the strength of the contact exceeding the first strength threshold during the input triggers the first response. In some embodiments, the response of the device to the input detected by the device depends on a criterion including both the contact strength during the input and the time-based criterion. For example, for some "deep press" inputs, the strength of the contact exceeding the second strength threshold greater than the first strength threshold of the light press during the input triggers the second response only if a delay time has passed between meeting the first strength threshold and meeting the second strength threshold. The duration of the delay time is typically less than 200ms (e.g., 40ms, 100ms, or 120ms, depending on the magnitude of the second strength threshold, where the delay time increases as the second strength threshold increases). This delay time helps avoid accidental deep press inputs. For another example, for some "deep press" inputs, a period of reduced sensitivity will occur after reaching the first strength threshold. During this period of reduced sensitivity, the second strength threshold is increased. This temporary increase in the second strength threshold also helps avoid accidental deep press inputs. For other deep press inputs, the response to detecting the deep press input is not dependent on time-based criteria.

[0225] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Exemplary factors are described in U.S. Patent Application Serial Nos. 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0226] The contact characteristic strength is below the light press strength threshold IT L The intensity increases to between the light pressure intensity threshold IT L and deep compression intensity threshold IT D The intensity between IT and the deep press intensity threshold is sometimes referred to as a "light press" input. D The intensity of the deep compression increases to above the deep compression intensity threshold IT D The strength of the contact feature is sometimes called a "deep press" input. The contact feature strength is below the contact detection strength threshold IT 0 The intensity increases to between the contact detection intensity threshold IT 0 With light pressure intensity threshold IT L The strength between IT and the contact detection strength threshold IT is sometimes referred to as detecting contact on the touch surface. 0The intensity decreases to below the contact detection intensity threshold IT 0 The strength of IT is sometimes referred to as detecting lift-off of the contact from the touch surface. 0 In some embodiments, IT 0 Greater than zero In some illustrations, shaded circles or ovals are used to represent the intensity of contacts on the touch-sensitive surface. In some illustrations, unshaded circles or ovals are used to represent corresponding contacts on the touch-sensitive surface without specifying the intensity of the corresponding contacts.

[0227] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the strength of the corresponding contact increases to above the press input strength threshold (e.g., the corresponding operation is performed on a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and the strength of the contact subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., the corresponding operation is performed on an "up stroke" of the corresponding press input).

[0228] In some embodiments, the device employs intensity hysteresis to avoid unexpected input, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., performing the corresponding operation on an "up stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the strength of the contact increases from an strength equal to or below a hysteresis strength threshold to an strength equal to or above a press input strength threshold and, optionally, the strength of the contact subsequently decreases to an strength equal to or below the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (for example, depending on the circumstances, the strength of the contact increases or the strength of the contact decreases).

[0229] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input are optionally triggered in response to detecting that the intensity of the contact increases above the press input intensity threshold, the intensity of the contact increases from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, the intensity of the contact decreases below the press input intensity threshold, or the intensity of the contact decreases below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where operations are described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operations are optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold. As described above, in some embodiments, triggering of these operations is also dependent on satisfying time-based criteria (e.g., a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold).

[0230] User interface and associated processes

[0231] Attention now turns to embodiments of a user interface ("UI") and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300 having a display, a touch-sensitive surface, and one or more sensors for detecting intensity of contacts with the touch-sensitive surface.

[0232] 7A to 7OO An exemplary user interface for updating a user interface based on a merged and / or predicted touch position according to some embodiments is shown. The user interfaces in these figures are used to illustrate the process described below, including FIG. 8A to FIG. 8B , 9A to 9D , FIG. 10A to FIG. 10C ,and Fig.11 Although some of the examples that follow will be given with reference to input on a touch screen display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface 651 that is separate from the display 650, such as Figure 6B as shown in .

[0233] In some embodiments, the display 650 is configured to display the user interface at a specific display rate (also referred to as a display refresh rate). For example, a specific monitor displays the user interface at 60 Hz (e.g., at a rate of 60 updates per second). In some embodiments, the display rate is a fixed display rate. As used herein, a display frame refers to a user interface displayed during a single display cycle (e.g., a user interface that is displayed for ~0.1667 seconds and then replaced by the next user interface when the display rate is 60 Hz).

[0234] In some embodiments, the touch-sensitive surface 651 is configured to detect touch input at a specific detection rate. In some embodiments, the detection rate is a fixed detection rate (e.g., 60 Hz).

[0235] In some cases, there is a delay between detecting a touch input and updating the user interface in response to the touch input, such as a delay in processing the touch input and / or a delay in preparing the user interface (before rendering the user interface). Such delays can result in a mismatch between the physical location of the touch input on the touch screen 112 and the update of the user interface displayed on the touch screen 112. An example of this is in 7A to 7E Shown in.

[0236] Fig. 7A FIG. 1 shows a user interface of a drawing application on the touch screen 112. Fig. 7A , touch input 705 is detected on touch screen 112 at position 705 -A. Fig. 7A Also shown is that in response to detecting touch input 705 , a point is displayed 702 on touch screen 112 .

[0237] Figure 7B It is shown that touch input 705 has moved on touch screen 112 to position 705-B on touch screen 112. Figure 7B , line 704-1 is displayed according to the movement of touch input 705 from position 705-A to position 705-B. However, due to the delay in processing the touch input, the endpoint of line 704-1 does not exactly match the position of touch input 705 (e.g., position 705-B) as touch input 705 is moving on touch screen 112.

[0238] Figure 7B A diagram is also included to illustrate this effect. Figure 7B In the illustration of , the movement of touch input 705 is represented by increasing travel distances 740-1, 740-2, and 740-3 over time. In some embodiments, due to delay, when the travel distance is at 740-2, the user interface prepared based on the travel distance at 740-1 is displayed (750-1). Similarly, when the travel distance is at 740-3, the travel distance 740-2 is used as the length 750-2 of line 704-1, and there is a difference between the actual travel distance (e.g., 740-3) and the displayed length of line 704-1 drawn according to the touch input (e.g., 750-2). 1. In some embodiments, these travel distances correspond to travel distances during a constant time interval (e.g., a single display update frame, or the amount of time between when a touch is detected on the touch-sensitive surface and when a display update is generated in response to the touch to account for changes to the user interface), such that different distances correspond to different speeds (e.g., larger changes in distance over corresponding time intervals correspond to faster movement of the contact on the touch-sensitive surface, while shorter changes in distance over corresponding time intervals correspond to slower movement of the contact on the touch-sensitive surface).

[0239] Figure 7C Continued movement of touch input 705 to position 705-C on touch screen 112 is shown. In the illustration, when the travel distance is at 740-4, travel distance 740-3 is used as length 750-3 of line 704-2.

[0240] Fig.7D A faster movement of the touch input 705 to position 705-D on the touch screen 112 is shown. When the travel distance is at 740-6, the travel distance 740-5 is used as the length 750-5 of the line 704-3, and there is a difference between the actual travel distance (e.g., 740-6) and the displayed length of the line 704-3 (e.g., 750-5). 2 .

[0241] like FIG. 7B to FIG. 7D As shown, when the touch input moves at a low speed on the touch screen 112, the difference is relatively small and the difference is less noticeable. However, the difference increases as the touch input moves faster on the touch screen 112, and the difference becomes more noticeable.

[0242] Fig. 7E The touch input 705 is shown to remain at position 705-D on the touch screen 112 for a period of time (e.g., one or more frames of the display rate) corresponding to the delay between receiving the touch input and displaying a user interface updated based on the touch input. While the touch input 705 remains at position 705-D on the touch screen 112, a user interface prepared based on the travel distance 740-6 is displayed, wherein the length 750-7 of the line 704-4 matches the travel distance 740-8 of the touch input 705.

[0243] As described above, the delay between receiving a touch input and displaying a user interface updated based on the touch input results in a mismatch between the physical location of the touch input and the location of the touch input reflected in the displayed user interface.

[0244] Although in FIG. 7B to FIG. 7E In some embodiments, the delay between receiving a touch input and displaying a user interface reflecting the state (e.g., position) of the touch input corresponds to a single display frame, but in some embodiments, the delay corresponds to more than one display frame.

[0245] FIG. 7F to FIG. 7O is a timing diagram showing the timing of receiving touch input (or touch input changes) and the timing of displaying (or updating) a user interface according to some embodiments. FIG. 7F to FIG. 7O , the upward arrows (eg, 720 and 722) represent the timing of a touch input (or a touch input change), and the downward arrow 730 represents the timing of displaying (or updating) the user interface.

[0246] Figure 7F The receiving of touch input at different times 720-1, 720-2, 720-3, and 720-4 and updating of the user interface at times 730-1, 730-2, 730-3, and 730-4 are shown. Figure 7F In , the detection rate and the display rate are the same (for example, both the detection rate and the display rate are 60 Hz). Figure 7F In the embodiment, the detection rate and the display rate are synchronized (e.g., there is no phase difference). For example, the touch input received at 720-1 occurs simultaneously with the update of the user interface at 730-1. Similarly, the touch input received at 720-2 occurs simultaneously with the update of the user interface at 730-2.

[0247] Figure 7F It is also shown that the touch input received at 720-1 is reflected after one display frame (e.g., 730-2), after two display frames (e.g., 730-3), or after three display frames (e.g., 730-4). In some embodiments, the received touch input is reflected after more than three display frames. In some embodiments, the number of display frames between receiving the touch input and displaying the user interface reflecting the touch input is determined based on the delay in processing the received touch input and preparing the user interface for display. In some embodiments, the delay between receiving a touch input and displaying a user interface that is updated based on the touch input includes the time taken for one or more of the following: the time taken for the touch sensor hardware to scan to detect a touch, the time taken for the contact motion module to identify the touch and determine the application to which to deliver the touch (or a touch event associated with the touch); the time taken for the application to process the touch to determine how to update the display in response to the touch, and to generate display update information; the time taken for the operating system graphics component (e.g., a display driver) to render display frames based on the display update information from the application; the time taken for the graphics processing unit (GPU) to convert the display frames generated by the operating system graphics component into display update instructions (e.g., a pixel map to be displayed on the display); and the display refresh operation (e.g., the time taken to actually update the pixels of the display to reflect the display update instructions and display the updates to the user interface generated by the application in response to receiving the touch).

[0248] Figure 7G and Figure 7FSimilar, except that the detection rate and display rate are not synchronized. Figure 7G In the example, the detection rate and the display rate are the same (for example, both the detection rate and the display rate are 60 Hz). However, there is a phase difference between the detection timing and the display timing. For example, the timing of receiving the touch input deviates from the timing of updating the user interface.

[0249] Figure 7G It is also shown that the touch input received at 720-1 is reflected in the next display frame (e.g., 730-2), in the display frame after the next display frame (e.g., 730-3), or in two display frames after the next display frame (e.g., 730-4). In some embodiments, the received touch input is reflected in the display frame after 730-4.

[0250] FIG. 7F to FIG. 7G The timing of touch input and display update in a conventional device according to some embodiments is shown. Such conventional devices are subject to the above reference 7A to 7E The delay described.

[0251] FIG. 7H to FIG. 7O A timing diagram showing a detection rate (e.g., a touch detection rate corresponding to the rate at which a touch-sensitive surface is scanned for changes in touch data that are interpreted as touches and touch events) that is higher than a display rate (e.g., a display refresh rate corresponding to the frequency at which pixels on a display are refreshed) according to some embodiments.

[0252] Figure 7H It is shown that touch input 722 is received at a detection rate four times the display rate. Figure 7H In , touch input is received four times when the display is updated once (e.g., the detection rate is 240Hz and the display rate is 60Hz).

[0253] In some embodiments, touch input is received at a rate that is less than the rate at which the touch-sensitive surface (or one or more touch sensors) is capable of receiving touch input. Fig.7IIt is shown that when the touch-sensitive surface is capable of receiving touch input at a detection rate four times the display rate (e.g., the touch-sensitive surface is capable of receiving (or detecting) touch input four times within each display frame), the touch input is received only twice within each display frame (e.g., the touch input is received at 722-3 and 722-5, and the device does not receive touch input at 722-2 and 722-4). In some embodiments, whether to receive more or fewer touch inputs within each display frame is determined by the device based on predefined criteria such as accuracy requirements, available computing resources, etc. In some embodiments, the number of touch inputs received within each display frame is dynamically adjusted by the device based on predefined criteria (e.g., more touches are received when the difference between the currently detected touch position and the actual position of the touch may be greater than a predetermined amount, such as when the touch moves quickly or a larger difference has been detected in the most recent display frame, and fewer touches are received when the difference between the currently detected touch position and the actual position of the touch is less than a predetermined amount, such as when the touch moves slowly or a smaller difference has been detected in the most recent display frame).

[0254] Figures 7J to 7K Touch detection frames, touch processing frames, and touch display frames are shown according to some embodiments.

[0255] Figure 7J It is shown that the touch detection frame 724-1 includes touch inputs at 722-2, 722-3, 722-4, and 722-5, and the touch detection frame 724-2 includes touch inputs at 722-6, 722-7, 722-8, and 722-9.

[0256] Figure 7J Touch processing frames 732-1, 732-2, 732-3, 732-4, and 732-5 are also shown. In some embodiments, the touch processing frame completely overlaps with the touch detection frame (e.g., touch processing frame 732-2 completely overlaps with touch detection frame 724-1). In some embodiments, the touch input (or corresponding position) in touch detection frame 724-1 is processed during touch processing frame 732-2 (overlapping with touch detection frame 724-1) and displayed during display frame 734-2. In some embodiments, the touch input (or corresponding position) in touch detection frame 724-1 is processed during touch processing frame 732-3 (not overlapping with touch detection frame 724-1) and displayed during display frame 734-3.

[0257] Figure 7K It is shown that the touch detection frame 724-3 includes touch inputs at 722-1, 722-2, 722-3, and 722-4, and the touch detection frame 724-4 includes touch inputs at 722-5, 722-6, 722-7, and 722-8. Figure 7KIn some embodiments, touch detection frames are not aligned with display frames (e.g., touch detection frame 724-4 includes touch 722-5 received during display frame 734-1 and touch 722-6 received during display frame 734-2). In some embodiments, touch input in touch detection frame 724-3 is processed during touch processing frame 732-2 that partially overlaps with touch detection frame 724-3, and is displayed during display frame 734-2.

[0258] Figure 7L and Figure 7M Touch input selection based on timing criteria according to some embodiments is shown.

[0259] Figure 7L A group of touch inputs is shown selected based on a first predefined timing criterion. For example, touch input group 724-5 is selected so that the touch inputs in group 724-5 are at least a certain timing margin (eg, Δt) ahead of the next update of the user interface. Figure 7L , the last touch input in group 724 - 5 is earlier than the timing 730 - 3 for updating the user interface by a certain timing margin (eg, Δt). Figure 7L It is also shown that the same timing margin is used for each selection. For example, touch input group 724-6 is selected so that the touch inputs in group 724-6 are also ahead of the next update 730-4 to the user interface by at least the same specific timing margin (eg, Δt).

[0260] Figure 7M The use of different timing margins is shown. Figure 7M In the example, the touch input group 724-7 is earlier than the timing 730-2 for updating the user interface by a timing margin Δt. 1 The first time point is earlier than the timing 730-3 for updating the user interface by a timing margin Δt. 2 The touch input in group 724-8 is after the touch input in the previous group (ie, group 7247-7) and is earlier than the timing 730-4 for updating the user interface by a timing margin Δt 3 In some embodiments, different timing margins are used to define corresponding touch processing frames, such as Figure 7M shown.

[0261] Figure 7N Representative touch selections according to some embodiments are shown. Figure 7N, touch input set 724-5 includes touch inputs 722-3, 722-4, 722-5, and 722-6, and touch input 722-6 is selected as the representative touch of touch input set 724-5 (and the corresponding touch position is selected as the representative touch position). Similarly, touch input set 724-6 includes touch inputs 722-7, 722-8, 722-9, and 722-10, and touch input 722-10 is selected as the representative touch of touch input set 724-6 (and the corresponding touch position is selected as the representative touch position). Figure 7N , the last touch input in touch input set 724-5 (e.g., touch input 722-6) is selected as the representative touch of touch input set 724-5. Alternatively, the first touch input in touch input set 724-5 (e.g., touch input 722-3) or any other touch input in touch input set 724-5 is selected as the representative touch of touch input set 724-5. Touch inputs located between representative touches (e.g., touch inputs 722-7, 722-8, and 722-9 between representative touches 722-6 and 722-10) are referred to herein as gap touches (and their corresponding positions are referred to herein as gap touch positions).

[0262] The selection of a representative touch has the effect of shifting the touch processing frame. For example, selecting touch 722-6 as the representative touch for update time 730-3 has the effect of shifting the touch processing time by one-quarter of a display update frame (e.g., ~16.667 ms) relative to selecting touch 722-5 as the representative touch. By shifting the touch processing time (by selecting touch 722-6 as the representative touch), the difference between the detected touch and the displayed user interface is reduced (e.g., because more recent touch information from the shifted touch processing time is used to update the user interface than touch information without the touch processing time shift). If there is sufficient processing margin time between touch 722-7 and update time 730-3, selecting touch 722-7 as the representative touch for update time 730-3 will have the effect of shifting the touch processing time by half a display update frame (e.g., ~33.333 ms) and further reduce the difference between the detected touch and the displayed user interface. However, the touch 722-7 is not ahead of the update time 730-3 by the processing margin time Δt, and therefore, the touch 722-7 is not selected as the representative touch for the update time 730-3.

[0263] Fig.7O The touch input set 724-7 is shown to include detected touches such as representative touch 722-6 and gap touches 722-3, 722-4, and 722-5. In addition, the set 724-7 includes one or more predicted touches. Fig.7OIn some embodiments, set 724-7 includes predicted interstitial touches 722-7, 722-8, and 722-9. In some embodiments, set 724-7 includes one or more predicted representative touches (eg, touch 722-10).

[0264] Figure 7P Using interstitial touch locations when updating a user interface according to some embodiments is shown. Figure 7P and Fig.7D Similarly, the difference is that the gap positions 742-1, 742-2, 742-3, 742-4 and 742-5 are used to update the user interface. When the travel distance is at 740-6, the travel distance 742-5 (instead of Fig.7D The travel distance 740-5 shown in FIG. 7 is used as the length 750-5 of line 704-5, and the difference d between the actual travel distance (eg, 740-6) and the displayed length (eg, 750-5) of line 704-5 is 3 Reduce (for example, Fig.7D The difference shown in 2 Small).

[0265] Figures 7Q to 7OO The use of predicted touch locations and associated operations according to some embodiments is shown.

[0266] Figure 7Q to Figure 7V It is shown that the use of predicted touch locations significantly reduces or eliminates the difference between detected touch locations and displayed touch locations.

[0267] When the touch input 707 is from position 707-A (such as Figure 7Q ) moves to position 707-B (as shown Figure 7R ), the user interface may not be immediately updated to reflect the actual location of touch input 707 (e.g., 707-B) due to various delays. Rather than waiting for the actual location of touch input 707 to be available, the device makes a prediction of the location of touch input 707. Figure 7R , predicted position 710-1 matches actual position 707-B of touch input 707, and any difference between the actual position of touch input 707 and the displayed position of touch input 707 (eg, the endpoints of line 704-6) is reduced or eliminated.

[0268] Similarly, Figure 7S to Figure 7V It is shown that the use of the predicted positions reduces or eliminates the difference between the detected touch positions and the displayed touch positions.

[0269] Figures 7W to 7BB and Figure 7Q to Figure 7VSimilarly, the difference is that one or more user interface objects corresponding to one or more predicted touch locations are visually distinguished (displayed by different colors, line widths, shapes, shading, highlights, line styles, etc.) from one or more user interfaces independent of the one or more predicted touch locations.

[0270] exist Figure 7W , from the initial position of the touch input 707 (eg Figure 7Q Line 706-1 extending from position 707-A in FIG. 707 to predicted position 710-1 of touch input 707 is depicted as a thin dashed line. Figure 7X , after touch input 707 is received at the actual location of position 707-B, line 706-1 representing the predicted movement of touch input 707 is replaced by line 708-1 (e.g., a thick solid line) that is visually distinguishable from 706-1. Figure 7W ) indicates the predicted position 710-1 ( Figure 7W ) matches the actual position 707-B of the touch input 707.

[0271] Figure 7X Also shown is line 706 - 2 extending from position 707 -B to predicted position 710 - 2 of touch input 707 , displayed visually differently to indicate that line 706 - 2 is based on the predicted position of touch input 707 (eg, position 710 - 2 ).

[0272] Similarly, Figure 7Y to Figure 7BB A user interface object or portion thereof is shown visually distinct to indicate that such user interface object (or portion) is based on one or more predicted touch locations.

[0273] Although Figure 7Q to Figure 7BB An embodiment is shown in which the predicted position matches the actual position of the touch input, but the prediction need not be completely accurate. For example, a predicted position that is between the previous position of the touch input and the current position of the touch input (rather than exactly at the current position of the touch input) reduces the difference between the current position of the touch input and the displayed user interface, thereby improving the accuracy of the displayed user interface.

[0274] Figures 7CC to 7FF A user interface associated with a predicted touch location according to some embodiments is shown.

[0275] exist Figure 7CC , line 706-3 extends to predicted position 710-3 corresponding to touch position 707-D of touch input 707. Then, as Figure 7DDAs shown, line 706-4 extends to predicted position 710-4, and touch input 707 moves to position 707-G that does not correspond to predicted position 710-4.

[0276] Figure 7EE Line 706-4 is shown removed (e.g., because predicted position 710-4 does not correspond to actual position 707-G of touch input 707, optionally with a distance between predicted position 710-4 and actual position 707-G being greater than a predefined threshold). Instead, line 708-6 corresponding to actual position 707-G of touch input 707 is displayed.

[0277] Figure 7EE Also shown is a line 706-6 extending to the predicted position 710-6 displayed on the touch screen 112. The touch input 707 moves to a position 707-H that does not correspond to the predicted position 710-6. In some embodiments, based on determining that the distance between the predicted position 710-6 and the actual position 707-H is less than a predefined threshold, the line 706-6 is included in the line 708-7 to indicate that the predicted position 710-6 does not deviate from the actual position 707-H by more than the predefined threshold, such as Figure 7FF shown.

[0278] Figure 7GG and Figure 7HH Similar to Figure 7DD and Figure 7EE , the difference is, Figure 7GG and Figure 7HH The user interface in does not include user interface objects (or portions thereof) that are visually distinguished to indicate that the user interface objects (or portions thereof) correspond to one or more predicted touch locations.

[0279] Fig.7II Detection of touch input (e.g., movement of contacts on touch screen 112) is shown at multiple locations on touch screen 112. These locations include detected touch locations (e.g., detected representative touch locations 712-1, 712-2, and 712-3, and detected interstitial touch locations 714-1 through 714-6) and predicted touch locations (e.g., predicted representative touch locations 716-1 and 716-2, and predicted interstitial touch locations 718-1 through 718-6).

[0280] In some embodiments, device 100 includes one or more sensors for detecting the intensity of contacts applied on touch screen 112. Fig.7II Also shown is detecting the intensity of a contact at multiple locations (eg, 760) and, optionally, determining a predicted intensity (eg, 762).

[0281] In some embodiments, device 100 includes one or more sensors for detecting a tilt and / or orientation of a stylus associated with device 100 . Fig.7II Determining the tilt and / or orientation of the stylus (eg, 770 ) and, optionally, determining a predicted tilt and / or orientation (eg, 772 ) are shown.

[0282] In some embodiments, the amount of time spent detecting the intensity of contact applied by a touch on touch screen 112, detecting the tilt of the stylus, and / or detecting the orientation of the stylus is different than the amount of time spent detecting the location of a touch input. In some embodiments, one or more sensors used to detect the intensity of contact applied by a touch on touch screen 112, detecting the tilt of the stylus, and / or detecting the orientation of the stylus have a longer latency than one or more sensors used to detect the location of a touch input. Thus, in some embodiments, a predicted intensity, predicted tilt, and / or predicted orientation is used even for a "known" (e.g., detected and processed) touch location.

[0283] Figure 7JJ It is shown that the number of predicted locations is determined based at least in part on the confidence level associated with the predicted touch. For example, the difference between the predicted location and the detected location (e.g., the difference between the predicted location 718-1 and the detected location 714-7 and / or the difference between the predicted location 718-2 and the detected location 714-8) is used to determine the number of predicted locations. When the difference increases, the number of predicted locations decreases (e.g., because the prediction is less reliable), and when the difference decreases, the number of predicted locations increases (e.g., because the prediction is more reliable).

[0284] Figure 7KK The number of predicted locations is shown to be determined based at least in part on a confidence level associated with the detected touch. For example, a change in the detected touch (e.g., relative to Figure 7KK An increase in the deviation from the reference line shown decreases the number of predicted touches, and a decrease in the variation of the detected touches increases the number of predicted touches.

[0285] Figure 7LL The number of predicted positions is shown to be determined based at least in part on the speed of the touch input. For example, as the speed of the touch input decreases, the number of predicted touches decreases (e.g., because the slower the speed, the more jitter there may be), and as the speed of the touch input increases, the number of predicted touches increases (e.g., because a touch input such as a finger or stylus is less likely to suddenly change direction). In general, a fast-moving touch input has a smaller difference or error between the predicted touch and the actual touch than a slow-moving touch input. In addition, as described above with reference to FIG. 7B to FIG. 7EAs described above, without the use of merged touches or predicted touches, fast-moving touch inputs suffer from large discrepancies between actual touches and displayed user interfaces more than slow-moving touch inputs. Such large discrepancies can be reduced or eliminated using predicted touches, which are generally more accurate for fast-moving touch inputs than for slow-moving touch inputs. Therefore, it is particularly advantageous to use predicted touches for fast-moving touch inputs, thereby significantly reducing or eliminating the discrepancy between detected touches and displayed user interfaces.

[0286] Although Figure 7II to Figure 7LL The predicted positions are shown arranged in a straight line, but the predicted positions are not limited to a straight line. Figure 7MM It is shown that the detected positions 712-12 and 712-13 and 714-19 to 714-22 define a curve, and the predicted positions (eg, 718-11 to 718-13 and 716-5) on the curve are determined.

[0287] Figures 7NN to 7OO Some embodiments are shown in which a line is drawn with touch input. Figures 7NN to 7OO In the example, the width of the corresponding portion of the line is based on the intensity of the touch input applied at the position corresponding to the corresponding portion of the line. For example, moving the touch input with high intensity causes a wide line to be drawn, and moving the touch input with low intensity causes a thin line to be drawn.

[0288] Figure 7NN It is shown that the touch input 709 moves on the touch screen 112 and the corresponding line is displayed on the touch screen 112. The movement of the touch input 709 includes a first portion 780 and a second portion 782. Figure 7NN , touch screen 112 displays a portion 784 of the line that corresponds to a first portion 780 of the movement of touch input 709 and a portion 786 of the line that corresponds to the predicted movement of touch input 709 (e.g., portion 786 is based on the predicted location and the predicted intensity). For example, while touch input 709 has followed path 780 and path 782, the movement of touch input 709 along path 780 is detected and processed such that portion 784 that corresponds to first portion 780 of the actual movement of touch input 709 is displayed. The movement of touch input 709 along path 782 is not reflected in the user interface, and instead, portion 786 of the line that is based on the predicted touch information (e.g., the predicted location and intensity) is displayed.

[0289] Figure 7OO Shown in the display Figure 7NN782, the user interface is updated such that portion 786 of the line is replaced by portion 788 of the line based on the detected movement of touch input 709 (e.g., portion 788 is based on detected movement 782 of touch input 709, including the detected position and the detected intensity of touch input 709). The change in width in portion 788 of the line indicates the actual change in the intensity of touch input 709 as touch input 709 follows path 782. In some embodiments, portion 786 of the line is replaced by portion 788 of the line in a smooth transition (e.g., sub-portions of portion 786 are sequentially replaced by sub-portions of portion 788), rather than suddenly replacing portion 786 with portion 788 all at once.

[0290] Although 7A to 7OO Processing finger input is shown, but those skilled in the art will understand that 7A to 7OO The method and user interface shown in can be applied to handle stylus input and / or mouse input in a similar manner. For the sake of brevity, such details are not repeated here.

[0291] FIG. 8A to FIG. 8B A flowchart of a method 800 for updating a user interface based on a merged touch position according to some embodiments is shown. The method 800 is performed on an electronic device (e.g., device 300, Figure 3 ; or a portable multifunction device 100, Figure 1A ) is performed on a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0292] As described below, method 800 provides a way to update a user interface based on merged touch locations. The method reduces the difference between the detected touch input and the displayed user interface, thereby reducing the cognitive burden on the user when interacting with the touch screen. In addition, this generates a more efficient human-computer interface. For battery-powered electronic devices, enabling users to interact with the user interface more accurately reduces errors and unnecessary corrections, thereby saving power and increasing the time between battery charges.

[0293] The device displays (802) a user interface at a first display rate (see Figure 7P ).

[0294] While displaying the user interface, the device detects (804) movement of touch input at a series of locations on the touch-sensitive surface at a first detection rate that is higher than a first display rate. Figure 7H In some embodiments, the first detection rate is an integer multiple of the first display rate (e.g., the first display rate is 60 Hz and the first detection rate is 240 Hz, which is four times 60 Hz). Touch input can be monitored at a rate (e.g., 240 Hz or 250 Hz) that is lower than the maximum scan rate of the touch-sensitive surface (e.g., 500 Hz). For example, in Fig.7I In , instead of monitoring touch input at four times the display rate, touch input is monitored at twice the display rate.

[0295] At each update time in a series of update times, the device updates (806) the user interface from a corresponding current state to a corresponding next state based on a selected subset of the series of positions of the touch input, each selected subset of the series of positions including multiple positions of the touch input. Figure 7J As shown, the user interface is updated at each update time in a series of update times (e.g., at 730-1, 730-2, 730-3, and 730-4). In some embodiments, the corresponding next state of the user interface is different from the corresponding current state of the user interface (e.g., the user interface changes over time so that the corresponding next state of the user interface appears different from the corresponding current state of the user interface). In some embodiments, based on the processing time, the user interface may be updated in one or more frames immediately following the frame in which the last position in the first set of sequential positions is detected. In some embodiments, this series of update times is determined based on a first display rate. For example, for a user interface displayed at 60 Hz, the user interface is updated approximately every 16.67 ms (≈1 / (60 Hz)).

[0296] In some embodiments, the device sends (808) a message to the first software application having information including the selected subset of the series of locations, and the first software application updates the user interface based on the information in the message. Figure 7J In the example embodiment, touch input set 724-1 is sent to a first software application (eg, a drawing application) at 730-2, and the first software application updates a user interface according to the information in set 724-1 at 730-3.

[0297] In some embodiments, the message also includes (810) information identifying one or more of the following: the intensity of the touch input at the selected subset of the series of locations (e.g., Figure 1Dand the type of touch input detected at a selected subset of the series of locations (e.g., whether the touch input was performed by a finger or a stylus, such as Figure 1D In some embodiments, the message also includes information identifying the timing of the touch input at a selected subset of the series of locations (e.g., Figure 1D timestamp 256).

[0298] In some embodiments, the last detected position in each selected subset is at the next update time (e.g. Figure 7L ) is detected (812) at least a predefined time interval (e.g., a processing margin time) before. In such embodiments, the last detected position in each selected subset is processed and used to update the user interface at the next update time.

[0299] In some embodiments, the device updates (814) the user interface from the first state to the second state based on a first subset of the series of positions of the touch input. In some embodiments, the second state of the user interface is different from the first state of the user interface. After updating the user interface from the first state to the second state, the device updates the user interface from the second state to a third state based on a second subset of the series of positions of the touch input. In some embodiments, the third state of the user interface is different from the second state of the user interface. For example, Figure 7P A series of changes to the user interface are shown (represented by the length of the displayed line increasing over time based on the touch input). In some embodiments, the second subset of the series of positions for the touch input is different from the first subset of the series of positions for the touch input. In some embodiments, the second subset of the series of positions for the touch input does not overlap with the first subset of the series of positions for the touch input. For example, the first subset of the series of positions for the touch input and the second subset of the series of positions for the touch input do not include common positions for the touch input. In some embodiments, the first subset of the series of positions for the touch input and the second subset of the series of positions for the touch input include different numbers of positions.

[0300] In some embodiments, the device selects (816, Figure 8B ) as a representative touch position (e.g., Figure 7N Representative touch 722-6).

[0301] In some embodiments, the amount of time required for the first application to update the user interface is indicated (e.g., as described below with reference to 9A to 9DIn some embodiments, the touch handling criteria of the first application (described in more detail) selects (818) the corresponding touch location as the representative touch location. In some embodiments, the amount of time required for the first application is checked when the first application is launched. In some embodiments, the check of the amount of time required for the first application is repeated (e.g., at predefined intervals). In some embodiments, the amount of time required for the first application changes over time. In some embodiments, the longest amount of time required for the first application from the repeated checks of the amount of time selected for the first application is used.

[0302] In some embodiments, the first application updates the user interface including (820) associating the selected subset of the series of locations of the touch input with an indication of which location is a representative touch location (e.g., Figure 1D The first application may transmit the representative touch location to the first application together with an identifier 214 of the representative touch in the plurality of locations. For example, by identifying a representative touch location among the plurality of locations, an application that does not expect to receive the plurality of touch locations may simply use the representative touch location, while an application that knows that the plurality of touch locations are expected may use the representative touch location and other (interstitial) touch locations to provide a more accurate and responsive user interface.

[0303] In some embodiments, the selected subset of the series of positions includes (822) one or more gap positions corresponding to touch positions between the previous representative touch position and the representative touch position (e.g., Figure 7N The gaps in the touch screen are 722-3, 722-4 and 722-5).

[0304] In some embodiments, for each update time in the series of update times, the device selects (824) multiple locations of touch input for updating the user interface. The selected locations are locations of touch input detected after the last time a location was selected for updating the user interface (e.g., Figure 7M , the touches in set 724-8 are detected after the touches in set 724-7 are selected).

[0305] In some embodiments, the selected plurality of locations of the touch input includes (826) one or more predicted gap locations (e.g., Fig.7O The predicted gap touches 722-7, 722-8 and 722-9 in FIG. Figures 10A-10C described in more detail).

[0306] In some embodiments, for each update time in the series of update times, the device selects (828) multiple locations of touch input for updating the user interface. The selected locations are locations of touch input detected after the location of the touch input that was last selected for updating the user interface was detected (e.g., within Figure 7M , the touches in set 724-8 are detected after the touches in set 724-7 are detected). Alternatively, the selected location is a location of a touch input in the series of locations that is subsequent to a location of a touch input that was last selected for updating the user interface.

[0307] In some embodiments, the selected positions include (830) all positions of touch input after the position of the touch input last selected for updating the user interface was detected. Alternatively, the selected positions include all positions of touch input detected no later than the update time, including the position of the touch input in the series of positions after the position of the touch input last selected for updating the user interface.

[0308] In some embodiments, the selected position includes (832) only one of the positions of the touch input detected after the position of the touch input last selected for updating the user interface was detected (e.g., only a single representative touch position is used). In some embodiments, the selected position includes two or more sequential positions. In some embodiments, only one position is the most recently detected position. In some embodiments, the selected position includes the most recently detected position and the previous position.

[0309] In some embodiments, the positions of touch inputs that have not been selected are discarded. In some embodiments, the positions of touch inputs that have not been selected are included in the subsequently selected subset.

[0310] It should be understood that FIG. 8A to FIG. 8B The specific order in which the operations in the description are described is merely exemplary and is not intended to represent that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein in conjunction with other methods described herein (e.g., methods 900, 1000, and 1100) are also applicable in a similar manner to the details of the processes described above in conjunction with FIG. 8A to FIG. 8B The method 800 described herein. For example, the touch information, predicted touch positions, representative touch positions, gap touch positions, and information sending operations described above with reference to the method 800 may optionally have one or more of the features of the touch information, predicted touch positions, representative touch positions, gap touch positions, and information sending operations described herein with reference to other methods described herein (e.g., methods 900, 1000, and 1100). For the sake of brevity, these details are not repeated here.

[0311] 9A to 9D1 shows a flowchart of a method 900 for processing touch input with a touch processing module according to some embodiments. The method 900 is performed in an electronic device (e.g., device 300, Figure 3 ; or a portable multifunction device 100, Figure 1A ) is performed on a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0312] As described below, method 900 provides a way to process touch input using a touch processing module. The method provides an application-independent touch processing module configured to process touch input and send processed touch information to the application. Therefore, the application does not need its own instructions for processing raw touch input, and software applications that utilize representative touch locations can be developed faster and more efficiently.

[0313] The device displays (902) a user interface of a first software application (eg, Figure 7P The device may display a user interface for a drawing application as shown in FIG. 9 ) that is updated at a first display rate (e.g., a display update rate). While displaying a first frame of the user interface according to the first display rate, the device detects ( 904 ) corresponding movement of the touch input on the touch-sensitive surface (e.g., detecting movement of the contact at a first plurality of sequential locations on the touch-sensitive surface as the touch moves on the touch-sensitive surface).

[0314] The device processes the touch at an application-independent touch processing module (e.g., using an application-independent touch processing module such as Figure 1C In some embodiments, the touch processing module 220 in the embodiment performs the following operations (e.g., operations 906-912, 916-920, and 922-934). In some embodiments, the touch processing module is associated with a portion of the first software application that is specific to the first software application (e.g., Figure 1CThe touch processing module is different from the application kernel 1 (230-1) in the device. For example, the touch processing module is part of an application development framework provided to the application developer as a mix-in module that is integrated with the first application and enables the first application to interact with touch input information provided by the operating system on which the first application runs, or the touch processing module is part of the operating system that provides touch input information to the first application according to an application programming interface (API) that specifies a compatible format of the touch input information. In some embodiments, multiple different third-party applications running on the device include independent instances of the touch input module. In some embodiments, multiple different applications on the device include code for interacting with the touch processing module, and the touch processing module communicates with all third-party applications. In some embodiments, the touch processing module is separate from the first software application.

[0315] The device selects (906) a corresponding touch location of the touch input detected during the corresponding movement to identify as a representative touch location for the corresponding movement (e.g., in Figure 7N In some embodiments, the touch processing criteria of the first software application corresponds to the processing capabilities of the first software application (e.g., one or more positions in the first plurality of sequential positions are selected to be sent to the first software application based on how quickly the first software application can process touch input and generate an updated user interface for display on the display).

[0316] In some embodiments, selecting the corresponding touch position as the representative touch position includes (916, Fig. 9B ) detects a first touch location of a touch input during a touch detection frame, and in response to detecting the first location: selecting the first touch location as a representative touch location for a corresponding movement of the touch input (and optionally sending the first touch location to the first application) based on a determination that the first touch location satisfies touch processing criteria for a first application (e.g., criteria that the first application needs to process the touch input in real time to generate an updated user interface based on movement of the contact for display during a corresponding display frame before a next touch location of the touch input is detected less than a minimum amount of time); and not selecting the first touch location as a representative touch location for a corresponding movement of the touch input (e.g., not sending the first touch location to the first application, or waiting to send the first touch location to the first application) based on a determination that the first touch location does not satisfy the touch processing criteria for the first application. For example, in Figure 7N , touch 722-7 is not selected for update at 730-3 because the amount of time between touch 722-7 and update time 730-3 is less than the minimum amount of time Δt.

[0317] In some embodiments, selecting the corresponding touch location as the representative touch location includes (918) detecting a second touch location of the touch input during a touch detection frame, and in response to detecting the second location, selecting the second touch location as the representative touch location of the corresponding movement of the touch input (and optionally, sending the second touch location to the first application) based on determining that the second touch location meets touch processing criteria of the first application (e.g., criteria where the first application needs to process the touch input in real time based on movement of the contact to generate an updated user interface for displaying the next touch location that will be detected less than a minimum amount of time before the touch input is displayed during the corresponding display frame). For example, in Figure 7N , touch 722-6 is selected for the update at 730-3 because the amount of time between touch 722-6 and update time 730-3 is greater than the minimum amount of time Δt.

[0318] In some embodiments, the method includes (920): based on determining that the second touch position does not meet the touch processing criteria of the first application, not selecting the second touch position as a representative touch position for the corresponding movement of the touch input (e.g., not sending the first touch position to the first application, or waiting for the first touch position to be sent to the first application). In some embodiments, the device detects 3, 4 or more touch positions during the touch detection frame, and determines whether to identify any of the touch positions as a representative touch position based on the touch processing criteria of the first application. In some embodiments, different applications have different touch processing criteria (e.g., some applications generate updated user interfaces faster and therefore can receive later touches and still generate updated user interfaces in real time, while other applications generate user interfaces more slowly and therefore need to receive earlier touches to generate updated user interfaces in real time). In some embodiments, the same application can have different touch processing criteria at different times (e.g., depending on the amount of CPU or GPU processing bandwidth available to the first application, the complexity of the user interface of the first application, and / or the resources allocated by the first application to other tasks).

[0319] The device sends (908, Fig. 9A ) identifies the corresponding touch position as the touch position information of the touch input of the representative touch position of the corresponding movement. Figure 1C In the example, the touch position information is sent from the touch processing module 220 to the application kernel 1 (230-1).

[0320] In some embodiments, the device determines (910) at a touch processing module independent of an application (e.g., using a touch processing module independent of an application) a timing for sending the one or more selected locations to a first software application; and sends the one or more selected locations to the first software application (e.g., as shown in FIG. 1 ) according to the determined timing. Figure 7M As shown, the timing requirement changes over time, and the device determines the timing of sending the one or more selected locations based on the changed timing requirement).

[0321] In some embodiments, the device monitors (912) the state of the first software application (e.g., by monitoring a run loop) (e.g., using a touch processing module independent of the application). The timing is determined based on the state of the first software application (e.g., if the first software application is busy, send early (because the first software application cannot respond quickly), if the first software application is not busy, send later (because the first software application can respond quickly).

[0322] In some embodiments, the device determines (922, Fig. 9C ) processing margin time. In some embodiments, the device selects a default processing margin time for all applications executed by the device. In some embodiments, corresponding applications are associated with corresponding processing margin times (e.g., applications with high computational intensity have long processing margin times, and applications with low computational intensity have short processing margin times). At each communication time in a series of communication times, each of which is ahead of a display update time in a series of display update times by at least the determined processing margin time, the device sends a set of positions including one or more selected positions of touch input during a previous time period from a touch processing module to a first software application; and at the first software application (e.g., using the first software application), the first user interface is updated in real time for display in this series of display update times based on the set of positions sent by the touch processing module in this series of communication times. For example, a set 724-5 ( Figure 7L ) is used to update the user interface at update time 730-3, and the set 724-6 of touches processed by the margin time ahead of update time 730-4 is used to update the user interface at update time 730-4.

[0323] In some embodiments, determining the processing margin time includes (924) setting the processing margin time to an initial value and then determining an updated processing margin time based on one or more measurements of the performance of the first software application (e.g., the processing margin time changes over time, such as Figure 7MIn some embodiments, the initial value is determined by determining the performance of the first software application when the first software application is launched. In some embodiments, the updated processing margin time is determined at predefined time intervals. In some embodiments, the updated processing margin time is responsive to detecting each individual touch input on the touch-sensitive surface.

[0324] In some embodiments, the processing margin time is determined based on the longest processing time of the first software application when processing each of the multiple groups of touch input positions (926) (for example, if the processing time fluctuates or varies, a worst-case or conservative margin time is taken). In some embodiments, the processing margin time is selected from multiple candidate margin times based on confidence values ​​associated with the corresponding candidate margin times.

[0325] In some embodiments, the locations of the touch input included in the set of locations sent at each communication time correspond to (928) multiple detected locations of the touch input between consecutive communication times in the series of communication times.

[0326] In some embodiments, the device sends (930) predicted touch position information (e.g., Figure 1D In some implementations, the predicted touch location information is sent simultaneously with the touch location information (e.g., Figure 1D The event object 194 in includes both the predicted touch location information and the detected touch location information. In some embodiments, an application-independent submodule of the first software application (e.g. Figure 1C The touch processing module 220 in the embodiment of the present invention identifies one or more predicted touch positions of the corresponding movement according to the predefined prediction model.

[0327] In some embodiments, the device sends (922) touch position information of the touch input to multiple software applications, including the first software application. Figure 1C , the touch location information is sent to both application 1 (136-1) and application 2 (136-2).

[0328] In some embodiments, the device sends (934) touch location information of the touch input to the first software application based on determining that the first software application is configured to receive the touch location information; and sends subsequent touch location information of the touch input to the second software application based on determining that the second software application different from the first software application is configured to receive subsequent touch location information. Figure 1CIn some embodiments, when the displayed user interface includes both the user interface of application 1 (136-1) and the user interface of application 2 (136-2), the touch position information of the touch corresponding to the user interface of application 1 (136-1) is sent to application 1 (136-1), and the touch position information of the touch corresponding to the user interface of application 2 (136-2) is sent to application 2 (136-2).

[0329] The device updates (914, Fig. 9A ) user interface. For example, in Figure 7Q to Figure 7V In some embodiments, the user interface is updated according to the changes in the touch input (and the corresponding touch information). Figure 1C The touch processing module 220 in the embodiment of the present invention) sends a touch signal to the first software application portion (e.g., Figure 1C The application kernel 1 (230-1) in the embodiment sends the position information of the previous representative touch before the next representative touch is detected to provide the first software application with time to process the previous representative touch.

[0330] In some embodiments, the device is provided with a first portion of a first software application (including a submodule independent of the application (e.g., a touch processing module independent of the application, such as Figure 1C The touch processing module 220 in the embodiment sends (936, Fig.9D ) touch location information to include application-specific submodules (e.g. Figure 1C In some embodiments, the touch processing module is an application-independent module that communicates with the application-independent submodule. In some embodiments, the second application executed by the electronic device also includes the application-independent submodule, and when executed, the application-independent submodule is used to receive touch position information from the touch processing module and update the corresponding user interface of the second application with the second part of the second software application including an application-specific submodule different from the application-specific submodule of the first software application.

[0331] In some embodiments, the movement of the touch input is detected in the corresponding touch detection frame (e.g. Figure 7JThe first application program's updated user interface based on the movement of the touch input is detected (938) during the corresponding touch processing frame (e.g., Figure 7J In some embodiments, the corresponding touch detection frame partially overlaps the corresponding touch processing frame. In some embodiments, the corresponding touch detection frame is simultaneous with the corresponding touch processing frame. The updated user interface is displayed on the display for a corresponding display frame (e.g., Figure 7J The duration of the display frame 734-2).

[0332] In some embodiments, during the corresponding touch processing frame (e.g., for the entire duration of the corresponding touch processing frame), the device displays (940) a user interface of the first application generated during a previous touch processing frame (e.g., during a previous display frame that occurred simultaneously with the corresponding touch processing frame). Figure 7J , during touch processing frame 732-3, the user interface generated during touch processing frame 732-2 is displayed.

[0333] In some embodiments, during the corresponding display frame (e.g., for the entire duration of the corresponding display frame), the device detects (942) subsequent movement of the touch input on the touch-sensitive surface and sends touch position information of the subsequent movement of the touch input to the first software application (e.g., during the next touch detection frame occurring simultaneously with the corresponding display frame). For example, during display frame 734-2, the device detects a touch in touch detection frame 724-2.

[0334] It should be understood that Figure 9A-9D The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein in conjunction with other methods described herein (e.g., methods 800, 1000, and 1100) are also applicable in a similar manner to the details of the processes described above in conjunction with 9A to 9D The method 900 described herein. For example, the touch information, representative touch positions, timing, and information transmission operations described above with reference to the method 900 may optionally have one or more of the features of the touch information, representative touch positions, timing, and information transmission operations described herein with reference to other methods described herein, such as methods 800, 1000, and 1100. For the sake of brevity, these details are not repeated here.

[0335] FIG. 10A to FIG. 10CA flowchart of a method 1000 for updating a user interface based on a predicted touch position according to some embodiments is shown. The method 1000 is performed on an electronic device (e.g., device 300, Figure 3 ; or a portable multifunction device 100, Figure 1A ) is performed on a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0336] As described below, method 1000 provides a way to update a user interface based on predicted touch locations. The method reduces the difference between the detected touch input and the displayed user interface, thereby reducing the cognitive burden on the user when interacting with the touch screen. In addition, this generates a more efficient human-computer interface. For battery-powered electronic devices, enabling users to interact with the user interface more accurately reduces errors and unnecessary corrections, thereby saving power and increasing the time between battery charges.

[0337] The device displays (1002) a user interface (eg, Figure 7Q drawing application user interface).

[0338] While displaying the user interface according to the first display rate, the device detects (1004) movement of a touch input, including detecting a touch input at a first set of sequential locations on the touch-sensitive surface. The first set of sequential locations includes a plurality of locations on the touch-sensitive surface. For example, Figure 7H , multiple touches (eg, touches 722-2 to 722-5) are detected between display update times 730-1 and 730-2.

[0339] In some embodiments, at a first detection rate that is higher than the first display rate (e.g., Figure 7H Detect (1006) movement of the touch input at a detection rate four times the display rate).

[0340] The device predicts (1008) a first set of one or more predicted locations on the touch-sensitive surface for the touch input based on the plurality of locations in the first set of sequential locations. Fig.7O, touches 722-3 to 722-6 are used to predict touch positions 722-7 to 722-13. In some embodiments, the first set of one or more predicted positions are predicted based on derivatives (e.g., first-order derivatives and / or second-order derivatives) of the plurality of positions in the first set of sequential positions. In some embodiments, the first set of one or more predicted positions are predicted based on a linear fit (e.g., Figure 7KK shown).

[0341] In some implementations, each of the one or more predicted locations is (1010) a predicted representative touch location (e.g., Fig.7O The predicted representative touch position 722-10 in FIG.

[0342] In some embodiments, the one or more predicted locations include (1012) one or more predicted interstitial touch locations of the touch input on the touch-sensitive surface (e.g., Fig.7O The predicted gap touch positions 722-7 to 722-9 in FIG.

[0343] In some embodiments, the one or more predicted locations of the touch input on the touch-sensitive surface are predicted based at least in part on a plurality of representative touch locations of the touch input on the touch-sensitive surface (1014, Fig. 10B )(See above reference FIG. 8A to FIG. 8B For example, the predicted position is based on Fig.7O Representative touch locations 722-2 and 722-6 in FIG. 1. In some embodiments, each display frame has a single representative touch location.

[0344] In some embodiments, the one or more predicted locations of the touch input on the touch-sensitive surface are predicted based on multiple representative touch locations of the touch input on the touch-sensitive surface and one or more gap locations of the touch input on the touch-sensitive surface (1016) (see above reference FIG. 8A to FIG. 8B For example, the predicted position is based on Fig.7O In some embodiments, each display frame has one or more interstitial touch locations that are different from the representative touch locations.

[0345] In some embodiments, the number of predicted locations in the first set of one or more predicted locations of the touch input is determined based on one or more confidence values ​​associated with the one or more predicted locations (1018). Figure 7JJAs shown, the confidence value is determined based on whether the predicted touch position subsequently matches the detected (actual) touch position. Thus, when the predicted touch position matches the detected touch position, the confidence value increases (and, in turn, the number of predicted positions increases). When the predicted touch position does not match the detected touch position (or the predicted touch position deviates from the detected touch position by more than a predefined error tolerance), the confidence value decreases (and, in turn, the number of predicted positions decreases). In some embodiments, the touch processing module (e.g. Figure 1C In some embodiments, the touch processing module 220 in the first set predicts the touch position. In some embodiments, the touch processing module determines a confidence value associated with each touch position. In some embodiments, only touch positions whose associated confidence values ​​meet the confidence value criteria (e.g., are above a predefined confidence value threshold) are included in the first set of one or more predicted positions.

[0346] In some embodiments, the number of predicted locations in the first set of one or more predicted locations of the touch input is determined based on one or more confidence values ​​associated with the plurality of locations in the first set of sequential locations (1020). Figure 7KK As shown, the confidence value is determined based on whether the detected touch position matches the prediction model. For example, when the prediction is based on a linear fit, the device determines whether the detected touch position fits onto the fitting line. The deviation of the detected touch position from the line reduces the confidence value (and thus, the number of predicted positions decreases). When the detected touch position fits the line well, the confidence value increases (and thus, the number of predicted positions increases). In some embodiments, the touch processing module determines confidence values ​​for the multiple positions in the first set of sequential positions. The number of predicted positions is determined based on the confidence values ​​of the multiple positions in the first set of sequential positions (for example, the higher the confidence value, the more predicted positions, and the lower the confidence value, the fewer predicted positions).

[0347] In some embodiments, the number of predicted positions in the first set of one or more predicted positions for a touch input is equal to the number of positions in the first set of sequential positions. In some embodiments, the number of predicted positions in the first set of one or more predicted positions for a touch input is greater than the number of positions in the first set of sequential positions. In some embodiments, the number of predicted positions in the first set of one or more predicted positions for a touch input is an integer multiple of the positions in the first set of sequential positions.

[0348] In some embodiments, the one or more confidence values ​​associated with the plurality of positions in the first set of sequential positions are based at least in part on errors in fitting the plurality of positions in the first set of sequential positions to predefined constraints (e.g., fitting the detected touch positions to Figure 7KK In some embodiments, the predefined constraint is a linear fit to the multiple positions in the first set of sequential positions. In some embodiments, the predefined constraint is a polynomial fit to the multiple positions in the first set of sequential positions. In some embodiments, the predefined constraint is a circular fit to the multiple positions in the first set of sequential positions. In some embodiments, the one or more confidence values ​​associated with the first set of sequential positions are based at least in part on historical errors (e.g., errors from fitting a previous set of sequential positions). In some embodiments, the one or more confidence values ​​associated with the multiple positions in the first set of sequential positions are based at least in part on historical confidence values ​​(e.g., confidence values ​​from a previous set of sequential positions). In some embodiments, the one or more confidence values ​​associated with the multiple positions in the first set of sequential positions are based at least in part on the orientation of the touch input.

[0349] In some embodiments, the one or more confidence values ​​associated with the plurality of positions in the first set of sequential positions are based at least in part on the speed of movement of the (1024) touch input. For example, a high-speed touch input has a higher confidence value (because the touch input is less likely to change direction suddenly) and a low-speed touch input has a lower confidence value. For example, in Figure 7LL In , the touch input velocity is low and the confidence value is also low. Therefore, the number of predicted positions is reduced to 2.

[0350] In some embodiments, the device includes (1026) one or more sensors for detecting the intensity of a touch input on a touch-sensitive surface. The device predicts the intensity of the touch input at multiple locations on the touch-sensitive surface. For example, Fig.7II The predicted intensity of touch input at multiple locations is shown. Figure 7NN A line drawn based on the predicted intensity of the touch input is shown. In some embodiments, the multiple positions on the touch-sensitive surface correspond to a subset of the first group of one or more predicted positions on the touch-sensitive surface. In some embodiments, the multiple positions on the touch-sensitive surface correspond to the first group of one or more predicted positions on the touch-sensitive surface. The device updates the user interface based on the predicted intensity of the touch input. In some embodiments, the intensity prediction is performed at a different rate than the position prediction (for example, because the intensity prediction includes additional input from the intensity sensor on the stylus or a measurement of the contact size that takes additional processing time), and the position estimate for one or more touch inputs is updated before the intensity estimates for those touch inputs are updated. In this way, the predicted position of the touch can be updated in one frame, while the predicted intensity of the touch is updated in the next frame. In addition, in some embodiments, the predicted intensity of the touch is updated in multiple subsequent frames because the predicted value of the intensity of the touch becomes more and more accurate.

[0351] In some embodiments, the device predicts (1028) the tilt and / or orientation of the touch input at multiple locations on the touch-sensitive surface. For example, Fig.7II The tilt and / or orientation of a touch input (e.g., a stylus) is shown predicted at multiple locations. In some embodiments, the multiple locations on the touch-sensitive surface correspond to a subset of a first group of one or more predicted locations on the touch-sensitive surface. In some embodiments, the multiple locations on the touch-sensitive surface correspond to a first group of one or more predicted locations on the touch-sensitive surface. The device updates the user interface based on the predicted tilt and / or orientation of the touch input. In some embodiments, the tilt and / or orientation prediction is performed at a different rate than the position prediction (e.g., because the tilt and / or orientation prediction includes additional input from a force sensor on the stylus or measurement of the contact shape that takes additional processing time), and the position estimate for one or more touch inputs is updated before the tilt and / or orientation estimates for those touch inputs are updated. In addition, in some embodiments, the predicted tilt and / or orientation of the touch is updated in multiple subsequent frames because the predicted values ​​of the tilt and / or orientation of the touch become more and more accurate.

[0352] The device updates (1030, Fig. 10C ) user interface. For example, Figure 7Q to Figure 7V As shown, the user interface is updated to display an extension of the line based on the predicted touch location.

[0353] In some embodiments, the movement of the touch input is detected in the corresponding touch detection frame (e.g. Figure 7J The user interface is detected (1032) during a touch detection frame 724-1 in the display. An updated user interface based on the movement of the touch input is generated during a corresponding touch processing frame (e.g., touch processing frame 732-2). In some embodiments, the corresponding touch detection frame partially overlaps the corresponding touch processing frame (e.g., touch processing frame 732 overlaps touch detection frame 724-1). In some embodiments, the corresponding touch detection frame is simultaneous with the corresponding touch processing frame (e.g., touch detection frame 724-1 is simultaneous with touch processing frame 732). The updated user interface is displayed on the display for the duration of the corresponding display frame (e.g., display frame 734-2) that appears after the corresponding touch processing frame.

[0354] In some embodiments, the multiple locations do not have to be detected during the same display frame. The updated user interface can be at the next frame time after the representative locations of the touch input in the first group are detected, or it can be at a later time frame, depending on the processing time required (e.g., a user interface updated based on a touch detected during touch processing frame 732-2 is displayed during display frame 734-3 or 734-4).

[0355] In some embodiments, the device predicts (1034) a first set of one or more predicted locations on the touch-sensitive surface for the touch input at (e.g., using) an application-independent touch processing module; and sends the first set of one or more predicted locations of the touch input on the touch-sensitive surface to an application-specific portion of the first software application. Figure 1C In some embodiments, the predicted touch positions are sent from the touch processing module 220 to the application kernel 1 (230-1). In some embodiments, the first set of one or more predicted positions of the touch input on the touch-sensitive surface are sent to multiple software applications. In some embodiments, the first set of one or more predicted positions of the touch input on the touch-sensitive surface are published, and the published first set of one or more predicted positions of the touch input are retrieved by one or more software applications. In some embodiments, the first set of one or more predicted positions of the touch input on the touch-sensitive surface are sent from the touch processing module to the first software application in response to a request from the first software application for the first set of one or more predicted positions of the touch input on the touch-sensitive surface. The device updates the user interface at the first software application (e.g., using the first software application) according to the first set of one or more predicted positions of the touch input on the touch-sensitive surface.

[0356] In some embodiments, the device detects (1036) touch inputs at a second set of sequential locations on the touch-sensitive surface after detecting touch inputs at a first set of sequential locations on the touch-sensitive surface; and compares the second set of sequential locations of the touch inputs on the touch-sensitive surface with the first set of one or more predicted locations of the touch inputs on the touch-sensitive surface. Figure 7CC to Figure 7EEAs shown, if the difference between the first group of one or more predicted positions and the detected position exceeds a predefined standard, the user interface is updated to remove the user interface object displayed based on the wrong prediction. In some embodiments, each touch position in the first group of one or more predicted positions of the touch input on the touch-sensitive surface corresponds to a corresponding touch position in the second group of sequential positions of the touch input on the touch-sensitive surface. In some embodiments, each touch position in the first group of one or more predicted positions of the touch input on the touch-sensitive surface is associated with a corresponding timestamp and is compared with a corresponding touch position associated with a corresponding timestamp in a second group of sequential positions of the touch input on the touch-sensitive surface. For example, the predicted position corresponds to a specific time, such as a display update time, which provides an illusion of processing the touch input immediately. In some embodiments, each touch position in the first group of one or more predicted positions of the touch input on the touch-sensitive surface is associated with a corresponding identifier and is compared with a corresponding touch position associated with a corresponding identifier in a second group of sequential positions of the touch input on the touch-sensitive surface. In some embodiments, the first software application compares the second group of sequential positions of the touch input on the touch-sensitive surface with the first group of one or more predicted positions of the touch input on the touch-sensitive surface. Based on determining that a difference between the first set of one or more predicted locations of the touch input on the touch-sensitive surface and the second set of sequential locations of the touch input on the touch-sensitive surface satisfies a predefined criterion, the device updates the user interface based on the second set of sequential locations of the touch input on the touch-sensitive surface.

[0357] In some embodiments, the device predicts (1038) a second set of one or more locations on the touch-sensitive surface for the touch input (e.g., based on the second set of sequential positions and / or multiple positions in the first set of sequential positions), and updates the user interface based on the second set of sequential positions on the touch-sensitive surface and the second set of one or more predicted locations of the touch input on the touch-sensitive surface. Figure 7Z to Figure 7AA In , both the detected touch location and the predicted touch location are used to update the user interface.

[0358] In some embodiments, the portion of the user interface updated (1040) based on one or more predicted locations is visually distinguished from the portion of the user interface updated based on one or more detected locations (e.g., a path based on the predicted locations is drawn with a dashed line, and a path based on the measured locations is drawn with a solid line, such as Figures 7W to 7BB shown).

[0359] It should be understood that FIG. 10A to FIG. 10CThe specific order in which the operations in the description are described is merely exemplary and is not intended to represent that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein in conjunction with other methods described herein (e.g., methods 800, 900, and 1100) are also applicable in a similar manner to the details of the processes described above in conjunction with FIG. 10A to FIG. 10C The method 1000 described herein. For example, the application programming interface, predicted touch positions, representative touch positions, and interstitial touch positions described above with reference to the method 1000 optionally have one or more of the features of the application programming interface, predicted touch positions, representative touch positions, and interstitial touch positions described herein with reference to other methods described herein (e.g., methods 800, 900, and 1100). For the sake of brevity, these details are not repeated here.

[0360] Fig.11 100 is a flowchart of a method 1100 for transmitting predicted touch information according to some embodiments. The method 1100 is performed in an electronic device (e.g., device 300, Figure 3 ; or a portable multifunction device 100, Figure 1A ) is performed on a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors for detecting the intensity of contact with the touch-sensitive surface. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 1100 are optionally combined, and / or the order of some operations is optionally changed.

[0361] As described below, method 1100 provides a way to transmit predicted touch information. The use of predicted touch information reduces the difference between the detected touch input and the displayed user interface, thereby reducing the cognitive burden on the user when interacting with the touch screen. In addition, this generates a more efficient human-computer interface. For battery-powered electronic devices, enabling users to interact with the user interface more accurately reduces errors and unnecessary corrections, thereby saving power and increasing the time between battery charges.

[0362] The device displays (1102) a user interface of a first software application (eg, Figure 7Q drawing application user interface).

[0363] The device detects the signal at a first detection rate (e.g., at a higher detection rate than the first display rate) Figure 7H , the detection rate is higher than the display rate) detecting (1004) corresponding movement of the touch input on the touch-sensitive surface.

[0364] The device processes the touch at an application-independent touch processing module (e.g., using an application-independent touch processing module such as Figure 1C The touch processing module 220 in the first software application sends (1106) touch location information of the touch input to the application-specific portion of the first software application, the touch location information identifying: one or more predicted locations of the touch input on the touch-sensitive surface (e.g., location 250 in predicted touch 244, Figure 1D ); and one or more predicted intensity values ​​of the touch input at one or more intensity locations of the touch input on the touch-sensitive surface (e.g., the intensities in predicted touch 244, Figure 1D ), the one or more intensity positions include at least a subset of the one or more predicted positions. In some embodiments, Figure 1D The predicted position 250 of touch 244 includes one or more predicted positions of the touch input and / or one or more predicted intensity positions (e.g., one or more positions for which the intensity is predicted). In some embodiments, the corresponding intensity value corresponds to the force or pressure applied by the touch input on the touch-sensitive surface. In some embodiments, the one or more intensity positions are the one or more predicted positions on the touch-sensitive surface. In some embodiments, the one or more intensity positions are different from the one or more predicted positions on the touch-sensitive surface.

[0365] The device processes (1122) the touch location information at (eg, using) the first software application (eg, Figure 1C Application 1 (136-1) in is used to process the touch position information). In some embodiments, processing the touch position information includes updating the user interface based on the touch position information.

[0366] In some embodiments, the touch location information includes (1108) a corresponding touch identifier (e.g., a number or string identifying the touch associated with each predicted location) for each of the one or more predicted locations of the touch input on the touch-sensitive surface (e.g., Figure 1D 246 in the predicted touch 244).

[0367] In some embodiments, the touch location information also identifies (1110): a plurality of detected locations of the touch input on the touch-sensitive surface (e.g., Figure 1D and a plurality of intensity values ​​of the touch input at a plurality of intensity locations on the touch-sensitive surface (e.g., Figure 1D intensity 252 in detected touch 242).

[0368] In some embodiments, the plurality of intensity locations on the touch-sensitive surface are (1112) the plurality of detected locations (e.g., one or more intensity sensors detect intensities at the same locations corresponding to locations detected by the touch-sensitive surface). In some embodiments, the plurality of intensity locations are different from the plurality of detected locations. In some embodiments, the plurality of intensity locations are a subset of the plurality of detected locations. In some embodiments, the plurality of detected locations are a subset of the plurality of intensity locations.

[0369] In some embodiments, the touch location information includes (1114) one or more touch identifiers (e.g., Figure 1D In some embodiments, the one or more identifiers of the multiple detected locations do not include corresponding identifiers of the one or more predicted locations in the touch location information.

[0370] In some embodiments, the touch position information also identifies (1116) a predicted tilt and / or orientation of the touch input (e.g., Figure 1D 244 in predicted touch 244).

[0371] In some embodiments, the touch location information also identifies (1118) the type of touch input (e.g., whether the touch input was performed by a finger or a stylus, such as Figure 1D (shown in touch type 258).

[0372] In some embodiments, sending touch position information of the touch input to the application-specific portion of the first software application includes (1120) publishing the touch position information of the touch input. For example, Figure 1D The event object 194 shown is published on Figure 1C In queue 218-1.

[0373] It should be understood that Fig.11 The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein in conjunction with other methods described herein (e.g., methods 800, 900, and 1000) are also applicable in a similar manner to the details of the processes described above in conjunction with Fig.11The method 1100 described herein. For example, the touch information, predicted touch position, predicted intensity, and information sending operations described above with reference to the method 1100 may optionally have one or more of the features of the touch information, predicted touch position, predicted intensity, and information sending operations described herein with reference to other methods described herein (e.g., methods 800, 900, and 1000). For the sake of brevity, these details are not repeated here.

[0374] According to some embodiments, Fig.12 1 shows a functional block diagram of an electronic device 1200 configured according to the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof that implements the principles of various described embodiments. It should be understood by those skilled in the art that Fig.12 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0375] like Fig.12 As shown in , the electronic device 1200 includes a display unit 1202 configured to display a user interface; a touch-sensitive surface unit 1204 configured to receive contact; and a processing unit 1208 coupled to the display unit 1202 and the touch-sensitive surface unit 1204. In some embodiments, the electronic device includes one or more sensor units 1206 configured to detect input, and the processing unit 1208 is also coupled to the one or more sensor units 1206. In some embodiments, the one or more sensor units 1206 include one or more sensors configured to detect the intensity of contact with the touch-sensitive surface unit 1204. In some embodiments, the one or more sensor units 1206 include one or more sensors configured to detect signals from a stylus associated with the electronic device 1200. In some embodiments, the processing unit 1208 includes a display enabling unit 1210, a detection unit 1212, an updating unit 1214, a sending unit 1216, and / or a selecting unit 1218.

[0376] The processing unit 1208 is configured (eg, with the display enabling unit 1210 ) to enable display of the user interface at a first display rate.

[0377] The processing unit 1208 is also configured to detect movement of the touch input at a series of locations on the touch-sensitive surface unit 1204 at a first detection rate that is higher than the first display rate (eg, with the detection unit 1212 ) while the user interface is displayed.

[0378] Processing unit 1208 is also configured to update the user interface from a corresponding current state to a corresponding next state (e.g., with update unit 1214) at each update time in a series of update times based on a selected subset of the series of positions of the touch input, each selected subset of the series of positions including multiple positions of the touch input.

[0379] In some embodiments, processing unit 1208 is configured to send a message having information including a selected subset of the series of locations to the first software application (e.g., with sending unit 1216), and the first software application updates the user interface based on the information in the message.

[0380] In some embodiments, the message also includes information identifying one or more of: the intensity of the touch input at a selected subset of the series of locations; and the type of touch input detected at the selected subset of the series of locations.

[0381] In some embodiments, the last detected position in each selected subset is detected at least a predefined time interval before the next update time.

[0382] In some embodiments, processing unit 1208 is configured (e.g., with selection unit 1218) to select a corresponding touch position from a selected subset of the series of positions of the touch input as a representative touch position.

[0383] In some embodiments, the corresponding touch location is selected as the representative touch location based on a touch processing criterion of the first application that indicates an amount of time required for the first application to update the user interface.

[0384] In some embodiments, updating the user interface by the first application includes transmitting a selected subset of the series of locations of the touch input to the first application along with an indication of which location is a representative touch location.

[0385] In some embodiments, the selected subset of the series of positions includes one or more gap positions corresponding to touch positions between a previous representative touch position and the representative touch position.

[0386] In some embodiments, processing unit 1208 is configured (for example, with selection unit 1218) to select multiple locations of touch input for updating the user interface for each update time in the series of update times, where the selected locations are locations of touch input detected after the last time a location was selected for updating the user interface.

[0387] In some embodiments, the selected multiple locations of touch input include one or more predicted gap locations.

[0388] In some embodiments, processing unit 1208 is configured (for example, with selection unit 1218) to select multiple locations of touch input for updating the user interface for each update time in the series of update times, where the selected locations are locations of touch input detected after the location of the touch input that was last selected for updating the user interface is detected.

[0389] In some embodiments, the selected locations include all locations of touch input after the location of the touch input that was last selected for updating the user interface was detected.

[0390] In some embodiments, the selected location includes only one of the locations of touch input after the location of touch input that was last selected for updating the user interface was detected.

[0391] In some embodiments, processing unit 1208 is configured to (for example, with updating unit 1214) update the user interface from a first state to a second state based on a first subset of the series of positions of the touch input; and after updating the user interface from the first state to the second state, update the user interface from the second state to a third state based on a second subset of the series of positions of the touch input.

[0392] According to some embodiments, Fig.13 1 shows a functional block diagram of an electronic device 1300 configured according to the principles of various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof that implements the principles of various described embodiments. It should be understood by those skilled in the art that Fig.13 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0393] like Fig.13As shown in , the electronic device 1300 includes a display unit 1302 configured to display a user interface; a touch-sensitive surface unit 1304 configured to receive contact; and a processing unit 1308 coupled to the display unit 1302 and the touch-sensitive surface unit 1304. In some embodiments, the electronic device includes one or more sensor units 1306 configured to detect input, and the processing unit 1308 is also coupled to the one or more sensor units 1306. In some embodiments, the one or more sensor units 1306 include one or more sensors configured to detect the intensity of contact with the touch-sensitive surface unit 1304. In some embodiments, the one or more sensor units 1306 include one or more sensors configured to detect signals from a stylus associated with the electronic device 1300. In some embodiments, the processing unit 1308 includes: a display enabling unit 1310, a detection unit 1312, a selection unit 1314, a sending unit 1316, an updating unit 1318, a discarding unit 1320, a determining unit 1322, a monitoring unit 1324 and / or a setting unit 1326.

[0394] Processing unit 1308 is configured to enable (eg, with display enabling unit 1310 ) display of a user interface of a first software application that is updated at a first display rate.

[0395] The processing unit 1308 is also configured to, when a first frame of the user interface according to the first display rate is displayed: (for example, with the detection unit 1312) detect a corresponding movement of the touch input on the touch-sensitive surface unit 1304; and at a touch processing module independent of the application: (for example, with the selection unit 1314) select, based on the touch processing criteria of the first software application, a corresponding touch position of the touch input detected during the corresponding movement to be identified as a representative touch position of the corresponding movement; and (for example, with the sending unit 1316) send touch position information of the touch input to an application-specific portion of the first software application that is different from the touch processing module, the touch position information identifying the corresponding touch position as a representative touch position of the corresponding movement.

[0396] Processing unit 1308 is also configured to update a user interface at the first software application based on the touch location information (eg, with updating unit 1318 ).

[0397] In some embodiments, processing unit 1308 is configured to send touch location information (e.g., with sending unit 1316) with a first portion of a first software application (including application-independent sub-modules) and to update a user interface (e.g., with updating unit 1318) with a second portion of the software application including application-specific sub-modules.

[0398] In some embodiments, movement of the touch input is detected during a corresponding touch detection frame; an updated user interface of the first application based on the movement of the touch input is generated during a corresponding touch processing frame; and the updated user interface is displayed on the display for the duration of a corresponding display frame that occurs after the corresponding touch processing frame.

[0399] In some embodiments, processing unit 1308 is configured (e.g., with display enabling unit 1310) to enable display of a user interface of a first application generated during a previous touch processing frame during a corresponding touch processing frame.

[0400] In some embodiments, the processing unit 1308 is configured (e.g., with the detection unit 1312) to detect subsequent movement of the touch input on the touch-sensitive surface unit 1304 during the corresponding display frame and send touch position information of the subsequent movement of the touch input to the first software application.

[0401] In some embodiments, selecting a corresponding touch position as a representative touch position includes: detecting a first touch position of a touch input during a touch detection frame, and in response to detecting the first position: selecting the first touch position as a representative touch position of the corresponding movement of the touch input based on a determination that the first touch position satisfies touch processing criteria of a first application; and not selecting the first touch position as a representative touch position of the corresponding movement of the touch input based on a determination that the first touch position does not satisfy touch processing criteria of the first application.

[0402] In some embodiments, selecting a corresponding touch position as a representative touch position includes: detecting a second touch position of the touch input during a touch detection frame, and in response to detecting the second position, selecting the second touch position as a representative touch position of the corresponding movement of the touch input based on determining that the second touch position meets touch processing criteria of the first application.

[0403] In some embodiments, processing unit 1308 is configured to abandon selecting the second touch position as a representative touch position for a corresponding movement of the touch input (e.g., by abandoning unit 1320) based on determining that the second touch position does not meet touch processing criteria of the first application.

[0404] In some embodiments, the processing unit 1308 is configured to, at a touch processing module independent of the application: determine (e.g., with a determining unit 1322) a timing for sending the one or more selected positions to a first software application; and send (e.g., with a sending unit 1316) the one or more selected positions to the first software application according to the determined timing.

[0405] In some embodiments, processing unit 1308 is configured to monitor a state of the first software application (eg, with monitoring unit 1324), wherein the timing is determined based on the state of the first software application.

[0406] In some embodiments, the processing unit 1308 is configured to: determine a processing margin time (e.g., using a determining unit 1322); send a set of positions including one or more selected positions of touch input during a previous time period from the touch processing module to the first software application (e.g., using a sending unit 1316) for each communication time in a series of communication times, each of which is ahead of a display update time in a series of display update times by at least the determined processing margin time; and at the first software application, update the first user interface in real time (e.g., using an updating unit 1318) based on the set of positions sent by the touch processing module in the series of communication times for display in the series of display update times.

[0407] In some embodiments, determining the processing margin time includes setting the processing margin time to an initial value (eg, with setting unit 1326), and then determining an updated processing margin time based on one or more measurements of the performance of the first software application.

[0408] In some embodiments, the processing margin time is determined based on a longest processing time of the first software application when processing each of the plurality of sets of touch input locations.

[0409] In some embodiments, the locations of the touch input included in the set of locations sent at each communication time correspond to multiple detected locations of the touch input between consecutive communication times in the series of communication times.

[0410] In some embodiments, processing unit 1308 is configured to send predicted touch position information of a touch input identifying one or more predicted touch positions of a corresponding movement to a first software application (e.g., with sending unit 1316).

[0411] In some embodiments, processing unit 1308 is configured to send touch position information of the touch input to multiple software applications, including the first software application (e.g., with sending unit 1316).

[0412] In some embodiments, processing unit 1308 is configured to: send touch position information of a touch input to a first software application based on determining that the first software application is configured to receive touch position information (e.g., with sending unit 1316); and send subsequent touch position information of the touch input to a second software application based on determining that a second software application different from the first software application is configured to receive subsequent touch position information (e.g., with sending unit 1316).

[0413] According to some embodiments, Fig.14 1400 is a functional block diagram of an electronic device 1400 configured according to the principles of the various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig.14 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0414] like Fig.14 As shown in , the electronic device 1400 includes a display unit 1402 configured to display a user interface; a touch-sensitive surface unit 1404 configured to receive contact; and a processing unit 1408 coupled to the display unit 1402 and the touch-sensitive surface unit 1404. In some embodiments, the electronic device includes one or more sensor units 1406 configured to detect input, and the processing unit 1408 is also coupled to the one or more sensor units 1406. In some embodiments, the one or more sensor units 1406 include one or more sensors configured to detect the intensity of contact with the touch-sensitive surface unit 1404. In some embodiments, the one or more sensor units 1406 include one or more sensors configured to detect signals from a stylus associated with the electronic device 1400. In some embodiments, the processing unit 1408 includes a display enabling unit 1410, a detection unit 1412, a prediction unit 1414, an update unit 1416, a sending unit 1418, and / or a comparison unit 1420.

[0415] The processing unit 1408 is configured (eg, with the display enabling unit 1410 ) to enable display of the user interface at a first display rate.

[0416] The processing unit 1408 is also configured to, when displaying a user interface according to a first display rate: (e.g., with a detection unit 1412) detect movement of a touch input, including detecting the touch input at a first set of sequential positions on the touch-sensitive surface unit, wherein the first set of sequential positions includes multiple positions on the touch-sensitive surface unit; and (e.g., with a prediction unit 1412) predict a first set of one or more predicted positions on the touch-sensitive surface unit for the touch input based on multiple positions in the first set of sequential positions.

[0417] Processing unit 1408 is also configured to update the user interface (eg, with updating unit 1416 ) based on the first set of one or more predicted locations of the touch input on the touch-sensitive surface unit.

[0418] In some embodiments, movement of the touch input is detected at a first detection rate that is higher than a first display rate.

[0419] In some embodiments, processing unit 1408 is configured to, at an application-independent touch processing module: predict a first set of one or more predicted locations on touch-sensitive surface unit 1404 for a touch input (e.g., with prediction unit 1414); and send the first set of one or more predicted locations of the touch input on touch-sensitive surface unit 1404 to an application-specific portion of a first software application (e.g., with sending unit 1418). Processing unit 1408 is also configured to update a user interface at the first software application based on the first set of one or more predicted locations of the touch input on touch-sensitive surface unit 1404 (e.g., with updating unit 1416).

[0420] In some embodiments, after detecting touch input at a first set of sequential positions on the touch-sensitive surface unit 1404: (e.g., with detection unit 1412) detect touch input at a second set of sequential positions on the touch-sensitive surface unit 1404; (e.g., with comparison unit 1420) compare the second set of sequential positions of the touch input on the touch-sensitive surface unit 1404 with the first set of one or more predicted positions of the touch input on the touch-sensitive surface unit 1404; and based on determining that the difference between the first set of one or more predicted positions of the touch input on the touch-sensitive surface unit 1404 and the second set of sequential positions of the touch input on the touch-sensitive surface unit 1404 meets a predefined criterion, update the user interface based on the second set of sequential positions of the touch input on the touch-sensitive surface unit 1404 (e.g., with update unit 1416).

[0421] In some embodiments, the processing unit 1408 is configured to: predict a second set of one or more positions on the touch-sensitive surface unit 1404 for touch input (e.g., with a prediction unit 1414); and update the user interface based on the second set of sequential positions on the touch-sensitive surface unit 1404 and the second set of one or more predicted positions of the touch input on the touch-sensitive surface unit 1404 (e.g., with an update unit 1416).

[0422] In some embodiments, the portion of the user interface that is updated based on one or more predicted locations is visually distinguished from the portion of the user interface that is updated based on one or more detected locations.

[0423] In some embodiments, movement of touch input is detected during a corresponding touch detection frame; an updated user interface based on the movement of the touch input is generated during a corresponding touch processing frame; and the updated user interface is displayed on the display for the duration of a corresponding display frame that occurs after the corresponding touch processing frame.

[0424] In some embodiments, the one or more predicted locations of the touch input on the touch-sensitive surface unit 1404 are predicted based at least in part on multiple representative touch locations of the touch input on the touch-sensitive surface unit 1404 .

[0425] In some embodiments, the one or more predicted positions of the touch input on the touch-sensitive surface unit 1404 are predicted based on multiple representative touch positions of the touch input on the touch-sensitive surface unit 1404 and one or more gap positions of the touch input on the touch-sensitive surface unit 1404 .

[0426] In some implementations, each of the one or more predicted locations is a predicted representative touch location.

[0427] In some embodiments, the one or more predicted locations include one or more predicted interstitial touch locations of a touch input on the touch-sensitive surface unit 1404 .

[0428] In some embodiments, the number of predicted locations in a first set of one or more predicted locations of a touch input is determined based on one or more confidence values ​​associated with the one or more predicted locations.

[0429] In some embodiments, the number of predicted positions in a first set of one or more predicted positions of a touch input is determined based on one or more confidence values ​​associated with the multiple positions in the first set of sequential positions.

[0430] In some embodiments, the one or more confidence values ​​associated with the plurality of positions in the first set of sequential positions are based at least in part on errors in fitting the plurality of positions in the first set of sequential positions to predefined constraints.

[0431] In some embodiments, the one or more confidence values ​​associated with the multiple positions in the first set of sequential positions are based at least in part on the speed of movement of the touch input.

[0432] In some embodiments, the electronic device includes one or more sensors for detecting the intensity of a touch input on the touch-sensitive surface unit 1404. The processing unit 1408 is configured to: predict the intensity of the touch input at multiple locations on the touch-sensitive surface unit 1404 (e.g., with the prediction unit 1414); and update the user interface according to the predicted intensity of the touch input (e.g., with the update unit 1416).

[0433] In some embodiments, the processing unit 1408 is configured to: predict the tilt and / or orientation of the touch input at multiple locations on the touch-sensitive surface unit 1404 (e.g., with a prediction unit 1414); and update the user interface based on the predicted tilt and / or orientation of the touch input (e.g., with an update unit 1416).

[0434] According to some embodiments, Fig.15 1 shows a functional block diagram of an electronic device 1500 configured according to the principles of the various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig.15 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0435] like Fig.15As shown in , the electronic device 1500 includes a display unit 1502 configured to display a user interface; a touch-sensitive surface unit 1504 configured to receive contact; and a processing unit 1510 coupled to the display unit 1502 and the touch-sensitive surface unit 1504. In some embodiments, the electronic device includes one or more sensor units 1506 configured to detect input, and the processing unit 1510 is also coupled to the one or more sensor units 1506. In some embodiments, the one or more sensor units 1506 include one or more sensors configured to detect the intensity of contact with the touch-sensitive surface unit 1504. In some embodiments, the one or more sensor units 1506 include one or more sensors configured to detect signals from a stylus associated with the electronic device 1500. In some embodiments, the processing unit 1508 includes a display enabling unit 1510, a detection unit 1512, a sending unit 1514, a processing unit 1516, and / or a publishing unit 1518.

[0436] The processing unit 1508 is configured to enable display of a user interface of a first software application that is updated at a first display rate (e.g., with the display enabling unit 1510); and to detect corresponding movement of touch input on the touch-sensitive surface unit 1504 at a first detection rate that is higher than the first display rate (e.g., with the detection unit 1512).

[0437] The processing unit 1508 is also configured to send touch position information of the touch input to an application-specific portion of the first software application (e.g., with a sending unit 1514) at a touch processing module independent of the application, the touch position information identifying: one or more predicted positions of the touch input on the touch-sensitive surface unit 1504; and one or more predicted intensity values ​​of the touch input at one or more intensity positions of the touch input on the touch-sensitive surface unit 1504, the one or more intensity positions including at least a subset of the one or more predicted positions.

[0438] Processing unit 1508 is also configured to process the touch location information at the first software application (eg, with processing unit 1516).

[0439] In some embodiments, the touch location information includes a corresponding touch identifier for each of the one or more predicted locations of the touch input on the touch-sensitive surface unit 1504.

[0440] In some embodiments, the touch position information also identifies: multiple detected positions of the touch input on the touch-sensitive surface unit 1504; and multiple intensity values ​​of the touch input at multiple intensity positions on the touch-sensitive surface unit 1504.

[0441] In some embodiments, the multiple intensity positions on the touch-sensitive surface unit 1504 are the multiple detected positions.

[0442] In some embodiments, the touch location information includes one or more touch identifiers for the multiple detected locations.

[0443] In some embodiments, the touch location information also identifies the predicted tilt and / or orientation of the touch input.

[0444] In some embodiments, the touch location information also identifies the type of touch input.

[0445] In some embodiments, sending touch position information of the touch input to the application-specific portion of the first software application includes publishing (e.g., with publishing unit 1518) the touch position information of the touch input.

[0446] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (for example, as described above with respect to Figure 1A and Figure 3 described) or application-specific chips.

[0447] References FIG. 8A to FIG. 8B , 9A to 9D , FIG. 10A to FIG. 10C and Fig.11 The operation is optionally performed by Figure 1A to Figure 1B or Figures 12 to 15 . For example, the detection operation 804, the message sending operation 808, and the update operation 814 are optionally implemented by the event classifier 170, the event identifier 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects the contact on the touch-sensitive display 112, and the event distributor module 174 delivers the event information to the application 146-1. The corresponding event identifier 180 of the application 146-1 compares the event information with the corresponding event definition 186, and determines whether the first contact at the first position on the touch-sensitive surface (or whether the rotation of the device) corresponds to a predefined event or sub-event, such as the selection of an object on the user interface, or the rotation of the device from one orientation to another orientation. When the corresponding predefined event or sub-event is detected, the event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally uses or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know that based on Figure 1A to Figure 1B How the components shown in FIG. 1 can implement other processes.

[0448] For the purpose of explanation, the foregoing description is described with reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive, nor is it intended to limit the invention to the precise form disclosed. Many modifications and variations are possible based on the above teachings. The embodiments are selected and described in order to best illustrate the principles of the invention and its practical application, so that other persons skilled in the art can best use the invention and the various described embodiments with various modifications suitable for the specific purposes contemplated.

Claims

1. A method for detecting a touch input, comprising: At electronic devices with touch-sensitive surfaces and displays: displaying a user interface of a first software application that updates at a first display rate; While displaying one or more frames of the user interface according to the first display rate: detecting the touch input on the touch-sensitive surface, the detecting comprising detecting a plurality of touch locations of the touch input on the touch-sensitive surface during respective time periods having the same duration as each frame of the user interface, wherein the plurality of touch locations detected during the respective time periods correspond to respective movements of the touch input on the touch-sensitive surface; In the application-independent touch processing module: selecting, based on touch handling criteria of the first software application, a single corresponding touch location of the touch input from the plurality of touch locations detected during a corresponding time period as a representative touch location for the corresponding time period; as well as sending touch location information of the touch input identifying the representative touch location for the corresponding time period to an application-specific portion of the first software application distinct from the touch processing module; as well as, At the first software application, the user interface is updated by generating a display frame based on the touch location information.

2. The method according to claim 1, comprising: The touch location information is sent with a first portion of the first software application including an application-independent submodule, and the user interface is updated with a second portion of the first software application including an application-specific submodule.

3. The method according to claim 1, wherein: The display frame of the updated user interface of the first software application is generated during a corresponding touch processing frame; and After the corresponding touch processing frame, the updated display frame of the user interface is displayed on the display.

4. The method according to claim 3, comprising: During the corresponding touch processing frame, a display frame of the user interface of the first software application generated during a previous touch processing frame is displayed.

5. The method according to claim 3, comprising: While displaying a corresponding display frame of the user interface of the first software application, detecting subsequent movement of the touch input on the touch-sensitive surface and sending touch position information of the subsequent movement of the touch input to the first software application.

6. The method of claim 1 , wherein detecting the plurality of touch locations comprises: detecting a first touch position of the touch input during the corresponding time period earlier than the corresponding display update time, and in response to detecting the first touch position: selecting the first touch location as the representative touch location for the corresponding movement of the touch input based on determining that the first touch location satisfies touch processing criteria of the first software application; as well as Based on determining that the first touch location does not satisfy the touch processing criteria of the first software application, the first touch location is not selected as the representative touch location for the corresponding movement of the touch input.

7. The method according to claim 6, wherein selecting the corresponding touch position as the representative touch position further comprises: detecting a second touch position among the plurality of touch positions of the touch input during the corresponding time period, and in response to detecting the second touch position: Based on determining that the second touch location satisfies the touch processing criteria of the first software application, the second touch location is selected as the representative touch location for the corresponding movement of the touch input.

8. The method according to claim 7, comprising: Based on determining that the second touch location does not satisfy touch processing criteria of the first software application, the second touch location is not selected as the representative touch location for the corresponding movement of the touch input.

9. The method according to any one of claims 1 to 5, comprising: At the application-independent touch processing module: determining a timing for sending the one or more selected locations to the first software application; as well as The one or more selected locations are sent to the first software application according to the determined timing.

10. The method according to claim 9, comprising: A state of the first software application is monitored, wherein the timing is determined based on the state of the first software application.

11. The method according to any one of claims 1 to 5, comprising: Determine processing margin time; at each communication time in a series of communication times, wherein each communication time is ahead of a display update time in a series of display update times by at least the determined processing margin time, sending from the touch processing module to the first software application a set of locations including one or more selected locations of the touch input during a previous time period; and At the first software application, the user interface is updated in real time based on the set of locations sent by the touch processing module at the series of communication times for display at the series of display update times.

12. The method according to claim 11, comprising: Determining the processing margin time includes setting the processing margin time to an initial value and then determining an updated processing margin time based on one or more measurements of performance of the first software application.

13. The method of claim 11, wherein the processing margin time is determined based on a longest processing time of the first software application in processing each of a plurality of sets of touch input locations.

14. The method of claim 11, wherein the location of the touch input included in the set of locations transmitted at each communication time corresponds to a plurality of detected locations of the touch input between consecutive communication times in the series of communication times.

15. The method according to claim 14, further comprising: Predicted touch location information for the touch input identifying one or more predicted touch locations of the corresponding movement is sent to the first software application.

16. The method according to any one of claims 1 to 5, further comprising: The touch position information of the touch input is sent to a plurality of software applications, including the first software application.

17. The method according to any one of claims 1 to 5, comprising: sending the touch position information of the touch input to the first software application based on determining that the first software application is configured to receive the touch position information; as well as Subsequent touch position information of the touch input is sent to a second software application different from the first software application based on determining that the second software application is configured to receive subsequent touch position information.

18. An electronic device comprising: monitor; Touch-sensitive surfaces; one or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for: displaying a user interface of a first software application that updates at a first display rate; While displaying one or more frames of the user interface according to the first display rate: detecting a touch input on the touch-sensitive surface, the detecting comprising detecting a plurality of touch locations of the touch input on the touch-sensitive surface during respective time periods having the same duration as each frame of the user interface, wherein the plurality of touch locations detected during the respective time periods correspond to respective movements of the touch input on the touch-sensitive surface; as well as, In the application-independent touch processing module: selecting, based on touch processing criteria of the first software application, a single corresponding touch location of the touch input from the plurality of touch locations detected during the corresponding time period as a representative touch location for the corresponding time period; as well as sending touch location information of the touch input identifying the representative touch location for the corresponding time period to an application-specific portion of the first software application distinct from the touch processing module; as well as, At the first software application, the user interface is updated by generating a display frame based on the touch location information.

19. The electronic device of claim 18, wherein the one or more programs include instructions for executing the method of any one of claims 2-17.

20. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device having a display and a touch-sensitive surface, cause the electronic device to: displaying a user interface of a first software application that updates at a first display rate; While displaying one or more frames of the user interface according to the first display rate: detecting a touch input on the touch-sensitive surface, the detecting comprising detecting a plurality of touch locations of the touch input on the touch-sensitive surface during respective time periods having the same duration as each frame of the user interface, wherein the plurality of touch locations detected during the respective time periods correspond to respective movements of the touch input on the touch-sensitive surface; as well as, In the application-independent touch processing module: selecting, based on touch processing criteria of the first software application, a single corresponding touch location of the touch input from the plurality of touch locations detected during the corresponding time period as a representative touch location for the corresponding time period; as well as sending touch location information of the touch input identifying the representative touch location for the corresponding time period to an application-specific portion of the first software application distinct from the touch processing module; as well as, At the first software application, the user interface is updated by generating a display frame based on the touch location information.

21. The computer-readable storage medium of claim 20, wherein the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method of any one of claims 2-17.

Images