Devices, methods, and user interfaces for transmitting proximity-based and contact-based input events.

Abstract

An electronic device having a display and an input device separate from the display, the electronic device displaying a user interface including a representation of a first application and a pointer object. The device receives a user input event at the input device, the user input event pointing to a location in the user interface where the pointer object is displayed. In response to the user input event, when the corresponding location of the pointer object in the user interface is above the representation of the first application, the device provides the first application with information describing a simulated touch event, the information including information describing the location of one or more simulated touches in the user interface; and displays on the display a change in the representation of the first application determined by the first application in response to the information describing the simulated touch event.

Description

Technical Field

[0001] This disclosure generally relates to electronic devices having input devices, including but not limited to electronic devices having touch-sensitive surfaces that facilitate and transmit proximity-based and contact-based input events. Background Technology

[0002] The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Exemplary touch-sensitive surfaces include touchpads and touchscreen displays. Such surfaces are widely used to manipulate user interface objects on the display.

[0003] Some touch-sensitive surfaces also include sensors that detect input from objects (e.g., styluses) that are not in direct contact with the touch-sensitive surface but are very close to it. Proximity-based input provides an additional way to manipulate user interface objects on a display. However, touch-based and proximity-based inputs often do not work seamlessly together and can interfere with each other, causing confusion and frustration for the user. Summary of the Invention

[0004] Therefore, there is a need to facilitate methods for transmitting input events based on contact and proximity-based user input, as well as based on contact and proximity to appropriate applications and views within such applications. Furthermore, there is a need to shift other user inputs (e.g., user input not based on contact or proximity to a touch-sensitive surface, such as mouse or trackball input) into contact and proximity-based input events, making applications expecting such input events compatible with different operating systems and different peripheral input devices. Such methods and interfaces optionally complement or replace conventional methods for handling contact and proximity-based user input.

[0005] The disclosed device reduces or eliminates the aforementioned deficiencies and other problems associated with event handling in electronic devices. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, tablet, or handheld device). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a “touchscreen” or “touchscreen display”). In some embodiments, the device has a graphical user interface (GUI), one or more processors, memory, and one or more modules, and a program or set of instructions stored in memory for performing a number of functions. In some embodiments, the user interacts with the GUI primarily through stylus and / or finger contact and gestures on a touch-sensitive surface. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0006] According to some embodiments, a method is performed at an electronic device having a display and an input device separate from the display. The method includes simultaneously displaying on the display a user interface comprising: a representation of a first application including a plurality of user interface objects; and a pointer object. The method further includes, while simultaneously displaying the representation of the first application and the pointer object, receiving at the input device separate from the display a user input event pointing to a corresponding position in the user interface where the pointer object is displayed. The method further includes, in response to detecting the user input event, providing the first application with information describing a simulated touch event, the information including information describing the position of one or more simulated touches in the user interface, based on determining that the corresponding position in the user interface where the pointer object is displayed is above the representation of the first application; and displaying on the display a change in the representation of the first application determined by the first application in response to the information describing the simulated touch event.

[0007] According to some embodiments, a method is performed at an electronic device having a display and one or more input devices separate from the display. The method includes displaying a user interface of an application on the display. The method also includes receiving user input events at the one or more input devices separate from the display. The method further includes generating a plurality of independent event streams based on the user input events. The method also includes merging information from the plurality of independent event streams to generate a merged user input event stream; providing the application with information describing the merged user input event stream; and displaying a response of the application on the display, the response being at least partially based on the information describing the merged user input event stream.

[0008] According to some embodiments, a method is performed at an electronic device having a display and an input device. The method includes displaying a user interface of an application comprising a plurality of user interface objects on the display. The method further includes receiving user input at the input device while the user interface of the application is displayed. The method further includes providing the application with a first input event having a stage field with a first stage value, the first input event corresponding to input approaching but not touching a touch-sensitive surface, in response to the user input. The method further includes providing the application with a second input event having a stage field with a second stage value, the second input event corresponding to the user input touching the touch-sensitive surface, after providing the first input event. The method further includes displaying an update to the user interface of the application on the display, the update being at least partially based on the first input event and the second input event.

[0009] According to some embodiments, an electronic device includes a display, an input device (e.g., an optional touch-sensitive surface), optional one or more sensors for detecting proximity of an input object above the touch-sensitive surface, optional one or more sensors for detecting contact intensity with the touch-sensitive surface, one or more processors, a memory, and one or more programs; the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a computer-readable storage medium stores instructions therein that, when executed by an electronic device having a display, an input device (e.g., an optional touch-sensitive surface), optional one or more sensors for detecting proximity of an input object above the touch-sensitive surface, and optional one or more sensors for detecting contact intensity, cause the device to perform or cause the operation of any of the methods described herein to be performed. According to some embodiments, an electronic device includes: a display, an input device (e.g., an optional touch-sensitive surface), one or more optional sensors for detecting the proximity of an input object above the touch-sensitive surface, one or more optional sensors for detecting contact intensity, and means for performing or causing the operation of any of the methods described herein to be performed. According to some embodiments, an information processing apparatus for an electronic device having a display and an input device (e.g., an optional touch-sensitive surface) optionally includes one or more sensors for detecting the proximity of an input object above the touch-sensitive surface, optionally includes one or more sensors for detecting contact intensity, and includes means for performing or causing the operation of any of the methods described herein to be performed.

[0010] Therefore, an improved method is provided for facilitating proximity-based and contact-based user input and transmitting input events based on such user input, thereby improving the effectiveness and efficiency of such devices and increasing user satisfaction. This method can complement or replace conventional methods for handling proximity-based and contact-based input. Attached Figure Description

[0011] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0012] Figure 1A This is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some embodiments.

[0013] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some embodiments.

[0014] Figure 2 A portable multi-functional device with a touchscreen is shown according to some embodiments.

[0015] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some embodiments.

[0016] Figure 4 This is a block diagram of an exemplary electronic stylus according to some embodiments.

[0017] Figures 5A to 5B The positional state of the stylus relative to the touch-sensitive surface is shown according to some embodiments.

[0018] Figure 6A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0019] Figure 6B An exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0020] Figures 7A to 7Q An exemplary user interface is shown, according to some embodiments, for interacting with user interface objects via proximity-based (e.g., hover-based) and contact-based inputs (e.g., moving the cursor and selecting text).

[0021] Figure 8 This is a diagram of an input / output processing stack of an exemplary device having a display and independent input devices according to some embodiments.

[0022] Figures 9A to 9B Exemplary data structures for providing touch input events (including hover-based and touch-based input events) to an application are shown according to some embodiments.

[0023] Figure 10 This is a diagram of an input / output processing stack that, according to some embodiments, merges (e.g., demultiplexes) input events from independent event streams and then provides the merged user input events to the application.

[0024] Figures 11A to 11D A flowchart is shown, according to some embodiments, of a method for providing simulated touch events to an application.

[0025] Figures 12A to 12C A flowchart is shown of a method for merging input events from independent event streams and providing the merged input events to an application, according to some embodiments.

[0026] Figures 13A to 13E A flowchart is shown of a method for providing hover-based input events to an application according to some embodiments. Detailed Implementation

[0027] Portable multifunction devices and laptops typically use entirely different input devices. For example, a portable multifunction device (e.g., a smartphone) may include a touchscreen, while a laptop may include or be coupled to a physical keyboard, mouse, and / or touch-sensitive surface separate from the display. As a result, the input events provided by the device's operating system to the device's applications depend heavily on the type of input device included in the device. Consequently, software applications are often developed for specific types of devices (e.g., portable multifunction devices with touchscreens) and are not easily ported to different operating systems. Instead, application developers must rewrite the software for the different operating systems running on different types of devices.

[0028] Some embodiments described herein allow applications developed for a first operating system framework (e.g., a touchscreen framework) to interpret input events from a second operating system framework (e.g., a framework where user input is received at an input device separate from the display). To this end, input events from the separate display device are converted into simulated touch events that the application can interpret. In some embodiments, this conversion occurs at a second application native to the second operating system framework. The second application is an input event converter that acts as the host of the first application. Thus, the second application acts as an additional layer in the input / output processing stack.

[0029] By using different gestures, devices with touch-sensitive displays (e.g., touchscreens) can interpret a wide range of user input. For example, a single-finger or two-finger swipe on a touchscreen can be interpreted as a scrolling gesture, while a two-finger pinch can be interpreted as a zoom gesture. Devices that include input devices separate from the display (e.g., peripherals) can also disambiguate a wide range of user input, but doing so presents additional challenges. For example, while gesture disambiguation is typically limited to interpreting user input from a single input device (e.g., a touchscreen), user input from a peripheral input device (e.g., a mouse, keyboard, or touchpad separate from the display) is often modified by user input from different peripheral input devices. For example, actuating the mouse wheel can result in zooming when the "control" button on a separate keyboard is pressed, and in scrolling when the "control" button on the keyboard is not pressed.

[0030] The fact that user input from one input device can be modified by user input from different user input devices raises the problem of knowing when input events are relevant and should be interpreted together. This challenge is particularly acute for applications developed for touchscreens but still running on devices with one or more input devices separate from the display (e.g., as described above). Some embodiments described herein address this problem by merging (e.g., demultiplexing) input events from independent event streams into a single merged input event before serving them to the application. Therefore, in some embodiments, the process of determining which input events are relevant is moved (e.g., delegated) from the software application (which may have been developed by a third party) to a host or operating system that is better suited to determine which input events are relevant. Merging input events in this way results in third-party applications interpreting input events with greater consistency. For applications developed for touchscreen frameworks, these embodiments increase cross-platform compatibility by addressing a problem that developers of touchscreen applications (i.e., applications configured to receive input from a touchscreen) rarely have to consider in their native frameworks.

[0031] Some touch-sensitive surfaces are able to detect the proximity of an input object (e.g., a finger or stylus) above the surface. These so-called "hover" inputs can be used to manipulate user interface objects. However, while the fact that something hovers above a touch-sensitive surface can be considered part of the same gesture as a subsequent or previous touch, hover inputs are generally treated separately from other user inputs (e.g., touch-based user inputs). Treating proximity-based and touch-based input events as separate user inputs makes the user interface cumbersome and unintuitive for the user.

[0032] Some embodiments address this problem by providing hover input events in a normalized touch input event format with the same data structure as that used to represent contact-based touch events. These touch input events include a phase field of a presentation value that describes whether the input event is a hover event or a contact-based event, etc. (This phase value may also indicate whether the event is a change or the end of a previously detected event). In some embodiments, hover input events associated with contact-based input events are assigned the same input identifier, so that when these events are provided to a gesture recognizer, they can be interpreted as part of the same gesture. To aid cross-platform compatibility, as described above, some embodiments also provide simulated hover events based on user input from an input device that does not detect proximity to contact above a touch-sensitive surface. For example, a mouse pointer entering a window can be interpreted as a hover event above that window, while a left mouse click while the pointer is above the window can be interpreted as a contact within that window.

[0033] As described below, Figures 1A to 1B , Figure 2 and Figure 3 An exemplary device is shown. Figure 4 An exemplary electronic stylus is shown. Figures 5A to 5B The position of the stylus relative to the touch-sensitive surface is shown. Figures 6A to 6B and Figures 7A to 7Q An exemplary user interface is shown for interacting with user interface objects through user input. Figure 8 and Figure 10 It shows the method for using based on Figures 6A to 6B and Figures 7A to 7Q The example input / output processing stack shown provides input events to the application. Figures 9A to 9B It shows the method for using based on Figures 6A to 6B and Figures 7A to 7Q The input shown is an example data structure that provides the application with touch-based input events. Figures 11A to 11D , Figures 12A to 12C and Figures 13A to 13E It shows the basis Figures 6A to 6B and Figures 7A to 7Q The method shown provides input events to the application.

[0034] Exemplary device

[0035] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0036] It will also be understood that, although in some cases the terms “first,” “second,” etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0037] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0038] As used herein, depending on the context, the term “if” is optionally interpreted as meaning “when…” followed by “at…” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrase “if determination…” or “if [the stated condition or event] is optionally interpreted as meaning “in response to determination…” or “in response to detection of [the stated condition or event]” or “in response to detection of [the stated condition or event].”

[0039] This document describes embodiments of electronic devices, user interfaces of such devices, and related processes for using such devices. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functionality. Exemplary embodiments of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0040] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, trackball, and / or joystick.

[0041] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0042] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0043] Now let’s turn our attention to examples of portable devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 having a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touchscreen" for convenience, and is sometimes simply referred to as a touchscreen display. Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more haptic output generators 163 for generating haptic outputs on device 100 (e.g., generating haptic outputs on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0044] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. For example, even when the smoothness of the tactile surface remains unchanged, the movement of the tactile surface can optionally be interpreted or sensed by the user as the "roughness" of the tactile surface. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user.

[0045] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application-specific integrated circuits).

[0046] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100 (such as CPU 120 and peripheral interface 118) is optionally controlled by memory controller 122.

[0047] Peripheral interface 118 can be used to couple the device's input peripherals and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0048] In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0049] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals and vice versa, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). This wireless communication may optionally use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolved Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE...). 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extended Protocol (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including those not yet developed as of the date of this document submission.

[0050] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a mono or binaural headphone) and input (e.g., a microphone).

[0051] I / O subsystem 106 couples input / output peripherals on device 100, such as touch-sensitive display system 112 and other input or control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from / send electrical signals to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, stylus, and / or pointing device such as mouse. One or more buttons (e.g., Figure 2 Optionally, 208) includes an up / down button for volume control of speaker 111 and / or microphone 113. One or more buttons optionally include a push-down button (e.g., Figure 2 (206 in the middle).

[0052] The touch-sensitive display system 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. Visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object. As used herein, the term "enabled representation" is a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0053] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile / haptic contact. The touch-sensitive display system 112 and the display controller 156 (along with any associated modules and / or instruction set in the memory 102) detect contact on the touch-sensitive display system 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In one exemplary embodiment, the contact point between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0054] The touch-sensitive display system 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed thereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In one exemplary embodiment, projected mutual capacitance sensing technology, such as that from Apple Inc. (Cupertino, California), is used. iPod and The technology discovered in [the text].

[0055] The touch-sensitive display system 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or greater). Users optionally use any suitable object or accessory such as a stylus, finger, etc., to interact with the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0056] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0057] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0058] The device 100 may optionally also include one or more optical sensors 164. Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) phototransistors. The one or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also referred to as a camera module), the one or more optical sensors 164 optionally capture still images and / or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to the touch-sensitive display system 112 on the front of the device, enabling the touchscreen to function as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to acquire images of the user (e.g., for selfies, for video conferencing while the user views other video conference participants on the touchscreen, etc.).

[0059] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor coupled to a strength sensor controller 159 in I / O subsystem 106 is shown. One or more contact strength sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact strength sensors 165 receive contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100 opposite to the touch-sensitive display system 112 located on the front of device 100.

[0060] The device 100 optionally also includes one or more proximity sensors 166. Figure 1A A proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 is coupled to an input controller 160 in an I / O subsystem 106. In some embodiments, when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call), the proximity sensor is turned off and the touch-sensitive display system 112 is disabled.

[0061] The device 100 may optionally also include one or more tactile output generators 163. Figure 1A A haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 163 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on the device). One or more haptic output generators 163 receive haptic feedback generation instructions from haptic feedback module 133 and generate haptic outputs on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0062] The device 100 may optionally also include one or more accelerometers 167, gyroscopes 168 and / or magnetometers 169 (e.g., as part of an inertial measurement unit (IMU)) for obtaining information about the device’s position (e.g., attitude). Figure 1A Sensors 167, 168, and 169, coupled to peripheral interface 118, are shown. Alternatively, sensors 167, 168, and 169 may be coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 100 may optionally include a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of device 100.

[0063] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a position module (or instruction set) 131, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 stores device / global internal state 157, as shown in Figures 1A and 1B. Figure 3 As shown in the diagram. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views or other information occupy the various areas of the touch-sensitive display system 112; sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information regarding the device's position and / or orientation.

[0064] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0065] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, Wireless LAN, etc.). In some embodiments, the external ports are compatible with some Apple Inc. (Cupertino, California) devices. iPod and The device uses the same or similar and / or compatible multi-pin (e.g., 30-pin) connectors as the 30-pin connector used in the device. In some embodiments, the external port is compatible with some Apple Inc. (Cupertino, California) connectors. iPod and The device uses the same or similar and / or compatible Lightning connectors.

[0066] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations can optionally be applied to single-point contact (e.g., single-finger contact or stylus contact) or multi-point simultaneous contact (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0067] The contact / motion module 130 optionally detects user gesture input. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the icon location). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event. Similarly, stylus taps, swipes, drags, and other gestures are optionally detected by detecting specific contact patterns of the stylus.

[0068] In some embodiments, detecting a finger tap gesture depends on detecting the duration between a finger press event and a finger release event, but is independent of the intensity of finger contact between the two events. In some embodiments, a tap gesture is detected if the duration between the finger press event and the finger release event is determined to be less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the intensity of finger contact during the tap reaches a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a light press or deep press intensity threshold. Therefore, a finger tap gesture can satisfy a specific input criterion that does not require the characteristic intensity of the contact to meet a given intensity threshold to satisfy that criterion. For clarity, finger contact in a tap gesture typically needs to meet a nominal contact detection intensity threshold to detect a finger press event; below this threshold, no contact is detected. Similar analysis applies to detecting tap gestures via a stylus or other contact method. When the device is capable of detecting contact from a finger or stylus hovering above a touch-sensitive surface, the nominal contact detection strength threshold may optionally not correspond to the physical contact between the finger or stylus and the touch-sensitive surface.

[0069] The same concept applies to other types of gestures in a similar manner. For example, swipe gestures, pinch gestures, spread gestures, and / or long press gestures can be optionally detected based on criteria that are independent of the intensity of the contact involved in the gesture or do not require one or more contacts performing the gesture to reach an intensity threshold for recognition. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a zoom gesture is detected based on the movement of two or more contacts toward each other; a spread gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of contact with less than a threshold amount of movement on a touch-sensitive surface. Therefore, the statement that a particular gesture recognition criterion does not require the contact intensity to meet a corresponding intensity threshold implies that a particular gesture recognition criterion can be met when the contact in the gesture does not reach the corresponding intensity threshold, and also when one or more contacts in the gesture reach or exceed the corresponding intensity threshold. In some embodiments, tap gestures are detected based on determining that a finger press event and a finger lift event are detected within a predefined time period, regardless of whether the contact is above or below a corresponding intensity threshold during the predefined time period, and swipe gestures are detected based on determining that the contact movement is greater than a predefined amount, even if the contact is above a corresponding intensity threshold at the end of the contact movement. Even in embodiments where gesture detection is affected by the intensity of the contact performing the gesture (e.g., the device detects a long press faster when the contact intensity is above an intensity threshold, or delays the detection of a tap input when the contact intensity is even higher), the detection of these gestures does not require the contact to reach a specific intensity threshold (e.g., even if the amount of time required to recognize the gesture varies) as long as the criteria for recognizing the gesture are met when the contact does not reach a specific intensity threshold.

[0070] In some cases, contact intensity thresholds, duration thresholds, and movement thresholds are combined in various different combinations to create heuristic algorithms that distinguish between two or more different gestures targeting the same input element or region, allowing for a richer set of user interactions and responses from multiple different interactions with the same input element. Statements that a particular set of gesture recognition criteria does not require the contact intensity to meet a corresponding intensity threshold to satisfy a particular gesture recognition criterion do not preclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures that meet criteria when the gesture includes contact with an intensity higher than the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the contact intensity to meet a corresponding intensity threshold to satisfy the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on the contact reaching the corresponding intensity threshold). In such competition, if the second gesture recognition criterion for the second gesture is satisfied first, the gesture is optionally not recognized as satisfying the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before the contact moves a predefined amount of movement, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves a predefined amount of motion before reaching the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in such cases, the first gesture recognition criterion for the first gesture still does not require the contact intensity to meet the corresponding intensity threshold to satisfy the first gesture recognition criterion, because if the contact remains below the corresponding intensity threshold until the gesture ends (e.g., a swipe gesture with a contact intensity that does not increase to above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criterion. Therefore, a specific gesture recognition criterion that does not require the contact intensity to meet the corresponding intensity threshold to satisfy a specific gesture recognition criterion will (A) in some cases ignore the contact intensity relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases fail to satisfy the specific gesture recognition criterion (e.g., for a long press gesture) if a set of competing intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognize the input as corresponding to an intensity-related gesture before the specific gesture recognition criterion recognizes the gesture corresponding to the input, in this sense, still depend on the contact intensity relative to the intensity threshold (e.g., for a long press gesture that competes with a deep press gesture for recognition).

[0071] In conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169, position module 131 optionally detects position information about the device, such as the device's attitude (roll, pitch, and / or yaw) in a particular reference frame. Position module 130 includes software components for performing various operations related to detecting the device's position and detecting changes in the device's position. In some embodiments, position module 131 uses information received from a stylus used with the device to detect position information about the stylus, such as detecting the stylus's positional state relative to the device and detecting changes in the stylus's positional state.

[0072] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, and non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0073] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like to specify the graphic to be displayed, and, if necessary, also receives coordinate data and other graphic attribute data, and then generates screen image data to output to the display controller 156.

[0074] The haptic feedback module 133 includes various software components for generating instructions that are used by the haptic output generator 163 to produce haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0075] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0076] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to phone 138 for location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services such as weather desktop apps, local yellow pages desktop apps, and map / navigation desktop apps).

[0077] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0078] • Contacts module 137 (sometimes called address book or contact list);

[0079] • Telephone module 138;

[0080] • Video conferencing module 139;

[0081] • Email client module 140;

[0082] • Instant Messaging (IM) module 141;

[0083] Fitness support module 142;

[0084] • Camera module 143 for still images and / or video images;

[0085] • Image management module 144;

[0086] • Browser module 147;

[0087] • Calendar module 148;

[0088] • Desktop mini-program module 149, which optionally includes one or more of the following: weather desktop mini-program 149-1, stock market desktop mini-program 149-2, calculator desktop mini-program 149-3, alarm clock desktop mini-program 149-4, dictionary desktop mini-program 149-5, and other desktop mini-programs obtained by the user, and desktop mini-programs created by the user 149-6;

[0089] • Desktop app creator module 150 for creating user-created desktop apps 149-6;

[0090] • Search module 151;

[0091] • A video and music player module 152, optionally composed of a video player module and a music player module;

[0092] • Memo module 153;

[0093] • Map module 154; and / or

[0094] • Online video module 155.

[0095] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0096] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contact module 137 includes executable instructions for managing an address book or contact list (e.g., in the application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communication via telephone 138, video conferencing 139, email 140, or instant messaging 141; and so on.

[0097] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 includes executable instructions for performing the following operations: inputting a character sequence corresponding to a telephone number, accessing one or more telephone numbers in the address book 137, modifying an input telephone number, dialing a corresponding telephone number, initiating a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communication optionally employs any of a variety of communication standards, protocols, and technologies.

[0098] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0099] Incorporating RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0100] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for internet-based instant messaging), receiving an instant message, and viewing a received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0101] Incorporating RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (in sports equipment and smartwatches); receiving fitness sensor data; calibrating sensors used for monitoring fitness; selecting and playing music for fitness activities; and displaying, storing, and transmitting fitness data.

[0102] In conjunction with the touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for performing the following operations: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0103] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, displaying (e.g., in a digital slideshow or photo album), and storing still images and / or video images.

[0104] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) according to user instructions.

[0105] Incorporating RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0106] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, desktop applet module 149 is optionally a micro-application downloaded and used by a user (e.g., weather desktop applet 149-1, stock market desktop applet 149-2, calculator desktop applet 149-3, alarm clock desktop applet 149-4, and dictionary desktop applet 149-5) or a user-created micro-application (e.g., user-created desktop applet 149-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop applet includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop applet).

[0107] Combining RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, desktop applet creator module 150 includes executable instructions for creating desktop applets (e.g., transferring user-specified portions of web pages to desktop applets).

[0108] In conjunction with the touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0109] In conjunction with the touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, the video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on the touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0110] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0111] Combining RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and map-related data (e.g., driving routes; data on shops and other points of interest at or near specific locations; and other location-based data) according to user instructions.

[0112] In conjunction with the touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes executable instructions that allow users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen 112 or on an external display connected wirelessly or via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, the instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos.

[0113] Each module and application identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures described above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0114] In some embodiments, device 100 is a device on which the operation of a predefined set of functions is performed exclusively via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on device 100 is optionally reduced.

[0115] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to a main menu, home menu, or root menu. In such embodiments, a touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0116] Figure 1B This is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A (middle) or memory 370 ( Figure 3 This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0117] Event classifier 170 receives event information and determines the application 136-1 to which the event information should be delivered and the application view 191 of application 136-1. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display system 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0118] In some embodiments, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0119] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display system 112 as part of a multi-touch gesture). Peripheral device interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (via audio circuitry 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or touch-sensitive surfaces.

[0120] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or exceeding a predetermined duration).

[0121] In some embodiments, the event classifier 170 further includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0122] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0123] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0124] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that forms an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.

[0125] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, the higher-level view in the hierarchy will still remain the actively participating view.

[0126] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver module 182.

[0127] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.

[0128] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0129] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies events from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0130] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).

[0131] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display system 112, and lifting off the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0132] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on a touch-sensitive display system 112, when a touch is detected on the touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0133] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0134] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0135] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0136] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and delaying) the sub-event to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag receives the tag and executes a predefined procedure.

[0137] In some embodiments, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and executes a predetermined procedure.

[0138] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0139] In some embodiments, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0140] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0141] Figure 2 It is shown that some embodiments have a touch screen (e.g., Figure 1AA portable multi-functional device 100 (a touch-sensitive display system 112). The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figures) or one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of one or more graphics occurs when the user interrupts contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0142] Device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on a touchscreen display.

[0143] In some embodiments, device 100 includes a touchscreen display, a menu button 204, a push-button 206 for powering on / off and locking the device, a volume control button 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In some embodiments, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting contact strength on the touch-sensitive display system 112, and / or one or more haptic output generators 163 for generating haptic outputs for a user of device 100.

[0144] Figure 3This is a block diagram of an exemplary multi-functional device having a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the above references). Figure 1A The one or more tactile output generators 163 and sensors 359 (e.g., similar to those mentioned above) Figure 1A The sensors 112, 164, 165, 166, 167, 168, and 169 are touch sensors, optical sensors, contact strength sensors, proximity sensors, acceleration sensors, attitude and / or magnetic sensors. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDRRAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores data with portable multifunction device 100 (…). Figure 1A The memory 370 stores programs, modules, and data structures similar to those stored in the memory 102 of the portable multifunction device 100, or subsets thereof. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 (… Figure 1A The memory 102 may optionally not store these modules.

[0145] Figure 3Each of the elements identified above is optionally stored in one or more of the previously mentioned memory devices. Each of the modules identified above corresponds to an instruction set for performing the functions described above. The modules or programs identified above (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures described above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0146] Figure 4 This is a block diagram of an exemplary electronic stylus 203 according to some embodiments. The electronic stylus 203 is sometimes simply referred to as a stylus. The stylus 203 includes a memory 402 (which optionally includes one or more computer-readable storage media), a memory controller 422, one or more processing units (CPUs) 420, a peripheral device interface 418, RF circuitry 408, an input / output (I / O) subsystem 406, and other input or control devices 416. The stylus 203 optionally includes an external port 424 and one or more optical sensors 464. The stylus 203 optionally includes one or more intensity sensors 465 for detecting the intensity of contact of the stylus 203 on device 100 (e.g., when the stylus 203 is used with a touch-sensitive surface such as a touch-sensitive display system 112 of device 100) or on other surfaces (e.g., a table surface). The stylus 203 optionally includes one or more haptic output generators 463 for generating haptic output on the stylus 203. These components optionally communicate via one or more communication buses or signal lines 403.

[0147] In some embodiments, the term "haptic output" discussed above refers to the physical displacement of an accessory (e.g., stylus 203) of a device (e.g., device 100) relative to a previous position of the accessory, the physical displacement of a component of the accessory relative to another component of the accessory, or the displacement of a component relative to the center of mass of the accessory, which is detected by the user through the user's tactile sense. For example, when the accessory or a component of the accessory comes into contact with a touch-sensitive surface of the user (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the accessory or a component of the accessory. For example, movement of a component (e.g., the housing of stylus 203) may optionally be interpreted by the user as a "click" on a physically actuated button. In some cases, the user will feel a tactile sensation such as a "click" even when the physically actuated button associated with the stylus, which is physically pressed (e.g., displaced) by the user's movement, does not move. While such interpretations of touch by the user will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Therefore, when haptic output is described as corresponding to a specific user perception (e.g., "click"), unless otherwise stated, the generated haptic output corresponds to a physical displacement of the device or its components that would generate the perception of a typical (or ordinary) user.

[0148] It should be understood that stylus 203 is merely an example of an electronic stylus, and stylus 203 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 4 The various components shown are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application-specific integrated circuits).

[0149] Memory 402 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more flash memory devices or other non-volatile solid-state memory devices. Access to memory 402 by other components of stylus 203 (such as one or more CPUs 420 and peripheral interface 418) is optionally controlled by memory controller 422.

[0150] Peripheral interface 418 can be used to couple the stylus's input and output peripherals to one or more CPUs 420 and memory 402. The one or more processors 420 run or execute various software programs and / or instruction sets stored in memory 402 to perform various functions of stylus 203 and process data.

[0151] In some embodiments, the peripheral interface 418, CPU 420, and memory controller 422 are optionally implemented on a single chip, such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0152] RF (Radio Frequency) circuit 408 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 408 converts electrical signals into electromagnetic signals and vice versa, and communicates with device 100 or device 300, communication networks, and / or other communication devices via electromagnetic signals. RF circuit 408 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 408 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). The wireless communication may optionally use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolved Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including those not yet developed as of the date of this document submission.

[0153] The I / O subsystem 406 couples input / output peripherals such as other input or control devices 416 on the stylus 203 to the peripheral interface 418. The I / O subsystem 406 optionally includes an optical sensor controller 458, an intensity sensor controller 459, a haptic feedback controller 461, and one or more input controllers 460 for the other input or control devices. The one or more input controllers 460 receive electrical signals from / send electrical signals to the other input or control devices 416. The other input or control devices 416 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, click wheels, etc. In some alternative embodiments, one or more input controllers 460 are optionally coupled to (or not coupled to) any of the following: an infrared port and / or a USB port.

[0154] The stylus 203 also includes a power system 462 for supplying power to various components. The power system 462 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in portable devices and / or portable accessories.

[0155] The stylus 203 may optionally also include one or more optical sensors 464. Figure 4 An optical sensor 464 coupled to an optical sensor controller 458 in an I / O subsystem 406 is shown. The optical sensor 464 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 464 receives light projected through one or more lenses from the environment and converts the light into data representing an image.

[0156] The stylus 203 may optionally also include one or more contact intensity sensors 465. Figure 4 A contact strength sensor coupled to a strength sensor controller 459 in I / O subsystem 406 is shown. One or more contact strength sensors 465 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a surface). Contact strength sensor 465 receives contact strength information (e.g., pressure information or pressure information proxy) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to the tip of stylus 203.

[0157] The stylus 203 may optionally also include one or more proximity sensors 466. Figure 4 A proximity sensor 466 coupled to a peripheral device interface 418 is shown. Alternatively, the proximity sensor 466 is coupled to an input controller 460 in an I / O subsystem 406. In some embodiments, the proximity sensor determines the proximity of the stylus 203 to an electronic device (e.g., device 100).

[0158] The stylus 203 may optionally also include one or more haptic output generators 463. Figure 4 A haptic output generator coupled to a haptic feedback controller 461 in I / O subsystem 406 is shown. The haptic output generator 463 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on the device). One or more haptic output generators 463 receive haptic feedback generation instructions from haptic feedback module 433 and generate a haptic output on stylus 203 that can be felt by a user of stylus 203. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to the length (e.g., body or housing) of stylus 203, and optionally, the haptic output is generated by moving stylus 203 vertically (e.g., in a direction parallel to the length of stylus 203) or laterally (e.g., in a direction perpendicular to the length of stylus 203).

[0159] The stylus 203 may optionally also include one or more accelerometers 467, gyroscopes 468 and / or magnetometers 470 (e.g., as part of an inertial measurement unit (IMU)) for acquiring information about the position and position state of the stylus 203. Figure 4 Sensors 467, 469, and 470, coupled to peripheral interface 418, are shown. Alternatively, sensors 467, 469, and 470 may be coupled to input controller 460 in I / O subsystem 406. Stylus 203 may optionally include a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information about the location of stylus 203.

[0160] In some embodiments, the software components stored in memory 402 include an operating system 426, a communication module (or instruction set) 428, a contact / motion module (or instruction set) 430, a location module (or instruction set) 431, and a Global Positioning System (GPS) module (or instruction set) 435. Furthermore, in some embodiments, memory 402 stores device / global internal state 457, such as... Figure 4As shown. Device / global internal state 457 includes one or more of the following: sensor state, including information obtained from various sensors of the stylus and other input or control devices 416; position state, including information about the stylus's position relative to the device (e.g., device 100) (e.g., position, orientation, tilt, rotation, and / or distance, such as...). Figure 5A and Figure 5B Information (as shown); and location information about the stylus position (e.g., determined by GPS module 435).

[0161] The operating system 426 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.) and facilitates communication between various hardware and software components.

[0162] Communication module 428 is optionally adapted to communicate with other devices via one or more external ports 424, and also includes various software components for processing data received by RF circuitry 408 and / or external ports 424. External ports 424 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., Internet, Wireless LAN, etc.). In some embodiments, the external port is associated with some devices from Apple Inc. (Cupertino, California). iPod and The device uses the same or similar and / or compatible Lightning connectors.

[0163] The contact / motion module 430 optionally detects contact with the stylus 203 and other touch-sensitive devices of the stylus 203 (e.g., buttons or other touch-sensitive components of the stylus 203). The contact / motion module 430 includes software components for performing various operations related to the detection of contact (e.g., detection of contact between the tip of the stylus and a touch-sensitive display such as the touchscreen 112 of device 100 or with another surface such as a table surface), such operations as determining whether a contact has occurred (e.g., detecting a touch press event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a substitute for force or pressure), determining whether there is movement of the contact and tracking across that movement (e.g., across the touchscreen 112 of device 100), and determining whether the contact has stopped (e.g., detecting a lift-off event or a contact disconnection). In some embodiments, the contact / motion module 430 receives contact data from the I / O subsystem 406. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. As described above, in some embodiments, one or more of these operations related to the detection of contact are performed by the device using contact / motion module 130 (contact / motion module 430 is used as a supplement to or in place of the stylus).

[0164] The contact / motion module 430 optionally detects gesture input from the stylus 203. Different gestures performed with the stylus 203 have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single tap gesture includes detecting a touch press event, followed by detecting a lift-off event at the same (or substantially the same) location as the touch press event (e.g., at the location of an icon). As another example, detecting a swipe gesture includes detecting a touch press event, followed by detecting one or more stylus drag events, and then detecting a lift-off event. As described above, in some embodiments, gesture detection is performed by the device using the contact / motion module 130 (as a supplement to or alternative to the stylus).

[0165] In conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469, position module 431 optionally detects position information about the stylus, such as the stylus's attitude (roll, pitch, and / or yaw) in a particular frame of reference. In conjunction with accelerometer 467, gyroscope 468, and / or magnetometer 469, position module 431 optionally detects stylus movement gestures, such as flicking, tapping, and rotating the stylus. Position module 431 includes software components for performing various operations related to detecting the stylus's position and detecting changes in the stylus's position in a particular frame of reference. In some embodiments, position module 431 detects the stylus's positional state relative to the device and detects changes in the stylus's positional state relative to the device. As described above, in some embodiments, device 100 or device 300 uses position module 431 to determine the stylus's positional state relative to the device and changes in the stylus's positional state (using position module 431 as a supplement to or alternative to the stylus).

[0166] The haptic feedback module 433 includes various software components for generating instructions used by one or more haptic output generators 463 to produce haptic output at one or more locations on the stylus 203 in response to user interaction with the stylus 203.

[0167] GPS module 435 determines the location of the stylus and provides that information for use in various applications (e.g., applications that provide location-based services, such as applications for finding lost devices and / or accessories).

[0168] Each module and application identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 402 optionally stores a subset of the modules and data structures described above. Furthermore, memory 402 optionally stores additional modules and data structures not described above.

[0169] Figures 5A to 5B The diagram illustrates the positional state of a stylus 203 relative to a touch-sensitive surface (e.g., the touchscreen 112 of device 100) according to some embodiments. In some embodiments, the positional state of the stylus 203 corresponds to (or indicates): the projected position of the stylus tip (or other representative portion) on the touch-sensitive surface (e.g., Figure 5A The (x,y) position 504), the orientation of the stylus relative to the touch-sensitive surface (e.g., Figure 5A Orientation 506), the tilt of the stylus relative to the touch-sensitive surface (e.g., Figure 5BThe tilt (512) and / or the distance of the stylus relative to the touch-sensitive surface (e.g., Figure 5B (Distance 514 in the middle). In some embodiments, the position state of the stylus 203 corresponds to (or indicates) the pitch, yaw, and / or rotation of the stylus (e.g., the attitude of the stylus relative to a particular reference frame such as a touch-sensitive surface (e.g., touchscreen 112) or the ground). In some embodiments, the position state includes a set of position parameters (e.g., one or more position parameters). In some embodiments, the position state is detected based on one or more measurements sent from the stylus 203 to an electronic device (e.g., device 100). For example, the stylus measures the tilt of the stylus (e.g., Figure 5B Inclination 512) and / or orientation (e.g., Figure 5A The orientation 506) and the measurement results are sent to device 100. In some embodiments, instead of or in combination with the position state detected based on one or more measurements from stylus 203, the position state is detected based on raw output from one or more electrodes in the stylus sensed by a touch-sensitive surface (e.g., touchscreen 112 of device 100). For example, the touch-sensitive surface receives raw output from one or more electrodes in the stylus and calculates the stylus tilt and / or orientation based on the raw output (optionally, in combination with position state information provided by the stylus based on sensor measurements generated by the stylus).

[0170] Figure 5A The image shows a stylus 203 relative to a touch-sensitive surface, viewed from a perspective directly above the touch-sensitive surface (e.g., the touchscreen 112 of device 100), according to some embodiments. Figure 5A In the figure, the z-axis 594 points out of the page (i.e., in the direction perpendicular to the plane of the touch screen 112), the x-axis 590 is parallel to the first edge (e.g., length) of the touch screen 112, the y-axis 592 is parallel to the second edge (e.g., width) of the touch screen 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0171] Figure 5AThe tip of stylus 203 is shown at (x,y) position 504. In some embodiments, the tip of stylus 203 is the end of the stylus, configured to determine the proximity of the stylus to a touch-sensitive surface (e.g., touchscreen 112). In some embodiments, the projection of the stylus tip onto the touch-sensitive surface is an orthogonal projection. In other words, the projection of the stylus tip onto the touch-sensitive surface is at the end of a line perpendicular to the surface of the touch-sensitive surface from the stylus tip (e.g., (x,y) position 504, where the stylus tip would touch the touch-sensitive surface if the stylus were moved directly along a path perpendicular to the touch-sensitive surface). In some embodiments, the (x,y) position at the lower left corner of touchscreen 112 is position (0,0) (e.g., (0,0) position 502), and other (x,y) positions on touchscreen 112 are relative to the lower left corner of touchscreen 112. Alternatively, in some embodiments, the (0,0) position is located at another location on the touchscreen 112 (e.g., at the center of the touchscreen 112) and other (x,y) positions are relative to the (0,0) position on the touchscreen 112.

[0172] in addition, Figure 5A The stylus 203 is shown to have an orientation 506. In some embodiments, orientation 506 is the orientation of the projection of the stylus 203 onto the touchscreen 112 (e.g., an orthogonal projection of the length of the stylus 203 or a line corresponding to the line between the projections of two different points of the stylus 203 onto the touchscreen 112). In some embodiments, orientation 506 is relative to at least one axis in a plane parallel to the touchscreen 112. In some embodiments, orientation 506 is relative to a single axis in a plane parallel to the touchscreen 112 (e.g., axis 508, having a clockwise rotation angle varying from 0 degrees to 360 degrees starting from axis 508, such as...). Figure 5A (As shown). Alternatively, in some embodiments, orientation 506 is relative to a pair of axes in a plane parallel to touchscreen 112 (e.g., x-axis 590 and y-axis 592, as shown). Figure 5A As shown, or a pair of axes associated with the application displayed on the touchscreen 112.

[0173] In some embodiments, an indication (e.g., indication 516) is displayed on a touch-sensitive display (e.g., touchscreen 112 of device 100). In some embodiments, indication 516 indicates the location on the touch-sensitive display where the stylus will touch (or mark) the touch-sensitive display before the stylus touches it. In some embodiments, indication 516 is part of a mark drawn on the touch-sensitive display. In some embodiments, indication 516 is separate from the mark drawn on the touch-sensitive display and corresponds to a virtual “pen tip” or other element indicating the location where a mark will be drawn on the touch-sensitive display.

[0174] In some embodiments, the indicator 516 is displayed based on the position state of the stylus 203. For example, in some cases, the indicator 516 is shifted from the (x,y) position 504 (e.g., Figure 5A and Figure 5B As shown), and in other cases, indicator 516 does not shift from (x,y) position 504 (e.g., when tilt 512 is zero degrees, indicator 516 is displayed at or near (x,y) position 504). In some embodiments, indicator 516 is displayed with varying colors, sizes (or radii or areas), opacities, and / or other characteristics depending on the stylus's position state. In some embodiments, the displayed indicator takes into account the thickness of the glass layer on the touch-sensitive display so that the indicator is applied to the “pixel” of the touch-sensitive display, rather than displayed on the “glass” covering the pixel. As used herein, “indicator” can also refer to a location not visible on the touchscreen when describing an exemplary user interface.

[0175] Figure 5B The stylus 203 is shown relative to the touch-sensitive surface, viewed from a side view of the touch-sensitive surface (e.g., the touchscreen 112 of device 100) according to some embodiments. Figure 5B In the figure, the z-axis 594 points in a direction perpendicular to the plane of the touch screen 112, the x-axis 590 is parallel to the first edge (e.g., length) of the touch screen 112, the y-axis 592 is parallel to the second edge (e.g., width) of the touch screen 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0176] Figure 5B The stylus 203 is shown to have a tilt 512. In some embodiments, the tilt 512 is an angle relative to the normal to the surface of the touch-sensitive surface (also simply referred to as the normal to the touch-sensitive surface) (e.g., normal 510). Figure 5B As shown, when the stylus is perpendicular to / perpendicular to the touch-sensitive surface (e.g., when stylus 203 is parallel to normal 510), the tilt 512 is zero, and the tilt increases as the stylus tilts closer to being parallel to the touch-sensitive surface.

[0177] in addition, Figure 5B The distance 514 between the stylus 203 and the touch-sensitive surface is shown. In some embodiments, the distance 514 is the distance from the tip of the stylus 203 to the touch-sensitive surface in a direction perpendicular to the touch-sensitive surface. For example, in Figure 5B In the middle, distance 514 is the distance from the tip of stylus 203 to position 504 at (x,y).

[0178] Although this document uses the terms "x-axis," "y-axis," and "z-axis" to indicate certain directions in specific figures, it should be understood that these terms do not refer to absolute directions. In other words, the "x-axis" can be any corresponding axis, and the "y-axis" can be a specific axis different from the x-axis. Typically, the x-axis is perpendicular to the y-axis. Similarly, the "z-axis" is different from the "x-axis" and "y-axis," and is typically perpendicular to both the "x-axis" and "y-axis."

[0179] in addition, Figure 5B The rotation 518 is shown—a rotation about the length (major axis) of the stylus 203.

[0180] Now turn attention to an embodiment of the user interface (“UI”) optionally implemented on the portable multifunction device 100.

[0181] Figure 6A An exemplary user interface for a menu for an application on a portable multifunction device 100, according to some embodiments, is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 600 includes elements or a subset or superset thereof:

[0182] • One or more signal strength indicators 602 for one or more wireless communications such as cellular signals and Wi-Fi signals;

[0183] • Time 604;

[0184] Bluetooth indicator 605;

[0185] • Battery status indicator 606;

[0186] • Tray 608 with icons for commonly used applications, such as:

[0187] The telephone module 138 has an icon 616 labeled "telephone", which optionally includes an indicator 614 indicating the number of missed calls or voicemails;

[0188] The email client module 140 has an icon 618 labeled "Mail", which optionally includes an indicator 610 for the number of unread emails;

[0189] The icon 620 of browser module 147, labeled "browser"; and

[0190] The video and music player module 152 (also known as the iPod (a trademark of Apple Inc.) module 152) is labeled with an icon 622 marked "iPod"; and

[0191] • Icons of other applications, such as:

[0192] Icon 624 of module 141 marked as “text”;

[0193] The icon 626 labeled "Calendar" in the calendar module 148;

[0194] The image management module 144 has an icon 628 labeled "Photo".

[0195] The icon 630 of the camera module 143, labeled "camera";

[0196] The icon 632 of the online video module 155, labeled "Online Video";

[0197] The icon labeled "Stock Market" in the Stock Market Desktop Mini Program 149-2 is 634.

[0198] The icon 636 labeled "Map" in map module 154;

[0199] The icon labeled "Weather" in the Weather Desktop Mini Program 149-1 is 638.

[0200] The icon labeled "Clock" in the alarm clock desktop mini-program 169-6 is 640.

[0201] The icon 642 labeled "Fitness Support" in the Fitness Support module 142;

[0202] The icon 644 labeled "Notepad" in the Notepad module 153; and

[0203] An icon 646 is used to set up an application or module, which provides access to settings for the device 100 and its various applications 136.

[0204] It should be pointed out that, Figure 6A The icon labels shown are merely exemplary. For example, in some embodiments, the icon 622 of the video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a corresponding application icon includes the name of the application corresponding to that application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0205] Figure 6B A touch-sensitive surface 651 separate from the display 650 is shown (e.g., Figure 3 Devices (e.g., tablets or touchpads 355) Figure 3An exemplary user interface on device 300. Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting contact intensity on the tactile surface 651 and / or one or more tactile output generators 357 for generating tactile output for the user of device 300.

[0206] Figure 6B A touch-sensitive surface 651 separate from the display 650 is shown (e.g., Figure 3 Devices (e.g., tablets or touchpads 355) Figure 3 An exemplary user interface on device 300. In some embodiments, a touch-sensitive surface (e.g., ...) is used. Figure 6B 651) has a main axis (e.g., on the display (e.g., 650) that is aligned with the main axis on the display (e.g., 651). Figure 6B The principal axis corresponding to 653 in the middle (e.g., Figure 6B (652 in the text). According to these embodiments, the device detects contact with the touch-sensitive surface 651 at a position corresponding to a corresponding position on the display (e.g., ...). Figure 6B (660 and 662 in the example) Figure 6B In this context, 660 corresponds to 668 and 662 corresponds to 670. Thus, on touch-sensitive surfaces (e.g., Figure 6B 651 in the middle) and the display of a multi-functional device (e.g., Figure 6B When 650 is separated from the touch-sensitive surface, user input detected by the device on the touch-sensitive surface (e.g., touches on 660 and 662 and their movement) is used by the device to manipulate the user interface on the display. Some examples below will be given with reference to devices that detect input on a touch-sensitive surface separate from the display, such as... Figure 6B As shown. Some examples below will be given with reference to input on a touchscreen display 112 (in which a touch-sensitive surface and a display are combined). It should be understood that similar methods may be used optionally for other user interfaces described herein.

[0207] Additionally, while some of the examples below are given primarily with reference to finger input (e.g., finger touch, finger tap gestures, finger press gestures, and finger swipe gestures, etc.), it should be understood that in some embodiments, one or more finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus-based input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the touch). Similarly, a tap gesture is optionally replaced by a mouse click when the cursor is above the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mouse and finger touch are optionally used simultaneously.

[0208] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3 The touchpad 355 or Figure 6B When an input (e.g., a press input) is detected on the touch-sensitive surface 651 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays that enable direct interaction with user interface elements on the touchscreen display. Figure 1A The touch-sensitive display 112 or Figure 6AIn some implementations of a touchscreen, a touch detected on the touchscreen acts as a "focus selector," causing the specific user interface element to be adjusted based on the detected input when input (e.g., a press input via a touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display. In some implementations, focus moves from one area of ​​the user interface to another without corresponding cursor movement or touch movement on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these implementations, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, it is typically a user-controlled user interface element (or touch on the touchscreen display) that delivers the user's expected interaction with the user interface (e.g., by indicating to the device the element the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate that the user expects to activate that button (rather than other user interface elements shown on the device display). As used in this article, a “focus selector” can be a visible “indicator” or an invisible “indicator” for hovering over an input object (such as a finger or stylus).

[0209] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact on a tactile surface (e.g., finger contact or stylus contact), or to an alternative (proxy) to the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface near the contact, and / or the resistance and / or variation of the touch-sensitive surface near the contact may optionally be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement) has been exceeded. In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in pressure units) has been exceeded. Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that would otherwise not be easily accessible to the user on smaller devices with limited physical area for displaying power indications and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0210] In some embodiments, contact / motion module 130 and / or contact / motion module 430 use a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether a user has “clicked” an icon). In some embodiments, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen display can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some embodiments, the user of the device is provided with software settings for adjusting one or more of these intensity thresholds (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0211] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on a plurality of intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted away, before or after contact is detected to begin moving, before contact is detected to end, before or after contact intensity is detected to increase and / or before or after contact intensity is detected to decrease). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some embodiments, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding the second threshold results in a third action. In some embodiments, the comparison between the feature intensity and one or more intensity thresholds is used to determine whether to perform one or more actions (e.g., whether to select an option or abandon the action), rather than to determine whether to perform the first or second action.

[0212] In some embodiments, a portion of the gesture is identified to determine the characteristic intensity. For example, a touch-sensitive surface may receive a series of swipe contacts that transition from a starting position to an ending position (e.g., a drag gesture), where the intensity of the contact increases. In this example, the characteristic intensity of the contact at the ending position may be based only on a portion of the series of swipe contacts, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the ending position). In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe gesture before determining the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic intensity.

[0213] The user interface diagrams described below (e.g., Figures 7A to 7Q Optionally, various intensity maps are included, which show the contact on the touch-sensitive surface relative to one or more intensity thresholds (e.g., contact detection intensity threshold IT0, light press intensity threshold IT0, etc.). L Deep pressure intensity threshold IT D The current intensity of the light press intensity threshold (and / or one or more other intensity thresholds). This intensity graph is not typically part of the displayed user interface, but is provided to aid in interpreting the graph. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an action typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an action different from an action typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity lower than the light press intensity threshold (e.g., and higher than the nominal contact detection intensity threshold IT0, where contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector based on the movement of the contact on the touch-sensitive surface, without performing the action associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.

[0214] In some embodiments, the device's response to input detected by the device depends on a criterion based on the contact intensity during the input. For example, for some "light press" inputs, a first response is triggered by the intensity of contact exceeding a first intensity threshold during the input. In some embodiments, the device's response to input detected by the device depends on a criterion that includes both the contact intensity during the input and a time-based criterion. For example, for some "deep press" inputs, a second response is triggered by the intensity of contact exceeding the second intensity threshold (greater than the first light press threshold) during the input, provided a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold. The duration of this delay time is typically less than 200 ms (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, where the delay time increases as the second intensity threshold increases). This delay time helps prevent accidental deep press inputs. As another example, for some "deep press" inputs, there is a period of decreased sensitivity after the first intensity threshold is reached. During this period of decreased sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps prevent accidental deep press inputs. For other deep press inputs, the response to the detected deep press input does not depend on time-based criteria.

[0215] In some embodiments, one or more of the input intensity threshold and / or the corresponding output varies based on one or more factors, such as user settings, contact motion, input timing, application operation, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Exemplary factors are described in U.S. Patent Application Serials 14 / 399,606 and 14 / 624,296, the entire contents of which are incorporated herein by reference.

[0216] Contact characteristic strength from below the light press strength threshold IT L The intensity increases to within the light press intensity threshold IT L With deep pressure intensity threshold IT D The intensity of this contact is sometimes referred to as a "light press" input. The characteristic intensity of the contact ranges from below the deep press intensity threshold (IT). D The intensity increases to above the deep press intensity threshold IT D The intensity is sometimes referred to as "deep press" input. The contact feature intensity increases from below the contact detection intensity threshold IT0 to between the contact detection intensity threshold IT0 and the light press intensity threshold IT0. LThe strength between these strengths is sometimes referred to as the detection of a contact on the touch surface. The characteristic strength of a contact decreasing from above a contact detection strength threshold IT0 to below a contact detection strength threshold IT0 is sometimes referred to as the detection of a contact being lifted off the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some examples, shaded circles or ellipses are used to represent the strength of a contact on the touch-sensitive surface. In some examples, unshaded circles or ellipses are used to represent the corresponding contact on the touch-sensitive surface without specifying the strength of the corresponding contact.

[0217] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., performing the corresponding operation on the "downward stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing to above a press input intensity threshold and the intensity of the contact subsequently decreasing to below a press input intensity threshold, and the corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below a press input threshold (e.g., performing the corresponding operation on the "upward stroke" of the corresponding press input).

[0218] In some embodiments, the device employs intensity hysteresis to avoid unintended inputs, sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold in relation to a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units smaller than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact above a press input intensity threshold and a subsequent decrease in the intensity of that contact below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., performing the corresponding operation on the "up stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from an intensity equal to or below a hysteresis strength threshold to an intensity equal to or above a press input strength threshold and optionally the contact strength subsequently decreases to an intensity equal to or below the hysteresis strength, and a corresponding operation is performed in response to the detection of a press input (e.g., depending on the environment, the contact strength increases or decreases).

[0219] For ease of explanation, the description of an operation triggered optionally in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is provided: the contact strength increases to above the press input strength threshold; the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold; the contact strength decreases to below the press input strength threshold; or the contact strength decreases to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example where the operation is described as being performed in response to detecting a decrease in contact strength to below the press input strength threshold, the operation is optionally performed in response to detecting a decrease in contact strength to below a hysteresis strength threshold corresponding to and less than the press input strength threshold. As described above, in some embodiments, the triggering of these operations also depends on meeting a time-based criterion (e.g., a delay time has elapsed between meeting a first strength threshold and meeting a second strength threshold).

[0220] User interface and related processes

[0221] Now turn attention to embodiments of user interfaces (“UIs”) and associated processes that can be implemented on electronic devices such as portable multifunction devices 100 or 300, which have displays, touch-sensitive surfaces, and (optionally) one or more sensors for detecting the intensity of contact with the touch-sensitive surfaces.

[0222] Figures 7A to 7Q Exemplary user interfaces are illustrated according to some embodiments for interacting with user interface objects via proximity-based and contact-based input of the input object. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 11A to 11D , Figures 12A to 12C and Figures 13A to 13E The process in.

[0223] Figures 7A to 7K The process is illustrated whereby an input object (e.g., a finger or stylus) moves its cursor and positions itself within selectable text (e.g., editable or non-editable text) when it moves laterally in a hover state, and where the input object selects text (e.g., expands the text selection from the current cursor position) when it moves laterally in a touch state.

[0224] exist Figure 7AIn this context, a user interface (e.g., user interface 730) is displayed on the touchscreen 112. User interface 730 includes optional text 732. When the input object, shown in these figures as a finger or hand 734 (but alternatively, it could be a stylus or other object whose proximity and contact can be detected by the touchscreen 112), is above a threshold hovering distance from the touchscreen 112 (e.g., distance 514 is greater than the threshold hovering distance), no indicator corresponding to that input object is displayed in user interface 730. In this case, the contact strength 655 is zero because the input object has not yet made contact with the touchscreen 112.

[0225] Figure 7B The diagram illustrates that when an input object (e.g., a finger or hand 734) is within a threshold hover distance from the touchscreen 112, an indicator (e.g., a cursor 738) is displayed within optional text 732 at position 701 on the touchscreen 112 (e.g., at a position corresponding to (x,y) position 504 on the touchscreen 112). The indicator displayed within the optional text 732 is a result of providing a first hover event (e.g., a "hover start" event, as described below) to the application. In some embodiments, the first hover event is a normalized touch event with a first hover stage value. Position 701 may be the same as or offset from the lateral (x,y) position 504 of the finger 734, and optionally the offset is determined based on the hover distance of the finger 734 (e.g., represented by distance 514) and / or positional state (e.g., tilt). In some embodiments, in addition to the cursor 738, the device 100 also displays a magnifying glass (e.g., a magnifying glass 736) above the optional text 732. A magnifying glass 736 is displayed above the cursor 738 in the user interface 730 and moves with the cursor 738 within the user interface 730. Within the magnifying glass 736, a magnified version of a portion of the optional text 732 is displayed, and a copy 742 of the cursor 738 is also displayed at the position within the magnified text, depending on the position of the cursor 738 within the optional text 732.

[0226] Figure 7C This shows that when finger 734 is held within the hovering range relative to... Figure 7B When the cursor 738 moves laterally to the indicated position, it moves within the optional text 732 according to the lateral movement of the finger 734. The movement of the cursor 738 is a result of a second hover event provided to the application (e.g., a "hover change" input event, i.e., a hover-based touch event with a "hover change" phase, as described below). In some embodiments, the second hover event is a normalized touch event with a second hover phase value. In embodiments including a magnifying glass 736, the magnifying glass 736 moves with the cursor 738 and displays the updated portion of the text 732 and the updated position of the cursor 738. Figure 7CIt is also shown that the hover distance does not need to remain constant during the movement of finger 734 in order to move cursor 738 into optional text 732. However, in some embodiments, the "hover change" input event indicates a change in the proximity (e.g., hover distance) and position of finger 734.

[0227] Figure 7D It shows when finger 734 is relative to Figure 7B When the cursor 738 moves laterally to another location on the touchscreen 112, the cursor 738 moves to a new location within the optional text 732. A touch press of finger 734 is detected on the touchscreen 112 (e.g., the contact strength 655 between finger 734 and touchscreen 112 is higher than the contact detection threshold IT0). Information about the touch press is provided to the application as a first contact event. In some embodiments, the first contact event is a normalized touch event having a first contact phase value (e.g., a touch start phase value). Before finger 734 moves laterally while in contact with touchscreen 112, cursor 738 is displayed at its current location within the optional text 732. In embodiments including a magnifying glass 736, magnifying glass 736 is displayed above cursor 738. A magnified version of the text surrounding a copy 742 of cursor 738 is displayed within magnifying glass 736, and the position of cursor 738 within the optional text 732 is represented by the copy 742 of cursor 738 in the magnifying glass.

[0228] Figure 7E The diagram illustrates a lateral movement of finger 734 while it remains in contact with touchscreen 112 after contact with the touchscreen 112. This movement causes cursor 738 to move within selectable text 732, and the movement of cursor 738 from its previous position (e.g., the cursor position when finger 734 touches and presses on touchscreen 112) to its current position results in the selection of a portion of the text between the previous and current positions (e.g., as shown in selection 740). The movement of cursor 738 is a result of providing a second contact event (e.g., a "touch change" input event, i.e., a contact-based touch event with a "touch change" phase, as described below) to the application. In some embodiments, the second contact event is a normalized touch event with a second contact phase value.

[0229] As the movement of cursor 738 continues with the movement of finger 734 in contact with touchscreen 112, selection 740 expands or contracts according to the movement of cursor 738 (e.g., provided to the application as an additional "touch change" input event). Figure 7E As shown, selection 740 is also indicated within magnifying glass 736 (for example, as selection 744).

[0230] Figure 7FThis demonstrates that after contact is made between a moving finger 734 and the touchscreen 112, and the selection 740 is adjusted via the moving cursor 738, the lifting of the finger 734 is detected (e.g., ...). Figure 7F (As shown in the figure, distance 514 is greater than zero and contact strength 655 is equal to zero). In some embodiments, information about lift-off is provided to the application as a third hover event. In some embodiments, the third hover event is a normalized touch event with a second hover phase value (e.g., a "hover change" phase value).

[0231] After being lifted from touchscreen 112, finger 734 hovers over touchscreen 112 again. While finger 734 hovers over touchscreen 112, cursor 738 detaches from selection 740, and moves within selectable text 732 as finger 734 remains within the hovering area above touchscreen 112, as finger 734 moves laterally. (As...) Figure 7G As shown, in some embodiments, after the contact between finger 734 and touchscreen 112 is detected to be lifted, selection 740 is maintained in text 732. A magnifying glass continues to move with cursor 738 and displays a magnified version of the text surrounding cursor 738, a copy 742 of cursor 738.

[0232] Figures 7H to 7I This shows that when a touch press of finger 734 is detected again at a new location, deselection 740 is performed. Figure 7H Selection 740 is no longer shown. The cursor 738 is displayed at a new position (e.g., position 701), which corresponds to the touch-press position of finger 734 on touchscreen 112 (e.g., (x,y) position 504), as shown below. Figure 7H As shown. In Figure 7I In the process, lateral movement of finger 734 on touchscreen 112 is detected, and cursor 738 moves from its previous position (e.g., Figure 7H When the middle finger 734 presses a second touch, the movement of the cursor 738 from its previous position to its current position causes a portion of the text between the cursor 738's previous and current positions to be selected (e.g., as shown in selection 746). Magnifying glass 736 moves with the cursor 738 and displays the current text selection (e.g., represented by a magnified copy 748 of selection 746).

[0233] Figure 7J This illustrates another lift-off from touchscreen 112, during which finger 734 again hovers above touchscreen 112. While finger 734 hovers above touchscreen 112, cursor 738 detaches from selection 746, and moves within selectable text 732 as finger 734 remains within the hovering range above touchscreen 112, as finger 734 moves laterally. (As shown...) Figure 7JAs shown, in some embodiments, after the contact between finger 734 and touchscreen 112 is detected to be lifted, selection 746 is maintained in text 732. The magnifying glass continues to move with cursor 738 and displays a magnified version of the text around cursor 738, copy 742.

[0234] Figure 7K It is shown that when finger 734 is lifted outside the hover range (e.g., beyond a threshold hover distance above touchscreen 112), magnifier 736 is replaced by menu 750 (e.g., a "Cut / Copy / Find" menu), and selection 746 changes appearance (e.g., replaced by selection object 752 with adjustable boundaries). Information about finger 734 being lifted outside the hover range is provided to the application as a fourth hover event. In some embodiments, the fourth hover event is a normalized touch event with a third hover phase value (e.g., a hover end phase value).

[0235] Figures 7L to 7P It shows the relationship with Figures 7A to 7I Examples of the same application are shown, but the application runs on a non-native device (e.g., a device with an input device separate from the display). In such embodiments, hover-based input events and touch-based input events are analog touch events, or are converted to analog touch events, as described elsewhere in this document.

[0236] For this purpose, a user interface (e.g., user interface 730) is displayed on the display 702 of the electronic device. User interface 730 is for... Figures 7A to 7K The user interface of the application shown is illustrated. However, display 702 is not a touch-sensitive display. Instead, the electronics are coupled to a mouse 704, which controls the position and movement of a pointer 706 displayed on display 702. Therefore, in Figures 7L to 7P In this context, the application is not native to the operating system framework of the computer system running the application. Instead of directly interpreting mouse events, the application receives simulated touch events converted from mouse events.

[0237] The display 702 also displays the user interface of the second application 708, which is different from the user interface 730 of the application (for example, the application and the second application are different applications).

[0238] Figure 7L User input 710 is shown, indicating the movement of mouse 704. In response to the movement of mouse 704, such as... Figure 7MAs shown, pointer 706 moves above the application's user interface 730. When pointer 706 is above the application's user interface 730, the device delivers simulated hover events to the application without actuating a mouse button. These hover events have similar phases and the resulting user interface functionality, as referenced above. Figures 7A to 7K The topic under discussion. For example, in response to a hover start event, such as... Figure 7M As shown (for example, when pointer 706 initially enters user interface 730), pointer 706 is replaced by cursor 738, which is similar to the cursor that responds to a reference. Figures 7A to 7K The cursor 738 is placed for the aforementioned hover event. However, because the display 702 may be larger than the touchscreen on a portable multifunction device, in some embodiments, the magnifying glass 736 is not displayed. Figure 7B ).

[0239] Figure 7N User input 714 is shown that actuates a predefined mouse button 716 (sometimes referred to herein as the left mouse button 716). (e.g., user input 714 is a left mouse click and hold). It should be noted that in some cases, the input device is not a mouse, but a touchpad or other touch-sensitive surface that provides indirect manipulation of the pointer / cursor. For example, in some cases, instead of user input 714, user input is a downward press on the touchpad (e.g., a press that may press the touchpad or have an intensity higher than a predefined threshold corresponding to a "click"). In response to the left mouse click and hold 714, the device initially delivers a simulated contact touch event to the application (e.g., a touch start input event, i.e., a contact-based touch event with a "touch start" phase). The touch start input event is handled by the application in the same manner as a physical contact when the application is running on a touchscreen device (e.g., as...). Figure 7D (As shown). That is, the device begins to highlight the selected text. For example, when a left mouse click is received and held at 714, the appearance of the cursor 738 at its initial position is changed to indicate the start of text selection (e.g., by displaying concentric circles at the top of the cursor 738).

[0240] like Figures 7N to 7O As shown, as long as the simulated touch is active (e.g., as long as the left mouse button 716 is pressed), the application responds to the movement 718 of the mouse 704 to select text. Note that when the user inputs 714 actuates the left mouse button 716, the movement 718 of the mouse 704 causes a "touch change" input event to be delivered to the appropriate view (e.g., the view corresponding to the position of pointer 738 at the start when the left mouse button is clicked and held).

[0241] Figure 7P This shows that the user has released the left mouse button and held it at 714 (in). Figures 7N to 7O (As shown in the diagram). In response, the device begins to provide hover-based input events and restores the display pointer 706. For example, the initial event after releasing the left mouse button and holding 714 is a hover change event.

[0242] It should be noted that in some embodiments, non-native applications do not need to be able to interpret hover-based touch events. In some embodiments, the host / event converter 808 ( Figure 8 As described above, hover-based touch events are provided to all applications it is serving (e.g., simulated events). Applications that have not been developed to handle hover-based touch events may ignore them.

[0243] Figure 7Q The provision of touch input events is shown when finger 734 moves from one view (e.g., user interface 782 of application A) to a different second view (e.g., user interface 784 of application B).

[0244] In some embodiments, when finger 734 is located at position 788-a (where the position is not in contact with touchscreen 112 and is not within a threshold hover distance (e.g., not close to touchscreen 112)), the device does not provide touch events to any view.

[0245] In some embodiments, when the device initially detects the presence of finger 734 within a threshold hover distance at location 788-b, the device provides a first touch input event with a hover phase value indicating user input that approaches but does not touch touchscreen 112 (e.g., a "hover start" phase value indicating the start of input that approaches but does not touch touchscreen 112). In some embodiments, the first input event is provided to a view corresponding to location 788-b (e.g., an application user interface) (e.g., the hover event is sent to the view based on the current location of the user input; in this example, the view would be view 782 of application A). It should be noted that in some embodiments, hover-based touch events are delivered to any view above the cursor / finger, while (as further described below) touch events are sent to the view corresponding to the "touch start" location (e.g., the location where the initial contact occurred), even if the cursor / finger moves above a different view.

[0246] In some embodiments, when the finger 734 is within a threshold hovering distance and the device detects a change in the position of the finger 734 (e.g., proximity or position, such as from position 788-b to position 788-c), the device provides a second touch input event with a hover phase value indicating user input that approaches but does not touch the touchscreen 112 (e.g., a "hover change" phase value indicating an update to the proximity or position of a previous hover touch event). In some embodiments, the second input event is provided to a view corresponding to position 788-c (e.g., an application user interface).

[0247] In some embodiments, when the device detects that a finger 734 is in contact with the touchscreen 112 at position 788-d, the device provides a third touch input event with a contact phase value indicating user input (e.g., a "touch start" phase value) in relation to the touchscreen 112. In some embodiments, the third input event is provided to a view corresponding to position 788-d (e.g., an application user interface) (e.g., a "touch start" input event is provided to a view corresponding to the initial position of the contact).

[0248] In some embodiments, when the device detects that the finger 734 has moved to position 788-e from contact with the touchscreen 112, the device provides a fourth touch input event with a contact phase value indicating user input that has contacted the touchscreen 112 (e.g., a "touch change" phase value indicating an update to the position of a previous touch event). In some embodiments, the fourth touch input event is provided to a view (e.g., an application user interface) corresponding to position 788-d instead of position 788-e (e.g., a "touch change" input event is provided to a view corresponding to the position of the initial contact, e.g., a "touch start" event).

[0249] In some embodiments, when the device detects the lifting of the finger 734 from the touchscreen 112 at a position 788-f of the finger 734 within a threshold hover distance, the device provides a fifth touch input event. In some embodiments, the fifth touch input event has a "touch end" phase value, and optionally, the fifth touch input event is provided to the same view as a previous contact-based touch event. In some other embodiments, the fifth touch input has a "hover change" phase value, and optionally, the fifth touch input event is provided to a view corresponding to position 788-f (e.g., an application user interface) (e.g., the fifth touch input event is provided to a view corresponding to the position of the hover event, as with other hover events). In some embodiments, when the position 788-f of finger 734 is within a threshold hover distance, the device detects a lift-off of the finger 734 from the touchscreen, and provides two touch input events: A) a touch input event with a "touch end" phase value, which is provided to the same view as the previous contact-based touch event, in this example, the user interface view 782 of application A; and B) another touch input event with a "hover change" phase value, which is provided to the view corresponding to position 788-f (e.g., the application user interface), in this example, the user interface view 784 of application B.

[0250] In some embodiments, when the device initially detects that finger 734 is no longer within a threshold hover distance at position 788-g, the device provides a sixth touch input event with a hover phase value indicating the end of hover input (e.g., a "hover end" phase value). In some embodiments, the sixth input event is provided to the view corresponding to position 788-g (e.g., an application user interface) (e.g., sending the hover event to the view based on the current position of the user input). In some embodiments, a "hover end" input event (e.g., a hover-based touch event with a "hover end" phase value) is sent to the view to which the preceding event was sent (e.g., to notify those views that user input has ended).

[0251] Figure 8 This is an exemplary device according to some embodiments for having a display and one or more input devices 802 separate from the display (e.g., Figure 3 A diagram of the input / output processing stack 800 of device 300. It should be understood that the division between layers of the input / output processing stack is usually arbitrary and is presented only for illustrative purposes.

[0252] The input / output processing stack 800 allows some applications to execute on devices where these applications are not native. Specifically, the input / output processing stack 800 allows applications 812 native to the touchscreen operating system framework to execute on an operating system framework that depends on or utilizes input events from peripheral input devices that are not touchscreens.

[0253] For this purpose, one or more input devices 802 are provided at the base layer of the input / output processing stack 800. The input device 802 is separate from the display (e.g., not a touchscreen). In some embodiments, the input device 802 includes one or more of a mouse, a scroll wheel (e.g., a component of a mouse), a keyboard, and / or a touch-sensitive surface separate from the display (e.g., a stand-alone touchpad).

[0254] A driver or a group of drivers receives and processes input data 801 from input device 802. An operating system 804, communicating with the drivers, processes the raw input data received from these drivers. In some embodiments, the drivers may be considered part of the operating system 804.

[0255] A first set of application programming interfaces (APIs) 806 (e.g., associated with an operating system) communicates with the operating system 804. In some embodiments, the first set of APIs 806 is also considered part of the operating system 804. The first set of APIs 806 includes a set of APIs designed for use by applications developed for running on the device. For example, the first set of APIs 806 includes APIs for providing input events 805 from input device 802 to applications running on the device (e.g., word processing applications, web browsers).

[0256] The input / output processing stack 800 includes a host / event converter 808. In some embodiments, the host / event converter 808 is an application running at the same layer of the input / output stack 800 as other native applications available to the user (e.g., word processing applications, web browsers). However, unlike most other applications that interact with the first set of APIs 806, the host event translator 808 acts as a service that bridges the gap between the operating system 804 and the application 812, which is not native to the operating system 804 (e.g., applications developed for different operating systems, such as different input devices, such as touchscreens). To this end, the host / event converter receives input events 805 provided by the first set of APIs 806 from the input device 802 and, in some embodiments, translates the input events into simulated touch events 809 that the application 812 can interpret. A second set of APIs 810 makes the simulated touch events 809 available to the application 812 (e.g., the second set of APIs 810 includes APIs similar to those that the application 812 expects to have available in its native framework).

[0257] Figures 9A to 9B Exemplary data structures for providing touch input events (including hover-based and touch-based input events) to an application are shown according to some embodiments.

[0258] Figure 9A An exemplary data structure for a “touch event” 902 (e.g., an event having a touch or hover phase) according to some embodiments is shown. Figure 9B Exemplary data structures for "scroll events" according to some embodiments are shown. In some cases, touch-based input events 809 are generated by user input not received from the touchscreen (e.g., touch-based input events are simulated touch-based input events). However, more generally, touch-based input events 809 are applicable to both native operating system frameworks (e.g., on devices with touchscreens and operating systems designed to run on devices with touchscreens) and non-native operating system frameworks (e.g., on devices with peripheral input devices separate from the display). In the latter case, in some embodiments, touch-based input events 809 are generated by a host / event converter (e.g., Figure 8 The host / event converter 808 is provided.

[0259] For this purpose, touch event 902 is an exemplary data structure for touch-based input event 809, which includes touch (e.g., as used herein, the term "touch" includes contact with a touch-sensitive display as well as input that is near but does not touch the touch-sensitive display).

[0260] Touch event 902 includes touch identifier 904. This is similar to the operating system (e.g., ...). Figure 8 As determined by the operating system 804, for all input events that are part of the same user input (e.g., a gesture), the value of the touch identifier 904 is shared (e.g., the same).

[0261] In some embodiments, touch event 902 includes a list of one or more locations 906 of one or more touches on a touch-sensitive surface. In some embodiments, these touches are analog touches, and the touch-sensitive surface is a virtual touch-sensitive surface (e.g., the touch-sensitive surface does not physically exist, but the device interprets user input events from peripheral devices separate from the display as touch inputs on the display, even though the display is not a touchscreen). In some embodiments, each touch (analog or non-analog) has a corresponding location 906 in the location list. In some embodiments, location 906 includes a two-dimensional (e.g., (x, y)) position corresponding to coordinates on the touch-sensitive surface. In some embodiments, location 906 includes three-dimensional (e.g., (x, y, z)) coordinates, where the third (e.g., "z") coordinate represents the distance relative to the touch-sensitive surface (e.g., for a hover phase) (e.g., above it, or in contact with it if zero or other default values).

[0262] Touch event 902 includes a list of stage values ​​908 for one or more touches on a touch-sensitive surface. In some embodiments, each touch (simulated or non-simulated) has a corresponding stage value 908 in the stage value list.

[0263] Each stage value 908 indicates a specific stage of the corresponding touch input, sometimes referred to as the current stage of the touch input. The stage field of touch event 902 can take various predefined values, such as: a "touch start" stage value, which indicates that the defined event is the initial contact with the touch-sensitive surface; a "touch change" stage value, which indicates that the defined event is an update of a previous touch event, whose position has moved from the position defined by the previous touch event; a "touch end" stage value, which indicates that the contact with the touch-sensitive surface has ended (e.g., the user has lifted his / her finger from the touch-sensitive surface), or other stage values.

[0264] Furthermore, in some embodiments, these stage values ​​include one or more hover stage values, indicating the characteristics of inputs that approach but do not touch the touch-sensitive surface.

[0265] In some embodiments, the one or more hover phase values ​​include a "hover start" phase value, which indicates that the defined event is the initial detection of an input that approaches but does not touch the touch-sensitive surface (e.g., indicating that a finger or stylus has moved within a threshold distance of the touch-sensitive surface). As described below, when the touch is an analog touch, other user interface events can trigger a transition to a touch event 902 with a touch start phase (e.g., ...). Figure 8 In the host / event converter 808, such as the mouse pointer moving on the corresponding view.

[0266] In some embodiments, the one or more hover phases include a "hover change" phase that indicates a change in proximity to the touch-sensitive surface or a change in position above the touch-sensitive surface (e.g., where the position is as described above).

[0267] In some embodiments, the one or more hovering phases include a "hover end" phase, which indicates that no further input near or touching the touch-sensitive surface is detected.

[0268] Furthermore, for simulated touch input events, touch event 902 optionally includes information from or referencing one or more local input events 801 / 805 (e.g., input events from peripheral input devices) from which the simulated touch input event is generated. For example, touch event 902 optionally includes a sub-event indicating a modifier (e.g., a pressed keyboard button) that is active when input is received.

[0269] It should be noted that Figures 9A to 9B The reference numerals for sub-events 801 / 805 are intended only to indicate that different sub-events may originate from different levels of the stack. In some embodiments, different sub-events may originate from the same level of the stack. Furthermore, the stack (e.g., Figure 8 The input / output processing stack (800) in the middle can utilize the lack of a reference. Figure 8 The additional event structure described.

[0270] The following paragraphs provide several specific examples of input events converted from input devices separate from the display (e.g., analog), along with their resulting data structures and sub-events. Thus, touchscreen-based applications can process analog touch events as if these touch events were received from the touchscreen.

[0271] Convert to simulate a left mouse click event

[0272] In some embodiments, actuation of a predefined button (e.g., a physical button) on a peripheral device separate from the display is converted into a touch event with a contact phase (e.g., indication of contact with the display). The touch event includes positional information corresponding to the position of the mouse pointer on the display. For example, a left mouse click (e.g., on a physical mouse separate from the display) is converted into a touch event corresponding to a finger or stylus contacting the display (even though the display is not actually touch-sensitive). When operating system 804 ( Figure 8 When the driver in the touchscreen receives a left mouse button click, the operating system generates a digitizer / pointer event 805, which has (x,y) coordinates based on the position of the mouse pointer at the time the left mouse button click is received. Here, the term "digitizer" refers to a software analogue of a physical digitizer, which is a component of the touchscreen. The digitizer / pointer event 805 includes a sub-event 801 that provides contextual information from which touch identifier 904 can be determined, and a sub-event that includes information about which modifiers (e.g., keyboard buttons, if any) are active at the time of the left mouse button click. The host / event converter translates the digitizer / pointer event 805 into a touch event with a "touch start" phase. The touch event includes a digitizer event as a sub-event (which itself may include additional sub-events).

[0273] Input events that move the displayed pointer when a predefined button is actuated are converted into touch events with a "touch change" phase. In some embodiments, input events with a "touch change" phase are provided to the same view that received the corresponding initial "touch start" input event. Thus, a user can click the mouse and drag the cursor out of the view, and the input event is still delivered to the view.

[0274] Convert mouse movement to simulate touch-hover events

[0275] In some embodiments, input events involving movement of the displayed pointer are converted into touch events with a hover phase (e.g., when the predefined touch input button described above is not actuated, in which case movement (e.g., mouse movement) is converted into touch movement). As mentioned above, the hover phase indicates input that approaches but does not touch the display (although in fact the display is not sensitive to input that approaches but does not touch the display). The coordinates of touch events converted with a hover phase are processed in a similar manner to the above-described "converting to a simulated touch touch event (left mouse click)".

[0276] For example, in some embodiments, mouse movement of the pointer on the display is translated into a touch event with a hover phase (e.g., a "hover event," also referred to as a "touch-hover event," to indicate that in some embodiments, the hover event is delivered using the same normalized data structure used for touch events). Whether the hover phase is a "hover start" phase or a "hover change" phase depends on whether the hover event is the initial hover event delivered to the corresponding view. For example, when the mouse moves the pointer over the view (e.g., without pressing a mouse button), the initial event provided to the view is a hover event with a "hover start" phase. Subsequent mouse movement within the corresponding view is delivered to the corresponding view as a "hover change" event.

[0277] In some embodiments, hover events are delivered to the view corresponding to its current position (e.g., the current position of the pointer on the display). Therefore, as the mouse pointer moves from one view to a different view, hover events with positions within the different views are provided to those views.

[0278] Convert to simulated multi-touch input events for zoom and scale input

[0279] In some embodiments, one or more user inputs on a device separate from the display are converted into simulated multi-touch input events (e.g., indicating multiple simulated touches). In some embodiments, a pointer object is displayed above a corresponding view on the display, and multiple simulated touches are simulated to have a centroid above the corresponding view (e.g., at the location of the pointer object on the display).

[0280] For example, in some embodiments, when the “control” button is pressed on a separate keyboard, user input including actuation of the mouse wheel is converted into a multi-touch pinch gesture (e.g., a pinch-to-zoom gesture) with a centroid at the pointer position on the display.

[0281] For example, in some embodiments, multi-touch gestures received on a touch-sensitive surface (e.g., a touchpad) separate from the display are converted into simulated multi-touch gestures on the display. For instance, multi-touch pinch gestures (e.g., pinch-to-zoom gestures) received on a touch-sensitive surface separate from the display are converted into simulated pinch gestures on the display. Similarly, multi-touch rotation gestures (e.g., two-finger twist gestures) on a touch-sensitive surface separate from the display are converted into simulated multi-touch rotation gestures on the display. In some cases, the individual touches of the simulated multi-touch gesture have different characteristics from the actual touches received on the touch-sensitive surface separate from the display (e.g., different locations from the actual touches, but with the same centroid, located at the midpoint or centroid of both the actual touches and the individual touches of the simulated multi-touch gesture). In some embodiments, such as Figure 9AAs shown, simulated touch events include references to actual touch events (e.g., "finger") (e.g., sub-events).

[0282] Convert scroll events to simulate touch-based scroll events

[0283] Figure 9B An exemplary data structure for a touch-based “scroll event” 910 according to some embodiments is shown. Scroll event 910 is similar to touch event 902, but with the following differences. Scroll event 910 is thus identified by a “scroll” type 912. Scroll event 910 optionally includes a phase (e.g., for when a scroll event is received from a touch-sensitive surface). In some embodiments, touch-based scroll events do not include a position (e.g., (x,y) field), so a transformed (e.g., simulated) scroll event includes a pointer sub-event 801 indicating the position of the pointer on the display, thereby allowing the scroll event to be routed to the correct view. Scroll events typically include one or more incremental position fields instead of position fields (e.g., a value “dx” representing the amount of scrolling in a first direction and a value “dy” representing the amount of scrolling in a second direction perpendicular to the first direction).

[0284] In some embodiments, multiple scroll events from the scroll wheel are converted into simulated touch-based scroll events 910. In some embodiments, the simulated touch-based scroll events correspond to simulated single touch movement on a touch-sensitive display. This is a layer below the input / output processing stack 800 of application 812. Figure 8 Determine which scroll events from the scroll wheel are part of a single user input.

[0285] Figure 10 This is an illustration of an input / output processing stack 1000 that, according to some embodiments, merges (e.g., demultiplexes) input events from independent event streams and then provides the merged user input events to an application.

[0286] Input / output processing stack 1000 is similar to input / output processing stack 800. Figure 8 However, there are the following differences.

[0287] Operating system 804 generates multiple independent event streams based on user input received from one or more input devices 802 coupled to the device. In some embodiments, operating system 804 generates multiple independent event streams based on user input received from a single input device 802-a. For example, input device 802-a may be a mouse that receives movement of the displayed pointer / cursor (e.g., ...). Figures 7L to 7P The mouse includes user input regarding the position of the pointer 706 / cursor 738. The mouse also includes an active (e.g., pressable) mouse button. Figure 7NThe operating system 804 generates a first event stream based on user movement of the mouse (e.g., moving the pointer / cursor) and a separate second event stream based on the actuation of the mouse button.

[0288] In some embodiments, the device is coupled to a plurality of input devices (e.g., input devices 802-a and 802-b). For example, optional input device 802-b is a keyboard coupled to the device, and operating system 804 generates an event stream corresponding to the actuation of each key on the keyboard.

[0289] In some embodiments, the operating system 804 multiplexes independent event streams into multiplexed event streams 1002 (e.g., event queues). For example, typically via a first set of APIs 806 ( Figure 8 The program provides input event 1005 from the event queue to the local application. Events 1005-a and 1005-c are events from a first event stream (e.g., labeled type I), and events 1005-b and 1005-d are events from a second independent event stream (e.g., labeled type II).

[0290] In some embodiments, the host / event converter 808 demultiplexes the multiplexed event stream (e.g., merges information from multiple independent event streams to generate a merged input event stream). For example, information corresponding to the position of the displayed pointer (e.g., determined from mouse movement input) is merged with information corresponding to the actuation of an active mouse button to create a merged event 1007. In some embodiments, the information corresponding to the position of the displayed pointer is located in a sub-event pointing back to a first multiplexed event (e.g., event 1005-a is a sub-event of event 1007). In some embodiments, the information corresponding to the actuation of an active mouse button is located in a sub-event pointing back to a second multiplexed event (e.g., event 1005-b is also a sub-event of event 1007).

[0291] In some embodiments, the merged event 1007 is a simulated touch input event (e.g., with reference). Figure 9A The described event 809 is an input event of a structured nature (e.g., a simulated touch-based scrolling event, with reference to...). Figure 9B The structure of the described event 809 is an input event. Alternatively, in some embodiments, the merged event is then converted into a simulated touch input event or a simulated touch-based scrolling event (e.g., another processing layer exists in the input / output stack).

[0292] Figures 11A to 11D A flowchart of a method 1100 for providing simulated touch events to an application, according to some embodiments, is shown. Method 1100 is used in electronic devices having a display (e.g., Figure 3The method is performed at device 300. The electronic device includes an input device separate from or coupled to the display. In some embodiments, the input device separate from the display is a touch-sensitive surface. Other examples of input devices separate from the display have been discussed above. In some embodiments, the electronic device includes one or more sensors to detect the proximity of an input object (e.g., a finger or stylus) above the touch-sensitive surface (e.g., a proximity sensor (such as an infrared sensor), a capacitive sensor in the touch-sensitive surface, or a camera adjacent to the touch-sensitive surface), and / or one or more sensors to detect the contact strength between the input object and the touch-sensitive surface. In some embodiments, the input device separate from the display is not a touch-sensitive surface (e.g., although the device may optionally include or be coupled to a touch-sensitive surface). Some operations in method 1100 may optionally be combined, and / or the order of some operations may optionally be changed. For simplicity, method 1100 is described with reference to device 300.

[0293] Portable multi-functional devices (e.g., Figures 1A to 1B Device 100) and laptop computers (e.g., Figure 3 Devices 300 in this context typically use entirely different input devices. For example, portable multifunction devices (e.g., smartphones or tablets) may include touchscreens, while laptops may include physical keyboards, mice, and / or touch-sensitive surfaces separate from the display. The result is that the device's operating system (e.g., ...) Figure 8 The first set of APIs (806) provides input events to device applications that are highly dependent on the type of input devices included in the device. Therefore, software applications are typically developed for specific types of devices (e.g., portable multifunction devices with touchscreens) and are not easily ported to different operating systems. Instead, application developers must rewrite the software for different operating systems running on different types of devices.

[0294] Method 1100 allows a first application developed for a first operating system framework (e.g., a touchscreen framework) to interpret input events from a second operating system framework (e.g., a framework in which user input is received at an input device separate from the display). In some embodiments, method 1100 converts user input from one or more input devices separate from the display into simulated touch events that the first application can interpret. In some embodiments, this conversion occurs at a second application native to the second operating system framework. The second application is an input event converter that acts as the host of the first application. Thus, the second application acts as an input / output processing stack (e.g., ...). Figure 8The host / event converter 808 and stack 800 are added as layers. Therefore, method 1100 solves the problem of how to transmit different types of user input events to applications that expect (e.g., are configured to receive) touchscreen input, which increases cross-platform compatibility for applications configured to receive input from hardware different from the hardware (e.g., input devices) of the electronic device on which the application is actually executed.

[0295] Device 300 simultaneously displays (1102) a user interface on a display, the user interface including a representation of a first application, the first application including multiple user interface objects and pointer objects (e.g., a displayed pointer or cursor). For example, Figure 7L A user interface 730 is shown for displaying optional text 732 and several buttons (e.g., "Memo", "Done"). Furthermore, in Figure 7L The pointer is 706.

[0296] In some embodiments, the input device is (1104) a mouse (e.g., Figure 3 Keyboard / mouse 350; Figures 7L to 7P Mouse 704 in some embodiments). In some embodiments, the input device is a scroll wheel (e.g., as a mouse). Figure 3 Keyboard / mouse 350 or Figures 7L to 7P (The scroll wheel is part of the mouse 704). In some embodiments, the input device is a keyboard (e.g., keyboard / mouse 350, Figure 3 In some embodiments, the input device is a touchpad separate from the display (e.g., Figure 6B (The touch-sensitive surface 651 in the middle). Converting user input events from any of these devices into analog touch input increases the cross-platform compatibility of touchscreen-based applications with different hardware systems.

[0297] While simultaneously displaying a representation and a pointer object of the first application, device 300 receives (1106) a user input event at an input device separate from the display, pointing to a corresponding location (e.g., a location in view coordinate space) of the pointer object displayed in the user interface. In some embodiments, stored state information is used to determine the corresponding user interface location; for mouse and keyboard input events, the stored location information is updated as the mouse moves or certain keys are pressed, and for touchpad input, the state information includes the origin or offset and the scale.

[0298] In response to the detection of a user input event (1108), and based on determining the corresponding position of the pointer object displayed in the user interface above the representation of the first application, the device 300 performs operations 1110 to 1126 or a subset thereof.

[0299] In some embodiments, device 300 generates (1110) a digitized event based on user input events, the digitized event including the position of a pointer object displayed in the user interface, and converts the digitized event into an analog touch event. In some embodiments, the generation of the digitized event is performed by a corresponding layer of the input / output processing stack (e.g., by a layer including...). Figure 8 The first user input processing operation performed by the driver in the 804 operating system.

[0300] In some embodiments, a first user input processing operation generates a digitized event configured for use by an application configured to execute within a first operating system framework (e.g., the operating system framework of a laptop computer), and an analog touch event is configured for use by an application configured to execute within a second operating system framework (e.g., the operating system framework of a smartphone or tablet computer) that is different from the first operating system framework. Converting digitized events from the first operating system framework to analog touch events from the second operating system framework increases application cross-compatibility.

[0301] In some embodiments, the digitization event includes (1112) a pointer event that is paired with a corresponding position of a pointer object displayed in the user interface or has position information corresponding to that position. For example, in some embodiments, the pointer event (e.g., typically accessible via...) Figure 8 The first set of pointer events obtained from API 806 are sub-events of the digitization event. In some embodiments, the digitization event is generated by a pointer event (1114) that is paired with a corresponding position of the pointer object displayed in the user interface or has position information corresponding to that position (e.g., the pointer event is not a sub-event of the digitization event, but is used to determine the value of the position field in the digitization event). In some embodiments, generating the digitization event includes associating the corresponding position of the pointer object displayed in the user interface with the data structure of the digitization event.

[0302] Device 300 provides (1116) information describing simulated touch events to a first application, including information describing the location of one or more simulated touches in the user interface (e.g., via...). Figure 8 The second set of APIs (810).

[0303] In some embodiments, the one or more simulated touches have a centroid (1118) at the corresponding location of the pointer object displayed in the user interface. For example, in some embodiments, as described above, input on an input device separate from the display is converted into a multi-touch gesture (pinch-to-zoom or rotate gesture) for the touchscreen. In some embodiments, input events from multi-touch gestures on a touch-sensitive surface separate from the display are converted into multi-touch gestures for the touchscreen. In either case, the simulated touch of the multi-touch gesture is such that the centroid is located at the location of the displayed pointer object. Generating simulated multi-touch gestures with a centroid at the location of the pointer object allows user input events from non-touchscreen input devices to be used in touchscreen-based applications, thereby increasing cross-platform compatibility of touchscreen-based applications.

[0304] In some embodiments, a simulated touch event includes (1120) a plurality of fields having information including location information corresponding to a corresponding location in the user interface, and a stage value selected from a predefined set of stage values, the predefined set of stage values ​​including at least a first predefined stage value indicating the start stage of a simulated touch and a second predefined stage value indicating that the simulated touch event is an update to a previous simulated touch event. In some embodiments, the predefined set of stage values ​​includes (1122) at least a third predefined stage value indicating a hover stage, which indicates a simulated touch that approaches but does not touch the display. Providing touch events that include hover stage values ​​increases the range of possible user inputs (e.g., by enabling the possibility of a single gesture that includes both hover and touch stages), thereby reducing the amount of input required to perform an operation and enhancing the operability of the device.

[0305] In some embodiments, the device provides a first application with information describing a plurality of simulated touch events, sometimes referred to as a sequence of simulated touch events. Each simulated touch event includes a field having a touch identifier field, and the same value is assigned to the touch identifier field for the plurality of simulated touch events (e.g., a sequence) of a single gesture. For example, a sequence of simulated touch events assigned the same touch identifier value could correspond to a movement simulating the movement of a single finger relative to the display, starting with a movement simulating the single finger approaching but not touching the display, followed by a movement simulating the single finger touching the display, and optionally followed by a movement simulating the single finger lifting off the display, and then a movement simulating the single finger approaching but not touching the display again. (See above reference) Figure 7Q An example of such a simulated touch event sequence is described.

[0306] In some embodiments, the input device includes (1124) buttons (e.g., a pressable portion of a touchpad, mouse, or other input device). User input events include changes in button states (e.g., ...). Figures 7N to 7O Left-click and hold (714). Based on the button state change, the simulated touch event has a first predefined phase value indicating the start phase of the simulated touch (e.g., as referenced). Figure 7N As stated above, in response to the left mouse button click and hold 714, a simulated "touch start" input event is delivered to the application.

[0307] In some embodiments, device 300 displays (1126) on a display a change in the representation of a first application determined by the first application in response to information describing a simulated touch event (e.g., the device displays a response to a simulated touch event, such as text selection or cursor placement, as referenced). Figures 7L to 7P The above).

[0308] In some embodiments, a change in button state includes (1128) activation of the button. In some embodiments, while a pointer object is displayed, device 300 receives (1130) a second user input event, including deactivation of the button, at an input device separate from the display. In response to detecting the second user input event, device 300 provides a first application with information describing a second simulated touch event having a stage value indicating that the second simulated touch event corresponds to the end of a simulated touch event. For example, as referenced... Figures 7N to 7O As described above, the release of the left mouse button click and hold (714) is delivered to the application as a simulated "touch end" input event. Converting button activation into a simulated touch event allows applications developed for touchscreen systems to be compatible with peripherals that are not touchscreens.

[0309] When a pointer object is displayed, device 300 receives (1132) third user input at an input device separate from the display, indicating that the pointer object has been moved from a corresponding position to a third position on the display. In response to detecting the third user input event, device 300 provides a first application with information describing a third simulated touch event having a second predefined stage value, which indicates that the third simulated touch event is an update to the simulated touch event. For example, in some embodiments, user input moving the pointer object when a predefined button is not actuated results in a "hover change" input event being provided to the application, while user input moving the pointer object when a predefined button is actuated results in a "touch change" input event.

[0310] In some embodiments, when displaying a pointer object, device 300 receives (1134) a fourth user input event at an input device separate from the display, pointing to a third location of the displayed pointer object, the third location corresponding to a representation of a corresponding application different from the first application. In response to detecting the fourth user input event: based on determining that the corresponding application is configured to execute within a first operating system framework, device 300 provides the corresponding application with information describing the fourth simulated touch event, including information describing the location of one or more additional simulated touches; and based on determining that the corresponding application is not configured to execute within the first operating system framework (e.g., based on determining that the corresponding application is configured to execute within a second operating system framework different from the first operating system framework), device 300 refrains from providing the corresponding application with information describing the simulated touch event (e.g., device 300 provides a local, unconverted user input event instead of a simulated touch event). Providing simulated touch events only when an application is configured to receive touch events allows applications configured to receive different types of input events to operate using the same hardware.

[0311] In some embodiments, the input device is (1136) a first input device. The simulated touch event is a normalized touch event with a predefined event type and field set (e.g., Figure 9A Touch event 902 and / or Figure 9B Touch event 910).

[0312] In some embodiments, device 300 receives (1138) a fifth user input event at a second input device. In response to detecting the fifth user input event, device 300 describes information about a fifth simulated touch event to a first application. This fifth simulated touch event is a normalized touch event having the same predefined event type and field set as simulated touch events. Therefore, user input from multiple peripheral devices can be converted into simulated touch events on a simulated touchscreen display, allowing applications developed for touchscreens to receive input from two or more input devices, such as a mouse, keyboard, touchpad, trackball, etc.

[0313] In some embodiments, the input device is (1140) a corresponding input device among a plurality of input devices separate from the display (e.g., the electronic device has a plurality of input devices separate from the display, and this input device is one of those input devices). The user input event is the first user input event among a plurality of user input events received from the plurality of input devices separate from the display.

[0314] In some embodiments, device 300 generates (1142) multiple independent event streams based on multiple user input events. Device 300 merges information from the multiple event streams to generate a merged user input event stream, wherein information describing simulated touch events includes information describing the merged user input event stream. (Refer to the above) Figure 10 The generation of independent event streams is described, and references are provided below. Figures 12A to 12C Method 1200 describes the merging of event streams (e.g., demultiplexing) in more detail.

[0315] It should be understood that, Figures 11A to 11C The specific order in which the operations described herein are presented is merely exemplary and not intended to indicate that this order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with respect to other methods (e.g., method 1200 and method 1300) are similarly applicable to the above description relative to these other methods. Figures 11A to 11D Method 1100 is described herein. For example, the touch, gesture, input device, pointer event, touch event, simulated touch event, event stream, user interface feature, intensity threshold, position indicator, and data structure described above with reference to method 1100 optionally have one or more of the characteristics of touch, gesture, input device, pointer event, touch event, simulated touch event, event stream, user interface object, intensity threshold, position indicator, and data structure described herein with reference to other methods described herein (e.g., methods 1200 and 1300). For the sake of brevity, these details will not be repeated here.

[0316] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as described above relative to...). Figure 3 (as described) or application-specific chips.

[0317] The above is for reference only. Figures 11A to 11D The operation is optionally performed by Figures 1A to 1B and / or Figure 3The components depicted are used to implement this. For example, providing operation 1116 and display operation 1126 are optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects user input, and an event dispatcher module 174 delivers event information to application 136-1. The corresponding event recognizer 180 of application 136-1 compares the event information with a corresponding event definition 186 and determines whether a first contact (e.g., simulated contact) at a first location on a touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on a user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. The event handler 190 optionally utilizes or invokes a data updater 176 or an object updater 177 to update the application's internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand how it can be based on Figure 1B and / or Figure 3 The components depicted in the diagram are used to perform other processes.

[0318] Figures 12A to 12C A flowchart of a method 1200 for merging input events from independent event streams and providing the merged input events to an application is shown, according to some embodiments. Method 1200 is used in electronic devices with displays (e.g., Figure 3 The method is performed at device 300. The electronic device includes an input device separate from or coupled to the display. In some embodiments, the input device separate from the display is a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors to detect the proximity of an input object (e.g., a finger or stylus) above the touch-sensitive surface (e.g., a proximity sensor (such as an infrared sensor), a capacitive sensor in the touch-sensitive surface, or a camera adjacent to the touch-sensitive surface), and / or one or more sensors to detect the contact strength between the input object and the touch-sensitive surface. In some embodiments, the input device separate from the display is not a touch-sensitive surface (although the device may optionally include or be coupled to a touch-sensitive surface). Some operations in method 1200 may optionally be combined, and / or the order of some operations may optionally be changed. For simplicity, method 1200 is described with reference to device 300.

[0319] By using different gestures, devices with touch-sensitive displays (e.g., touchscreens) can interpret a wide range of user input. For example, a single-finger or two-finger swipe on a touchscreen can be interpreted as a scrolling gesture, while a two-finger pinch can be interpreted as a zoom gesture. Devices that include input devices separate from the display (e.g., peripherals) can also disambiguate a wide range of user input, but doing so presents additional challenges. For example, while gesture disambiguation is typically limited to interpreting user input from a single input device (e.g., a touchscreen), user input from a peripheral input device (e.g., a mouse, keyboard, or touchpad separate from the display) is often modified by user input from different peripheral input devices. For example, actuation of the scroll wheel can result in zooming when the "control" button on a separate keyboard is pressed, and in scrolling when the "control" button on the keyboard is not pressed.

[0320] The fact that user input from one input device can be modified by user input from different user input devices raises the problem of knowing when input events are relevant and should be interpreted together. This challenge is particularly acute for applications developed for touchscreens but still running on devices with input devices separate from the display (e.g., as described above with reference to method 1100), as touchscreen devices rarely face this challenge. Method 1200 addresses this problem by merging (e.g., demultiplexing) input events from independent event streams into a single merged input event before providing the input events to the application. Therefore, in some embodiments, the process of determining which input events are relevant is moved (e.g., delegated) from the software application (which may have been developed by a third party) to a host or operating system that is better suited to determine which input events are relevant. Merging input events in this way results in third-party applications interpreting input events with greater consistency. For applications developed for touchscreen frameworks, method 1200 increases cross-platform compatibility by addressing a problem that developers of touchscreen applications (i.e., applications configured to receive input from a touchscreen) rarely have to consider in their native frameworks.

[0321] To this end, device 300 displays (1202) the user interface of the application on a display and receives (1204) user input events at one or more input devices separate from the display. For example, device 300 may receive input from a mouse or touchpad indicating the position of a displayed pointer object. Furthermore, device 300 may receive user input corresponding to the actuation of a mouse button (e.g., left click, right click). For devices with touchpads, these touchpads have sensors for detecting contact strength; in addition to the actuation of a pressed button, device 300 may also detect pressure values.

[0322] Device 300 generates (1206) multiple independent event streams based on user input events. In some embodiments, these multiple independent event streams are multiplexed into a multiplexed event stream (also known as an event queue).

[0323] Device 300 merges (1208) information from the multiple independent event streams to generate a merged user input event stream. In some embodiments, merging is performed based on determining that merging criteria (sometimes referred to as predefined merging criteria) are met. For example, in some embodiments, timing-based criteria for user input events are used to determine whether to merge user input events. For example, in some embodiments, user input events are merged unless a predefined time period has elapsed and no user input event has been received. In some embodiments, if it is determined that the merging criteria are no longer met, the device abandons the subsequent merging of user events and provides the previously generated merged user input event to the application if it has not already been provided to the application.

[0324] In some embodiments, one or more input devices separate from the display include (1210) a first input device and a second input device that is different from and separate from the first input device. User input events include a first event received from the first input device and a second event received from the second input device. The merged user input event stream includes user input events that include information corresponding to the first event modified by information corresponding to the second event. For example, in some embodiments, the first event is an actuation of the scroll wheel on a mouse. The second event is an actuation of the “control” button on a keyboard. Because the actuation of the scroll wheel is modified by the “control” button, the merged events result in scaling of the user interface rather than scrolling or panning of the user interface. Merging (e.g., demultiplexing) input events from different input devices allows applications that expect to react to a single input event (e.g., applications for touchscreens) to utilize a wide range of modifiers available from different peripheral devices, including mice and keyboards.

[0325] In some embodiments, one or more input devices separate from the display include (1212) a first input device. User input events include: a first event received from the first input device, the first event corresponding to movement of a pointer on the display; and a second event, distinct from the first event, corresponding to actuation of the first input device. The merged user input event stream includes user input events, each including information corresponding to the first event and information corresponding to the second event.

[0326] In some embodiments, the merged user input event stream includes (1214) scroll events, which include positional information corresponding to a specific location in the user interface. For example, separate event streams include event streams from mouse position (e.g., the position used to generate a pointer object on the display) and actuation of the scroll wheel. Demultiplexing input events in this way allows for the single input event (e.g., Figure 9B The scroll event 910 is sent to the appropriate view of the application (e.g., an application configured to receive input from a touchscreen, sometimes referred to herein as a touchscreen application) that expects each input event to include all the necessary information.

[0327] In some embodiments, the corresponding scroll event includes a plurality of fields selected from (1216) of the group consisting of: a field having a value indicating a first scroll amount; a field having a value indicating either a multiplier of the first scroll amount or a second scroll amount greater than the first scroll amount; a field having a value indicating a corresponding position in the user interface; and a field having a stage value selected from a predefined set of stage values, the predefined set of stage values ​​including at least a first predefined stage value indicating the start stage of the corresponding scroll event and a second predefined stage value indicating that the corresponding scroll event is an update of the scroll event.

[0328] In some embodiments, each scroll event is (1218) a normalized event having a predefined event type and a set of fields, wherein the scroll event is configured to be normalized according to any of the following: a corresponding user input event received from the scroll wheel; a corresponding user input event corresponding to a first predefined gesture on a touch-sensitive display; and / or a corresponding user input event corresponding to a second predefined gesture different from the first predefined gesture on a touch-sensitive surface separate from the display. Normalizing scroll events from different types of input devices allows touchscreen-based applications (i.e., applications configured to receive input from a touchscreen) to receive scroll input from these different types of input devices.

[0329] Device 300 provides (1220) the application with information describing the merged stream of user input events. In some embodiments, a single merged user input event is represented as a single data structure provided to the application. In some embodiments, a single merged user input event includes optional sub-data structures (e.g., references to other data structures, such as data structures representing the individual events merged to produce the single merged user input event (or containing information about the individual events)).

[0330] Device 300 displays (1222) a response of an application on a display, the response being at least in part based on information describing a merged stream of user input events. In some embodiments, the application's response to a scrolling event includes (1224) panning at least a portion of the user interface (e.g., a scrollable area). In some embodiments, the display is a touch-sensitive display, and the application is configured to pan (1226) at least that portion of the user interface in response to touch input from the touch-sensitive display and a scrolling event from the merged stream of user input events.

[0331] It should be understood that, Figures 12A to 12C The specific order in which the operations described herein are presented is merely exemplary and not intended to indicate that the order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with respect to other methods (e.g., method 1100 and method 1300) also apply in a similar manner to the above description relative to these other methods. Figures 12A to 12C Method 1200 is described.

[0332] The above is for reference only. Figures 12A to 12C The operation is optionally performed by Figure 1A , Figure 1B and / or Figure 3 The components depicted are implemented as described. For example, the receiving operation 1204 and the display operation 1222 are optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects user input, and an event dispatcher module 174 delivers event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with a corresponding event definition 186 and determines whether a first contact (e.g., a simulated contact) at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. The event handler 190 optionally utilizes or invokes a data updater 176 or an object updater 177 to update the application's internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand how it can be based on Figure 1A , Figure 1B and / or Figure 3 The components depicted in the diagram are used to perform other processes.

[0333] Figures 13A to 13EA flowchart of a method 1300 for providing hover-based input events to an application according to some embodiments is shown. Method 1300 is used in electronic devices having input devices (e.g., Figures 1A to 1B Device 100 and / or Figure 3 The method is performed at device 300. The electronic device includes an input device separate from or coupled to the display. In some embodiments, the input device separate from the display is a touch-sensitive surface. In some embodiments, the electronic device includes one or more sensors to detect the proximity of an input object (e.g., a finger or stylus) above the touch-sensitive surface (e.g., a proximity sensor (such as an infrared sensor), a capacitive sensor in the touch-sensitive surface, or a camera adjacent to the touch-sensitive surface), and / or one or more sensors to detect the contact strength between the input object and the touch-sensitive surface. In some embodiments, the input device separate from the display is not a touch-sensitive surface (although the device may optionally include or be coupled to a touch-sensitive surface). Some operations in method 1300 may optionally be combined, and / or the order of some operations may optionally be changed. For simplicity, method 1300 is described with reference to devices 100 / 300 (e.g., some operations of method 1300 refer to device 100 with a touchscreen, and some operations of method 1300 refer to device 300 with an input device separate from the display).

[0334] Some touch-sensitive surfaces are able to detect the proximity of an input object (e.g., a finger or stylus) above the surface. These so-called "hover" inputs can be used to manipulate user interface objects. However, while the fact that hovering over a touch-sensitive surface can be an intuitive part of the same gesture as subsequent contact, hover inputs are generally perceived as separate from other user inputs (e.g., touch-based user input). Treating proximity-based and touch-based input events as coming from different user inputs makes the user interface cumbersome and unintuitive for the user.

[0335] Method 1300 addresses this problem by providing hover input events in a normalized touch input event format with the same data structure used for contact-based touch events. These input events (sometimes referred to herein as touch input events) include a phase field of a presentation value that describes whether the input event is a hover event, a contact-based event, etc. (This phase value may also indicate whether the event is a change or the end of a previously detected event). Hover input events associated with contact-based input events are assigned the same input identifier, so that when these events are provided to a gesture recognizer, they can be interpreted as part of the same gesture. To aid cross-platform compatibility, as described above, some embodiments also provide simulated hover events based on user input from an input device that does not detect proximity to a touch-sensitive surface. For example, a mouse pointer entering a window can be interpreted as a hover event over that window, while a left mouse click while the pointer is over the window can be interpreted as a contact-based event within that window. The provision of simulated touch events (including hover and contact input events) is discussed elsewhere in this document.

[0336] Devices 100 / 300 display (1302) on a monitor a user interface of an application that includes multiple user interface objects. For example, Figure 7A The user interface 730 shown includes optional text 732 and several virtual buttons (e.g., “Memo”, “Done”).

[0337] In some embodiments, the input device includes (1303) the aforementioned touch-sensitive surface. In some embodiments, the touch-sensitive surface is part of a display (e.g., the display is a touchscreen display, such as touchscreen 112). In some embodiments, the touch-sensitive surface is separate from the display (e.g., the touch-sensitive surface is a touchpad separate from the display). In some embodiments, input as described below is received from an input device that is not a touch-sensitive surface, and the touch input events described below are simulated touch input events (e.g., as referenced). Figures 7L to 7P and method 1100, Figures 11A to 11D (Simulated as described).

[0338] When displaying the user interface of an application, device 100 / 300 receives (1304) user input at an input device. For example, the touchscreen may receive touch gestures, which include a first portion of the touch gesture (e.g., a finger or stylus) approaching but not touching a touch-sensitive surface (e.g., a hover gesture portion, as shown in the reference). Figures 7B to 7C The second part (e.g., the touch gesture portion, as described above) and the second part of the touch gesture that contacts the touch-sensitive surface. Figures 7D to 7E (as described above). In some embodiments (e.g., where the device is not a touchscreen), user input is mouse / touchpad input that moves a pointer object on the display (e.g., as referenced).Figure 7L In some embodiments, user input includes a first portion that moves the cursor over the corresponding view (e.g., when a predefined mouse button, such as the left mouse button, is not actuated), and a second portion that includes actuation of the predefined mouse button. The first portion is converted into a first input event (described below), and the second portion is converted into a second input event (described below).

[0339] In response to user input, device 100 / 300 performs (1306) operations 1308-1314 or a subset thereof.

[0340] Device 100 / 300 provides (1308) a first input event to the application having a stage field with a first stage value, the first input event corresponding to an input that approaches but does not touch the touch-sensitive surface. In some embodiments, the first input event has a hover stage. In some embodiments, the first input event has a hover start stage.

[0341] In some embodiments, input is detected within a predefined distance from the touch-sensitive surface (e.g., ...). Figure 7B As shown, when finger 734 moves across the threshold hover distance, a hover start input event is provided to the view corresponding to user interface 730 to trigger (1310) a first input event corresponding to user input that approaches but does not touch the touch-sensitive surface.

[0342] In some embodiments, the first input event and the second input event are configured using a normalized touch event data structure (e.g., ...). Figure 9A The touch event data structure (902) is provided (1312) to the application. This normalized touch event data structure includes a phase field and an input identifier field. The first input event and the second input event have the same value in the input identifier field. By having the same value in the input identifier field, the first input event (e.g., a hover touch input event) and the second input event (e.g., a touch touch input event) can be recognized as part of a single gesture. Providing a single gesture with a hover phase and a touch phase improves device operability by allowing the device to recognize more intuitive gestures (e.g., moving the cursor based on a hover input and placing the cursor based on a touch input). These considerations are particularly important for providing a full range of input capabilities for the device when screen size is limited and peripheral input devices are inconvenient (e.g., smartphones).

[0343] In some embodiments, the first stage value indicates (1314) that the first input event corresponds to the start of an input that approaches but does not touch the touch-sensitive surface (e.g., a "hover start" input event). Providing a "hover start" input event allows the device to track subsequent inputs as updates to the "hover start" input event (e.g., by using an input identifier field, as described above).

[0344] After providing the first input event, device 100 / 300 provides (1316) to the application a second input event having a stage field with a second stage value, which corresponds to user input touching the touch-sensitive surface (e.g., a touch event). For example, in response to... Figure 7D When the touchscreen 112 is touched, the device provides a "touch start" event to the view corresponding to the user interface 730.

[0345] Device 100 / 300 displays (1318) updates to the user interface of the application on a display, the updates being at least partially based on a first input event and a second input event. In some embodiments, the device displays a first update based on the first input event (e.g., displaying the cursor, Figure 7B ) and a second update based on a second input event (e.g., start text selection, Figure 7D In some embodiments, the device displays a single update based on both a first input event and a second input event (e.g., the device provides the first and second input events to a gesture recognizer, which recognizes the first and second input events as part of a single gesture).

[0346] In some embodiments, after providing a first input event to the application, device 100 / 300 detects (1320) a change in user input. In response to detecting a change in user input corresponding to a change in the position or proximity of the user input that is approaching but not touching the touch-sensitive surface, device 100 / 300 provides a third input event to the application, the third input event having information indicating a change in the position or proximity of the user input that is approaching but not touching the touch-sensitive surface (e.g., a "hover change" input event) (e.g., a stage field with a hover change stage value). Providing an input event indicating a change in the position or proximity of the user input that is approaching but not touching the touch-sensitive surface improves device operability by allowing the device to track changes in hover input as part of a single user input (e.g., allowing the user to continuously move the cursor around the user interface). These considerations are particularly important for providing a full range of input capabilities for the device when screen size is limited and peripheral input devices are inconvenient (e.g., smartphones).

[0347] In some embodiments, after providing a first input event to the application, device 100 / 300 detects (1322) a further change in the user input. In response to detecting a further change in the user input, device 100 / 300 provides a fourth input event to the application, the fourth input event having a hover-end state indicating the user input (e.g., in...). Figure 7KIn response to finger 734 leaving a threshold hover distance, information (e.g., a stage field with a hover end stage value) is provided to the view corresponding to user interface 730 regarding a "hover end" input event. The hover end stage is used when the touch event is the last touch event before (A) the input moves out of the range of the proximity sensor (e.g., when the input is lifted to a height exceeding the range of the proximity sensor), or (B) the input moves out of an area (e.g., a view) of the application's user interface. In the latter case, when the input moves over another area of ​​the user interface, a new stream of touch events can be provided to another view of the application, adjacent to the view to which the previous touch event was provided, or to another application having a user interface area adjacent to the application view to which the previous touch event was provided.

[0348] In some embodiments, providing a "hover end" input event improves device operability by notifying the active gesture recognizer that user input or a portion of user input is complete. This helps the active gesture recognizer determine whether its corresponding gesture has been completed, failed, or canceled. When screen size is limited and peripheral input devices (e.g., smartphones) are inconvenient, an improved gesture recognizer enhances device operability by helping to provide the device with a full range of input capabilities.

[0349] In some embodiments, the second stage value is (1324) a touch start stage value indicating the initial contact between the user input and the touch-sensitive surface (e.g., in...). Figure 7D In response to contact with the touchscreen 112, a "touch start" input event is provided to the view corresponding to the user interface 730.

[0350] In some embodiments, after providing a first input event and a second input event to the application, device 100 / 300 detects (1326) a change in user input. In response to detecting a change in user input corresponding to a change in the position of the user input when in contact with a touch-sensitive surface, device 100 / 300 provides a fourth input event to the application, the fourth input event having information indicating the change in the position of the user input when in contact with a touch-sensitive surface (e.g., reference...). Figure 7JThe aforementioned "touch change" input event. Providing input events that indicate a change in the position of user input upon contact with a touch-sensitive surface improves device operability by allowing the device to track changes in touch input as part of a single user input (e.g., allowing a user to continuously move the cursor around the user interface while selecting text), and also makes device operation more efficient by providing the application with information indicating that the input event is a continuation of a gesture represented by one or more previously provided input events, thereby relieving the application of the need to determine whether (and how) the provided input event is related to previously provided input events. These considerations are particularly important for providing a full range of input capabilities for the device when screen size is limited and peripheral input devices are inconvenient (e.g., smartphones).

[0351] In some embodiments, after providing a first input event, a second input event, and a fourth input event to the application, device 100 / 300 detects (1328) a further change in the user input. In response to detecting a further change in the user input, corresponding to the user input being lifted off the touch-sensitive surface, device 100 / 300 provides a fifth input event to the application, the fifth input event having an indication that the user input has been lifted off the touch-sensitive surface (e.g., a "touch end" event, as referenced). Figure 7K The information (e.g., a phase field with a touch end phase value) is described above. In some embodiments, providing a "touch end" input event improves device operability by notifying the active gesture recognizer that user input or a portion of user input is complete. This helps the active gesture recognizer determine whether the recognition of its corresponding gesture has been successful, failed, or canceled. When screen size is limited and peripheral input devices (e.g., smartphones) are inconvenient, an improved gesture recognizer improves device operability by helping to provide the device with a full range of input capabilities.

[0352] In some embodiments, after providing a first input event, a second input event, a fourth input event, and a fifth input event to the application, device 100 / 300 detects (1330) a further change in user input. In response to detecting a further change in user input corresponding to a change in the position or proximity of the user input when approaching but not touching the touch-sensitive surface, device 100 / 300 provides a sixth input event to the application, the sixth input event having information indicating a change in the position or proximity of the user input approaching but not touching the touch-sensitive surface (e.g., providing a "hover change" event after contact with the touch-sensitive surface has ended) (e.g., a stage field with a hover change stage value). In some embodiments, providing a "hover change" input event after contact with the touch-sensitive surface has ended improves device operability by allowing input gestures to continue after contact has ended, which helps provide a full range of input capabilities for devices (e.g., smartphones) when screen size is limited and peripheral input devices are inconvenient.

[0353] In some embodiments, after providing a first input event, a second input event, a fourth input event, a fifth input event, and a sixth input event to the application, device 100 / 300 detects (1332) a further change in the user input. In response to detecting a further change in the user input corresponding to a second contact between the user input and the touch-sensitive surface, a seventh input event is provided to the application, the seventh input event having information indicating that the user input has touched the touch-sensitive surface (e.g., a stage field with a touch start stage value). For example, after detecting a lift-off from the touch-sensitive surface, the device may detect one or more hover events followed by one or more additional contact events. Therefore, the range of input gestures can be extended to include gestures such as “touch hover touch” gestures, which helps to provide a full range of input capabilities for devices (e.g., smartphones) when screen size is limited and peripheral input devices are inconvenient.

[0354] In some embodiments, the input device is (1334) separate from the display. The above description of performing method 1300 using an input device separate from the display is based on touchpad input and analog touch input from a device that is not a touch-sensitive surface.

[0355] In some embodiments, device 100 / 300 displays (1336) a pointer object on a display. While displaying the pointer object of the application, device 100 / 300 receives second user input at an input device, the second user input corresponding to movement of the pointer object on the display into a first view of the application's user interface. In response to the second user input: device 100 / 300 provides the application with an eighth input event having a stage field with a first stage value, the first stage value being a hover stage value indicating that the eighth input event corresponds to input approaching but not touching a touch-sensitive surface. After providing the eighth input event, device 100 / 300 provides the application with a ninth input event having a stage field with a second stage value, the second input event corresponding to user input touching a touch-sensitive surface. In some embodiments, the ninth input event is an initial input event generated when the second user input location "inputs" into a second view. Device 100 / 300 displays an update to the application's user interface on the display, the update being at least partially based on the eighth and ninth input events. For example, as referenced... Figures 7L to 7P The method described above provides hover input to the corresponding view when the pointer object moves over the corresponding view, and provides touch input to the corresponding view when a left mouse click is detected and held.

[0356] In some embodiments, the application's user interface includes (1338) a first view (e.g., Figures 7L to 7P The user interface 730 in the display includes displaying a second view (e.g., Figures 7L to 7P (Application 708 in the first application). The first input event and the second input event point to the first view of the first application.

[0357] In some embodiments, while displaying the first view and the second view, device 100 / 300 receives (1340) a second user input at an input device, the second user input having a position moving from a first position corresponding to the first view to a second position corresponding to the second view. In response to the second user input, device 100 / 300 generates a ninth input event having a corresponding stage field with a corresponding stage value. The ninth input event includes information corresponding to the second position. Based on determining that the corresponding stage value of the ninth input event is a hover stage indicating a touch approaching but not touching a touch-sensitive surface, device 100 / 300 provides the ninth input event to an application associated with the second view. Based on determining that the corresponding stage value of a third input event is a touch stage indicating a touch contacting a touch-sensitive surface, device 100 / 300 provides the ninth input event to an application associated with the first view. Thus, as referenced Figure 7QIn some embodiments, hover-based touch input events are delivered to a view corresponding to their current position (e.g., the position of a finger / stylus or the position of a displayed pointer object), while touch-based touch input events are delivered to a view corresponding to the position of the corresponding "touch start" input event.

[0358] In some embodiments, method 1300 includes (1342) displaying a pointer object on a display, receiving second user input from an input device separate from the display, and the second user input corresponding to a movement of the pointer object from a first position corresponding to a first view to a second position corresponding to a second view.

[0359] In some embodiments, the display is a touch-sensitive display including a touch-sensitive surface (1344) that is sensitive to contact that comes into contact with but does not touch the display. A second user input is received from the touch-sensitive display, and the second user input corresponds to an input at the touch-sensitive display that moves from a position on the touch-sensitive display above a first position corresponding to a first view to a position on the touch-sensitive display above a second position corresponding to a second view. In some embodiments, the second view is (1346) a corresponding view of an application.

[0360] In some embodiments, the application is (1348) a first application, the first view is a corresponding view of the first application, and the second view is a corresponding view of a second application different from the first application. For example, as referenced Figure 7Q Regardless of whether different views come from different applications or from the same application, the appropriate view to which input events are to be sent will be selected in the same way.

[0361] The above is for reference only. Figures 13A to 13E The operation is optionally performed by Figures 1A to 1B and / or Figure 3The components depicted are used to implement this. For example, providing operation 1308 and displaying operation 1318 are optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects user input, and an event dispatcher module 174 delivers event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with a corresponding event definition 186 and determines whether a first contact (e.g., a simulated contact) at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. The event handler 190 optionally utilizes or invokes a data updater 176 or an object updater 177 to update the application's internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand how it can be based on Figures 1A to 1B and / or Figure 3 The components depicted in the diagram are used to perform other processes.

[0362] For purposes of explanation, the foregoing description has been given by reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible based on the teachings above. The embodiments were chosen and described to best elucidate the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention with various modifications suitable for the particular intended use, as well as the various described embodiments.

Claims

1. A method comprising: In an electronic device having a display and an input device separate from the display: The user interface, which includes the following items, is simultaneously displayed on the display: A representation of the first application, including multiple user interface objects; and Pointer objects; When simultaneously displaying the representation of the first application and the pointer object: Receive user input events at the input device separate from the display, wherein the user input events point to a corresponding position in the user interface where the pointer object is displayed; In response to detecting the user input event, based on determining that the corresponding position of the pointer object displayed in the user interface is above the representation of the first application: A first simulated touch event is generated based on the user input event, the first simulated touch event indicating a simulated touch that is close to but does not touch the display; as well as Provide the first application with first information describing the first simulated touch event, including first information describing the location of one or more simulated touches in the user interface and indicating that the first simulated touch event indicates a simulated touch that is close to but does not touch the display; A second simulated touch event is generated based on the user input event, the second simulated touch event indicating a simulated touch on the display; as well as Provide the first application with second information describing the second simulated touch event, including the second information indicating a simulated touch on the display; as well as The display shows changes in the representation of the first application determined by the first application in response to first information describing the first simulated touch event and second information describing the second simulated touch event, the changes including a first update based on the first simulated touch event and a second update based on the second simulated touch event.

2. The method according to claim 1, comprising: A digitized event is generated based on the user input event, the digitized event including the corresponding position in the user interface where the pointer object is displayed; as well as The digital event is converted into the first analog touch event.

3. The method of claim 2, wherein the digitization event includes a pointer event, the pointer event being paired with or having position information corresponding to the corresponding position in the user interface displaying the pointer object.

4. The method according to claim 2, The digitization event is generated by a pointer event, which is paired with the corresponding position of the pointer object displayed in the user interface or has position information corresponding to the corresponding position.

5. The method according to any one of claims 1 to 4, wherein the corresponding simulated touch event comprises: The system has multiple fields containing information, including location information corresponding to the respective location in the user interface. And a stage value selected from a predefined set of stage values, the predefined set of stage values ​​including at least a first predefined stage value and a second predefined stage value, the first predefined stage value indicating the start stage of a simulated touch, and the second predefined stage value indicating that the corresponding simulated touch event is an update of a previous simulated touch event.

6. The method of claim 5, wherein the predefined set of stage values ​​includes at least a third predefined stage value, the third predefined stage value indicating a hover stage, the hover stage indicating a simulated touch that approaches but does not contact the display.

7. The method according to claim 5, wherein: The input device includes a button, wherein the button is a physical button; The first corresponding input event includes a change in the state of the button; and Based on the change in the state of the button, the first corresponding simulated touch event has a first predefined stage value indicating the start stage of the simulated touch.

8. The method according to claim 7, wherein: The change in the state of the button includes the activation of the button; and, The method further includes: When the pointer object is displayed, a second corresponding user input event, including the deactivation of the button, is received at the input device separate from the display; and In response to the detection of the second corresponding user input event: Information is provided to the first application describing a subsequent simulated touch event, the subsequent simulated touch event having a stage value indicating the end of the subsequent simulated touch event.

9. The method according to claim 5, comprising: When the pointer object is displayed, a third user input event is received at the input device separate from the display, the third user input event moving the pointer object from the corresponding position in the user interface to a third position in the user interface; as well as In response to the detection of the third user input event: Information is provided to the first application describing a third simulated touch event having a second predefined stage value, the second predefined stage value indicating that the third simulated touch event is an update of the corresponding simulated touch event.

10. The method according to any one of claims 1 to 4, comprising: When the pointer object is displayed, a fourth user input event is received at the input device separate from the display, the fourth user input event pointing to a third position where the pointer object is displayed, the third position corresponding to a representation of a corresponding application different from the first application; as well as In response to the detection of the fourth user input event: Based on the determination that the corresponding application is configured to execute in a first operating system framework, information describing a fourth simulated touch event is provided to the corresponding application, the information including information describing the location of one or more additional simulated touches; as well as If it is determined that the corresponding application is not configured to execute in the first operating system framework, information describing simulated touch events is not provided to the corresponding application.

11. The method according to any one of claims 1 to 4, wherein: The one or more simulated touches display the centroid at the corresponding position of the pointer object in the user interface.

12. The method according to any one of claims 1 to 4, wherein the input device is selected from the group consisting of: mouse; Roller; Keyboard; and A touchpad separate from the display.

13. The method according to any one of claims 1 to 4, wherein the corresponding position of the pointer object corresponds to a plurality of views, and the first simulated touch event is provided to each of the plurality of views.

14. The method according to any one of claims 1 to 4, wherein: The input device is the first input device; The first simulated touch event is a normalized touch event with a predefined event type and a set of fields; and The method includes: Receive the fifth user input event at the second input device; In response to the detection of the fifth user input event: The first application is provided with information describing a fifth simulated touch event, which is a normalized touch event.

15. The method according to any one of claims 1 to 4, wherein: The input device is a specific input device among a plurality of input devices separate from the display; The user input event is a corresponding user input event among a plurality of user input events received from the plurality of input devices separate from the display; and The method further includes: Generate multiple independent event streams based on the multiple user input events; as well as Information from the multiple independent event streams is merged to generate a merged user input event stream, wherein the information describing the simulated touch event includes information describing the merged user input event stream.

16. An information processing apparatus for use in an electronic device having a display and an input device separate from the display, comprising: A means for simultaneously displaying a user interface including the following on the display: A representation of the first application, including multiple user interface objects; and Pointer objects; A means for the following operations enabled when simultaneously displaying the representation and the pointer object of the first application: Receive user input events at the input device separate from the display, wherein the user input events point to a corresponding position in the user interface where the pointer object is displayed; In response to detecting the user input event, based on determining that the corresponding position of the pointer object displayed in the user interface is above the representation of the first application: A first simulated touch event is generated based on the user input event, the first simulated touch event indicating a simulated touch that is close to but does not touch the display; as well as Provide the first application with first information describing the first simulated touch event, including first information describing the location of one or more simulated touches in the user interface and indicating that the first simulated touch event indicates a simulated touch that is close to but does not touch the display; A second simulated touch event is generated based on the user input event, the second simulated touch event indicating a simulated touch on the display; as well as Provide the first application with second information describing the second simulated touch event, including the second information indicating a simulated touch on the display; as well as The display shows changes in the representation of the first application determined by the first application in response to first information describing the first simulated touch event and second information describing the second simulated touch event, the changes including a first update based on the first simulated touch event and a second update based on the second simulated touch event.

17. The information processing apparatus of claim 16, further comprising means for performing the method of any one of claims 2 to 15.

18. A computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by an electronic device having a display and an input device separate from said display, cause the electronic device to: The user interface, which includes the following items, is simultaneously displayed on the monitor: A representation of the first application, including multiple user interface objects; and Pointer objects; When simultaneously displaying the representation of the first application and the pointer object: Receive user input events at the input device separate from the display, wherein the user input events point to a corresponding position in the user interface where the pointer object is displayed; In response to detecting the user input event, based on determining that the corresponding position of the pointer object displayed in the user interface is above the representation of the first application: A first simulated touch event is generated based on the user input event, the first simulated touch event indicating a simulated touch that is close to but does not touch the display; as well as Provide the first application with first information describing the first simulated touch event, including first information describing the location of one or more simulated touches in the user interface and indicating that the first simulated touch event indicates a simulated touch that is close to but does not touch the display; A second simulated touch event is generated based on the user input event, the second simulated touch event indicating a simulated touch on the display; as well as Provide the first application with second information describing the second simulated touch event, including the second information indicating a simulated touch on the display; as well as The display shows changes in the representation of the first application determined by the first application in response to first information describing the first simulated touch event and second information describing the second simulated touch event, the changes including a first update based on the first simulated touch event and a second update based on the second simulated touch event.

19. The computer-readable storage medium of claim 18, wherein the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method of any one of claims 2 to 15.

Images