Devices and methods for processing input using gesture recognizers

Abstract

An electronic device detects input via an input device. In response to detecting the input, the electronic device monitors the input using a gesture recognizer. The gesture recognizer includes a plurality of component gesture recognizers that define requirements for recognizing a gesture. A first component gesture recognizer from the plurality of component gesture recognizers defines a first requirement for recognizing a gesture, and a second component gesture recognizer from the plurality of component gesture recognizers defines a second requirement for recognizing a gesture. Based on determining that the input satisfies a first gesture recognition criterion including the first requirement of the first component gesture recognizer and the second requirement of the second component gesture recognizer, the electronic device recognizes a first gesture using the gesture recognizer.

Description

Technical Field

[0001] The present invention generally relates to electronic devices having input devices such as touch-sensitive surfaces, including but not limited to electronic devices that use gesture recognition technology to process input. Background Art

[0002] Input devices such as touch-sensitive surfaces are widely used to provide input to electronic devices to manipulate user interfaces on displays and objects therein. But conventional methods for performing these manipulations are cumbersome, inefficient, and limited. In some cases, conventional methods of processing input are limited in functionality by the complexity and number of inputs that can be recognized. In some cases, as the complexity and number of inputs that must be recognized increase, conventional methods of processing input impose an increased computational load on the electronic device. In addition, conventional methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the invention

[0003] Therefore, there is a need for electronic devices with improved methods and interfaces for processing input and recognizing gestures. Such methods and interfaces optionally supplement or replace conventional methods for processing input and recognizing gestures. Such methods and interfaces reduce the amount, degree, and / or nature of input from the user and produce a more efficient human-computer interface. For battery-driven devices, such methods and interfaces can save power and increase the time between battery charges.

[0004] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface can be reduced or eliminated by means of the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the device is a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI primarily through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0005] In accordance with some embodiments, a method is performed at an electronic device having an input device. The method includes detecting an input via the input device, and in response to detecting the input, monitoring the input using a gesture recognizer. The gesture recognizer includes a plurality of component gesture recognizers defining requirements for recognizing a gesture. A first component gesture recognizer from the plurality of component gesture recognizers defines a first requirement for recognizing a gesture, and a second component gesture recognizer from the plurality of component gesture recognizers defines a second requirement for recognizing a gesture. The method includes recognizing a first gesture using the gesture recognizer based on determining that the input satisfies a first gesture recognition criterion including the first requirement of the first component gesture recognizer and the second requirement of the second component gesture recognizer.

[0006] According to some embodiments, a method is performed at an electronic device having a display and an input device. The method includes displaying a user interface of an application on the display. The user interface includes a plurality of views arranged in a view hierarchy. The view hierarchy defines a first relationship between a first view and a second view. The first view includes a first gesture recognizer, and the second view includes a second gesture recognizer. The method includes detecting an input at a first location corresponding to the displayed user interface via the input device, and processing the input using a gesture recognition hierarchy including the first gesture recognizer and the second gesture recognizer. A second relationship between the first gesture recognizer and the second gesture recognizer is determined based on the first relationship between the first view and the second view in the view hierarchy.

[0007] According to some embodiments, a method is performed at an electronic device having an input device. The method includes detecting an input via the input device, and in response to detecting the input, monitoring the input using a gesture recognition tree having a plurality of nodes. Each respective node of the gesture recognition tree corresponds to a respective gesture recognizer or a respective component gesture recognizer, and one or more nodes include one or more parameters describing the input. Monitoring the input using the gesture recognition tree includes: processing the input using a first node of the plurality of nodes, including determining a value of a first parameter of the one or more parameters; transmitting the first parameter from the first node to a second node of the plurality of nodes; and processing the input using the second node, including determining whether the input satisfies a gesture recognition requirement defined by the second node based on the first parameter.

[0008] According to some embodiments, an electronic device includes: a display; a touch-sensitive surface; optionally one or more memories for detecting the intensity of contact with the touch-sensitive surface; optionally one or more tactile output generators; one or more processors; and a memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, a computer-readable storage medium has instructions stored therein, which, when executed by an electronic device having a display, a touch-sensitive surface, optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, causes the device to perform the operations of any of the methods described herein or causes the operations of any of the methods described herein to be performed. According to some embodiments, a graphical user interface on an electronic device having a display, a touch-sensitive surface, optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, a memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements displayed in any of the methods described herein, and the one or more elements are updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display, a touch-sensitive surface, one or more sensors for detecting the intensity of contact with the touch-sensitive surface, and one or more tactile output generators; and a device for performing or causing the performance of the operations of any of the methods described herein. According to some embodiments, an information processing device used in an electronic device having a display, a touch-sensitive surface, one or more sensors for detecting the intensity of contact with the touch-sensitive surface, and one or more tactile output generators includes a device for performing or causing the performance of the operations of any of the methods described herein.

[0009] Therefore, an electronic device having a display, a touch-sensitive surface, optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, optionally one or more device orientation sensors, and optionally an audio system is provided with improved methods and interfaces for processing input and recognizing gestures, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can supplement or replace conventional methods for processing input and recognizing gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0011] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.

[0012] Figure 1B is a block diagram illustrating exemplary components for event handling according to some embodiments.

[0013] Figure 2 A portable multifunction device with a touch screen is shown in accordance with some embodiments.

[0014] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.

[0015] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown in accordance with some embodiments.

[0016] Figure 4B An example user interface for a multifunction device with a touch-sensitive surface that is separate from the display is shown in accordance with some embodiments.

[0017] Figure 4C-4E An example of a dynamic intensity threshold in accordance with some embodiments is shown.

[0018] FIG. 5A to FIG. 5Q Exemplary gesture recognizers for processing inputs, such as touch inputs, and their operation are shown in accordance with some embodiments.

[0019] FIG. 6A to FIG. 6C An example of a gesture recognition hierarchy generated from a view hierarchy of views in a user interface is shown in accordance with some embodiments.

[0020] 7A to 7E is a flow diagram for processing input using a gesture recognizer assembled from component gesture recognizers, in accordance with some embodiments.

[0021] FIG. 8A to FIG. 8C is a flow diagram for generating a gesture recognition hierarchy from a view hierarchy of views in a user interface in accordance with some embodiments.

[0022] FIG. 9A to FIG. 9B is a flow diagram for processing an input using a gesture recognizer to which values ​​of parameters describing the input have been associated, according to some embodiments. DETAILED DESCRIPTION

[0023] Many electronic devices have graphical user interfaces and input devices, such as touch-sensitive surfaces, for receiving user input. As the number and complexity of input gestures increase, processing the input and recognizing such gestures becomes more difficult and places a greater computational load on electronic devices using conventional gesture recognition techniques. Embodiments disclosed herein provide electronic devices with more efficient ways to process input and recognize gestures (e.g., using modular gesture recognizers that divide a larger set of input processing requirements into smaller subsets).

[0024] under, Figure 1A to Figure 1B , Figure 2 and Figure 3 An exemplary device is shown. FIG. 4A to FIG. 4B An exemplary user interface of an exemplary device on which embodiments disclosed herein are implemented is shown. FIG. 5A to FIG. 5Q Exemplary gesture recognizers for processing inputs, such as touch inputs, and their operation are shown. FIG. 6A to FIG. 6C An example of a gesture recognition hierarchy generated from a view hierarchy of views in a user interface is shown. 7A to 7E A flow chart illustrating a method of processing input using a gesture recognizer assembled from component gesture recognizers. FIG. 8A to FIG. 8C A flow chart of a method for generating a gesture recognition hierarchy from a view hierarchy of views in a user interface is shown. FIG. 9A to FIG. 9B A flow chart of a method for processing input using a gesture recognizer to which values ​​of parameters describing the input are associated is shown. FIG. 5A to FIG. 5Q and FIG. 6A to FIG. 6C The gesture recognizers, view hierarchies, and gesture recognizer hierarchies in 7A to 7E , FIG. 8A to FIG. 8C and FIG. 9A to FIG. 9B process.

[0025] Exemplary Devices

[0026] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0027] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0028] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limiting. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0029] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0030] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).

[0031] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0032] The device typically supports a variety of applications, such as one or more of the following applications: a note-taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0033] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0034] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 in accordance with some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0035] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, the movement of the touch-sensitive surface may be optionally interpreted or sensed by the user as the "roughness" of the touch-sensitive surface. Although such interpretation of touch by the user will be limited by the user's individualized sensory perception, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user. Providing tactile feedback to the user using tactile output enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and more efficiently.

[0036] In some embodiments, the tactile output pattern specifies characteristics of the tactile output, such as the amplitude of the tactile output, the shape of the motion waveform of the tactile output, the frequency of the tactile output, and / or the duration of the tactile output.

[0037] When a device (e.g., one or more tactile output generators that generate tactile output via moving a movable mass) generates tactile output with different tactile output patterns, the tactile output can produce different tactile sensations in a user holding or touching the device. Although the user's senses are based on the user's perception of the tactile output, most users will be able to recognize changes in the waveform, frequency, and amplitude of the tactile output generated by the device. Therefore, the waveform, frequency, and amplitude can be adjusted to indicate to the user that different operations have been performed. In this way, a tactile output with a tactile output pattern designed, selected, and / or arranged to simulate the characteristics (e.g., size, material, weight, stiffness, smoothness, etc.) of an object in a given environment (e.g., a user interface including graphical features and objects, a simulated physical environment with virtual boundaries and virtual objects, a real physical environment with physical boundaries and physical objects, and / or a combination of any of the above); behavior (e.g., oscillation, displacement, acceleration, rotation, extension, etc.); and / or interaction (e.g., collision, adhesion, repulsion, attraction, friction, etc.) will provide helpful feedback to the user in some cases, which reduces input errors and improves the efficiency of the user's operation of the device. In addition, tactile output is optionally generated to correspond to feedback that is independent of simulated physical properties (such as input threshold or object selection). Such tactile output will provide helpful feedback to the user in some cases, which reduces input errors and improves the efficiency of the user's operation of the device.

[0038] In some embodiments, the tactile output with a suitable tactile output mode serves as a prompt for an event of interest in a user interface or behind a screen in a device. Examples of events of interest include activation of an affordance (e.g., a real or virtual button, or a toggle switch) provided on a device or in a user interface, the success or failure of a requested operation, reaching or crossing a boundary in a user interface, entering a new state, switching input focus between objects, activating a new mode, reaching or crossing an input threshold, detecting or identifying a type of input or gesture, etc. In some embodiments, a tactile output is provided to serve as a warning or prompt for an upcoming event or result that will occur unless a change in direction or interruption input is detected in a timely manner. Tactile output is also used in other contexts to enrich user experience, improve accessibility of devices for users with visual or motor difficulties or other accessibility needs, and / or improve the efficiency and functionality of user interfaces and / or devices. Optionally, the tactile output is compared with audio input and / or visual user interface changes, which further enhances the user's experience when the user interacts with the user interface and / or device, and is conducive to better transmission of information about the state of the user interface and / or device, and this reduces input errors and improves the efficiency of user operations on the device.

[0039] It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0040] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The access of other components of the device 100 (such as the CPU 120 and the peripheral device interface 118) to the memory 102 is optionally controlled by the memory controller 122.

[0041] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0042] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0043] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data-only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11d), IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0044] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0045] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0046] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0047] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on haptic and / or tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0048] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In some embodiments, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0049] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or greater). The user optionally uses any suitable object or appendage such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0050] In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0051] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0052] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, the optical sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user views other video conference participants on the touch screen, etc.).

[0053] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to force sensor controller 159 in I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on a rear portion of device 100 opposite touch-sensitive display system 112 located on a front portion of device 100.

[0054] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0055] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. In some embodiments, the tactile output generator 167 includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). The tactile output generator 167 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a back of device 100 opposite touch-sensitive display system 112 located on the front of device 100 .

[0056] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on an analysis of data received from the one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0057] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in Figures 1A and 1B. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0058] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0059] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod and The external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in the device. In some embodiments, the external port is compatible with some of Apple Inc. (Cupertino, California) iPod and A Lightning connector that is the same as or similar to and / or compatible with the Lightning connector used in the device.

[0060] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (eg, a single finger contact or stylus contact) or multiple points of simultaneous contact (eg, “multi-touch” / multi-finger contact). In some embodiments, contact / motion module 130 and display controller 156 detect contact on a trackpad.

[0061] The contact / motion module 130 optionally detects gesture input from a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0062] In some embodiments, detecting a finger tap gesture depends on detecting the length of time between a finger press event and a finger lift event, but is independent of the intensity of the finger contact between the finger press event and the finger lift event. In some embodiments, based on determining that the length of time between the finger press event and the finger lift event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), a tap gesture is detected regardless of whether the intensity of the finger contact during the tap reaches a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a light press or deep press intensity threshold. Thus, a finger tap gesture can satisfy a particular input criterion that does not require a characteristic intensity of the contact to satisfy a given intensity threshold to satisfy the particular input criterion. For clarity, a finger contact in a tap gesture is typically required to satisfy a nominal contact detection intensity threshold to detect a finger press event, below which the contact is not detected. A similar analysis applies to detecting a tap gesture by a stylus or other contact. In the case where the device is capable of detecting contact with a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0063] The same concepts apply in a similar manner to other types of gestures. For example, a swipe gesture, a pinch gesture, an expand gesture, and / or a long press gesture may be optionally detected based on satisfying criteria that are independent of the strength of the contacts included in the gesture or that do not require one or more contacts performing the gesture to reach a strength threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a zoom gesture is detected based on the movement of two or more contacts toward each other; a zoom gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of a contact on a touch-sensitive surface having less than a threshold amount of movement. Thus, the statement that a particular gesture recognition criterion does not require the strength of the contacts to satisfy a corresponding strength threshold in order to satisfy the particular gesture recognition criterion means that the particular gesture recognition criterion can be satisfied when the contacts in the gesture do not reach the corresponding strength threshold, and can also be satisfied when one or more contacts in the gesture reach or exceed the corresponding strength threshold. In some embodiments, a tap gesture is detected based on determining that a finger down event and a finger up event are detected within a predefined time period, regardless of whether the contact is above or below a corresponding intensity threshold during the predefined time period, and a swipe gesture is detected based on determining that the contact moves by more than a predefined amount, even if the contact is above the corresponding intensity threshold at the end of the contact movement. Even in embodiments where detection of gestures is affected by the intensity of the contact performing the gesture (e.g., the device detects a long press more quickly when the intensity of the contact is above the intensity threshold, or the device delays detection of a tap input when the intensity of the contact is higher), detection of these gestures does not require the contact to reach a particular intensity threshold as long as the criteria for recognizing the gesture can be met without the contact reaching the particular intensity threshold (e.g., even if the amount of time required to recognize the gesture varies).

[0064] In some cases, the contact intensity threshold, duration threshold, and movement threshold are combined in various combinations to create a heuristic algorithm to distinguish between two or more different gestures for the same input element or area, so that multiple different interactions with the same input element can provide a richer set of user interactions and responses. The statement that a particular set of gesture recognition criteria does not require the intensity of the contact to meet the corresponding intensity threshold to meet the particular gesture recognition criteria does not exclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures with criteria that are met when the gesture includes a contact with an intensity above the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the intensity of the contact to meet the corresponding intensity threshold to meet the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on the contact reaching the corresponding intensity threshold). In such a competition, if the second gesture recognition criterion for the second gesture is met first, the gesture is optionally not recognized as meeting the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before the contact moves a predefined amount of movement, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves the predefined amount of movement before the contact reaches the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in such cases, the first gesture recognition criteria for the first gesture still does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the first gesture recognition criteria because if the contact remains below the corresponding intensity threshold until the gesture ends (e.g., a swipe gesture with an intensity of the contact that does not increase above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criteria. Thus, a particular gesture recognition criterion that does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the particular gesture recognition criterion will (A) in some cases ignore the intensity of the contact relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases fail to satisfy the particular gesture recognition criterion (e.g., for a long press gesture) if a set of competing intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-related gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input, and in this sense still depends on the intensity of the contact relative to the intensity threshold (e.g., for a long press gesture competing with the deep press gesture for recognition).

[0065] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0066] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphics attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0067] Haptic feedback module 133 includes various software components for generating instructions (e.g., instructions used by haptic feedback controller 161) to generate tactile output at one or more locations on device 100 using tactile output generator 167 in response to user interaction with device 100.

[0068] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0069] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0070] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0071] Contacts module 137 (sometimes called address book or contact list);

[0072] Telephone module 138;

[0073] Video conferencing module 139;

[0074] Email client module 140;

[0075] Instant messaging (IM) module 141;

[0076] Fitness support module 142;

[0077] A camera module 143 for still images and / or video images;

[0078] Image management module 144;

[0079] Browser module 147;

[0080] Calendar module 148;

[0081] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0082] A widget creator module 150 for forming a user-created widget 149-6;

[0083] Search module 151;

[0084] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0085] Memo module 153;

[0086] Map module 154; and / or

[0087] Online video module 155.

[0088] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0089] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0090] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0091] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0092] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0093] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting the corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0094] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, fitness support module 142 includes executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0095] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0096] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0097] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0098] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0099] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0100] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0101] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0102] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0103] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, etc. according to user instructions.

[0104] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0105] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0106] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0107] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a trackpad. By using a touch screen and / or a trackpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0108] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, a home menu, or a root menu. In such embodiments, a "menu button" is implemented using a trackpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0109] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0110] Event classifier 170 receives event information and determines an application 136-1 and an application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event distributor module 174. In some embodiments, application 136-1 includes application internal state 192 that indicates one or more current application views displayed on touch-sensitive display system 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine application view 191 to which the event information is to be delivered.

[0111] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0112] Event monitor 171 receives event information from peripheral device interface 118. Event information includes information about sub-events (sometimes referred to simply as "events" or "input events") (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture, or a touch moving on touch-sensitive display system 112, or a mouse button press, etc.). Event information for an input (e.g., a touch) may include one or more of the following: location, timestamp, direction, and velocity of the input. Peripheral device interface 118 transmits information it receives from I / O subsystem 106 (e.g., from touch-sensitive display system 112, one or more sensors 164, 165, and / or one or more input devices 350, 355) or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (through audio circuit 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0113] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0114] In some embodiments, event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173 .

[0115] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0116] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0117] Although views are often thought of as windows or other parts of a user interface, technically, a view is a part of an application that manages a specific range or area of ​​the application's user interface. Views are typically implemented as instances of a specific class or one of its subclasses and manage a rectangular area in the application window. Therefore, a view is an object that has an associated display area or part of a user interface and also has one or more computer programs (sometimes called "methods") associated with the class of which the view is an instance. The view is responsible for drawing content, handling multi-touch events, and managing the layout of any subviews. Drawing content involves drawing shapes, images, and text within the rectangular area of ​​the view using a variety of graphics techniques. The view responds to touch events in its rectangular area by using gesture recognizers or by directly handling touch events. In a view hierarchy, the parent view is responsible for positioning and sizing its subviews, and can do this dynamically. This ability to dynamically modify subviews enables views to adapt to changing conditions, such as interface rotation and animation.

[0118] You can think of views as building blocks that programmers or application developers use to construct an application's user interface. Rather than using one view to present all of an application's content, you typically use several views to build a view hierarchy. Each view in the hierarchy presents a specific portion of the application's user interface and is typically optimized for a specific type of content. For example, an application can have different views dedicated to presenting images, text, and other types of content.

[0119] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0120] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0121] Event distributor module 174 distributes the event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0122] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0123] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, such as an application-independent software module 195 (e.g., a user interface toolkit) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0124] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and recognizes an event from the event information. The event recognizer 180 includes a plurality of component event recognizers 181. In some embodiments, the component event recognizers 181 used to form the event recognizer 180 are provided by an application-independent software module 195.

[0125] A corresponding component gesture recognizer 181 of event recognizer 180 includes component event recognizer-specific code 182, which optionally includes event definitions 184. Component gesture recognizer 181 also includes state / phase information 185 and one or more ports 187. In some embodiments, component event recognizer 181 also includes one or more parameter values ​​186.

[0126] Component event identifier 181 receives event information from port 187-1. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., rotating from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0127] Component event identifier-specific code 182 is used to compare event information with predefined event or sub-event definitions and determine an event or sub-event based on the comparison, or determine or update the state of an event or sub-event. In some embodiments, component event identifier-specific code 182 includes event definition 184. Event definition 184 contains definitions of requirements that must be met in order for event identifier 180 to recognize an event (or gesture) defined by its component event identifier 181 (e.g., position requirements, minimum and / or maximum movement requirements, minimum or maximum timing requirements, intensity threshold requirements, etc.). In some embodiments, requirements are associated with specific sub-events, including, for example, touch start, touch end, touch stationary, touch move, touch cancel, and multi-touch. In some embodiments, event definition 184 also includes information for (e.g., identifying) one or more associated event handlers 190.

[0128] Corresponding component event recognizer 181 also includes state / stage 185, which indicates the state of the state machine of the corresponding component event recognizer; state / stage 185 may have various state values, such as "event possible", "event active", "event ended", "event failed", etc. When corresponding component event recognizer 181 of corresponding event recognizer 180 determines that the corresponding sub-event does not match event definition 184, corresponding component event recognizer 181 enters the "failed" state, which causes corresponding event recognizer 180 to enter an event impossible or event failed state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0129] In some embodiments, corresponding component event recognizers 181 include multiple ports 187 that indicate how the event delivery system should perform sub-event delivery to component event recognizers 181 in actively participating event recognizers 180. In some embodiments, ports 187 define the relationships (e.g., connections, and therefore information flows) of component event recognizers in a gesture recognition tree structure. In some embodiments, corresponding component event recognizers include one or more parameter values ​​186 that store values ​​of different attributes of earlier received input events or sub-events from the touch input for comparison with later received input events or sub-events. For example, a component event recognizer includes an initial position parameter that stores an initial position of the touch input identified from the earliest received input event of the touch input for comparison with the current position of the touch input identified from the later received input event to determine whether and how much the touch input has moved on the touch-sensitive surface.

[0130] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0131] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0132] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0133] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining the event to be recognized.

[0134] Figure 2 A device with a touch screen (e.g., Figure 1A The portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In these embodiments and in other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics will occur when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of the fingers that have made contact with the device 100 (from right to left, from left to right, up and / or down). In some embodiments or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0135] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0136] In some embodiments, the device 100 includes a touch screen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and for locking the device, a volume adjustment button 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button pressed for a predefined time interval; to lock the device by pressing the button and releasing the button before the predefined time interval has passed; and / or to unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0137] Figure 3 300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuits (sometimes referred to as chipsets) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 167 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or sensors similar to those described above) Figure 1AMemory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located away from CPU 310. In some embodiments, memory 370 stores information related to portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0138] Figure 3 Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to an instruction set for performing the above-mentioned functions. The above-mentioned identified modules or programs (that is, instruction sets) do not have to be implemented as separate software programs, processes or modules, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In certain embodiments, memory 370 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0139] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0140] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 is shown in accordance with some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0141] One or more signal strength indicators of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0142] ·time;

[0143] Bluetooth indicator;

[0144] Battery status indicator;

[0145] A tray 408 with icons for commonly used applications, such as:

[0146] o an icon 416 of the phone module 138 labeled “Phone”, the icon 416 optionally including an indicator 414 of the number of missed calls or voice messages;

[0147] o an icon 418 of the email client module 140 labeled “Mail,” which optionally includes an indicator 410 of the number of unread emails;

[0148] o An icon 420 labeled “Browser” of the browser module 147; and

[0149] o Icon 422 labeled “Music” of the video and music player module 152;

[0150] as well as

[0151] Icons for other apps, such as:

[0152] o Icon 424 of IM module 141 labeled “Message”;

[0153] o Icon 426 labeled “Calendar” of calendar module 148;

[0154] o Icon 428 labeled “Photos” of the image management module 144;

[0155] o An icon 430 labeled “Camera” of the camera module 143;

[0156] o Icon 432 labeled “Online Video” of the online video module 155;

[0157] o Icon 434 labeled "Stock Market" of the Stock Market widget 149-2;

[0158] o an icon 436 labeled “MAP” of the map module 154;

[0159] o Icon 438 labeled "Weather" of the Weather widget 149-1;

[0160] o Icon 440 labeled as “Clock” of the alarm clock widget 149-4;

[0161] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0162] o An icon 444 labeled "Notepad" of the notepad module 153; and

[0163] o An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136.

[0164] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0165] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touchpad 355) Figure 3 In some embodiments, the following examples are performed using input on a touch screen display 112 (where the touch-sensitive surface and the display are combined). Alternatively, in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with touch-sensitive surface 451 at a location corresponding to a corresponding location on the display (e.g., Figure 4B 460 and 462) (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B 450 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0166] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0167] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface that a user is interacting with. In some embodiments that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that the touch-sensitive surface 451 in the display includes a touch-screen display (e.g., Figure 1A A touch-sensitive display system 112 or Figure 4AIn some embodiments of a touch screen (e.g., a touch screen in a touch screen display), a contact detected on the touch screen acts as a "focus selector" such that when an input (e.g., a press input by a contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some embodiments, the focus moves from one area of ​​the user interface to another area of ​​the user interface without a corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these embodiments, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to convey the user's desired interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display).

[0168] In some embodiments, the response of the device to an input detected by the device depends on a criterion based on the intensity of the contact during the input. For example, for some "light press" inputs, the intensity of the contact exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the response of the device to an input detected by the device depends on a criterion that includes both the intensity of the contact during the input and a time-based criterion. For example, for some "deep press" inputs, the intensity of the contact exceeding a second intensity threshold greater than the first intensity threshold for light press during the input triggers a second response as long as a delay time passes between satisfying the first intensity threshold and satisfying the second intensity threshold. The duration of the delay time is typically less than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, where the delay time increases as the second intensity threshold increases). The delay time helps avoid accidentally identifying a deep press input. As another example, for some "deep press" inputs, a period of reduced sensitivity will occur after reaching the first intensity threshold. During this period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps avoid accidental deep press inputs. For other deep press inputs, the response to detecting the deep press input is not dependent on time-based criteria.

[0169] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Example factors are described in U.S. Patent Applications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0170] For example, Figure 4C A dynamic intensity threshold 480 is shown that changes over time based in part on the intensity of touch input 476 over time. Dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predefined delay time p1 from when touch input 476 is initially detected, and a second component 478 that tracks the intensity of touch input 476 over time. The initial high intensity threshold of first component 474 reduces accidental triggering of a "deep press" response while still allowing an immediate "deep press" response if touch input 476 provides sufficient intensity. Second component 478 reduces unintentional triggering of a "deep press" response through gradual intensity fluctuations of the touch input. In some embodiments, when touch input 476 satisfies dynamic intensity threshold 480 (e.g., after Figure 4C ), triggering a "deep press" response.

[0171] Figure 4D Another dynamic intensity threshold 486 (eg, intensity threshold I D ). Figure 4D Two other intensity thresholds are also shown: a first intensity threshold I H and the second intensity threshold I L .exist Figure 4D Although the touch input 484 satisfies the first intensity threshold I before time p2, H and the second intensity threshold I L , but no response is provided until a delay time p2 has elapsed at time 482. Similarly, Figure 4D , the dynamic intensity threshold 486 decays over time, wherein the decay is from time 482 (triggering the second intensity threshold I L This type of dynamic intensity threshold reduction begins at time 488 after a predefined delay time p1 has elapsed when the trigger is associated with a lower threshold intensity (such as the first intensity threshold I H Or the second intensity threshold I L ) is triggered after or simultaneously with the response associated with the dynamic intensity threshold I D The associated response.

[0172] Figure 4E Another dynamic intensity threshold 492 (eg, intensity threshold I D ).exist Figure 4E In the example, after a delay time p2 has passed since the touch input 490 was initially detected, the trigger is triggered with the intensity threshold I L At the same time, the dynamic intensity threshold 492 decays after a predefined delay time p1 has passed since the touch input 490 was initially detected. L The associated response then decreases the intensity of the touch input 490, and then increases the intensity of the touch input 490 without releasing the touch input 490, which may trigger the touch input 490 to be in accordance with the intensity threshold I. D The associated response (e.g., at time 494) may be performed even when the intensity of touch input 490 is below another intensity threshold (e.g., intensity threshold I L ) is also the case.

[0173] User interface and associated processes

[0174] Attention now turns to embodiments of a user interface ("UI") and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300 having a display, a touch-sensitive surface, (optional) one or more tactile output generators for generating tactile output, and (optional) one or more sensors for detecting intensity of contact with the touch-sensitive surface.

[0175] FIG. 5A to FIG. 5Q Exemplary gesture recognizers for processing inputs, such as touch inputs, and their operation are shown in accordance with some embodiments. FIG. 6A to FIG. 6C An example of a gesture recognition hierarchy (sometimes referred to as a gesture recognition tree) generated from a view hierarchy of views in a user interface based on received input is shown. The gesture recognizers, view hierarchies, and gesture recognition hierarchies in these figures are used to illustrate the processes described below, including 7A to 7E , FIG. 8A to FIG. 8C and FIG. 9A to FIG. 9BFor ease of explanation, some of the embodiments will be discussed with reference to operations performed on a device having touch-sensitive display system 112 in response to receiving input on touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, optionally, similar operations are performed on a device having display 450 and a separate touch-sensitive surface 451 in response to receiving input on touch-sensitive surface 451, or on a device having keyboard / mouse 350 in response to receiving input via a mouse.

[0176] FIG. 5A to FIG. 5Q An exemplary gesture recognizer and its operation in processing input in accordance with some embodiments is shown.

[0177] FIG. 5A to FIG. 5F An exemplary tap gesture recognizer 500 and its operation in processing input is shown. Figure 5A The tap gesture recognizer 500 is shown to include a plurality of component gesture recognizers arranged in a tree structure, the plurality of component gesture recognizers defining requirements for recognizing a tap gesture. The plurality of component gesture recognizers of the tap gesture recognizer 500 include:

[0178] A spatial event listener 502, which is required to know the location of touch input received on the touch-sensitive surface by monitoring input events that include location information;

[0179] A maximum duration component 504, which requires that the amount of time that the touch input remains on the touch-sensitive surface is less than a predefined threshold amount of time T th ;and

[0180] A distance component 506, which requires that the amount of movement (if any) of the touch input on the touch-sensitive surface is less than a predefined threshold movement amount D max,th .

[0181] Tap gesture recognizer 500 also includes a leading pause component 508, which is optionally used to require that any gesture recognizer having a higher priority than tap gesture recognizer 500 (e.g., any predecessor gesture recognizer) fail in order to recognize the tap gesture. In other words, in certain situations described herein, leading pause 508 prevents tap gesture recognizer 500 from succeeding while any predecessor gesture recognizer of tap gesture recognizer 500 is active.

[0182] In addition, tap gesture recognizer 500 includes two gate combiners. Gate combiner 510 is controlled by leading pause component 508 and maximum duration component 504. When maximum duration component 504 is in a failed state (e.g., the input has exceeded a predefined threshold amount of time, sometimes referred to as a maximum threshold duration), gate combiner 510 is in a failed state. When maximum duration component 504 is in a state other than the failed state (e.g., an active state), gate combiner 510 passes the input processing results (e.g., states and parameter values) received from leading pause component 508. Similarly, gate combiner 512 is controlled by gate combiner 510 and distance component 506. When distance component 506 is in a failed state, gate combiner 512 is in a failed state. When distance component 506 is in a state other than the failed state (e.g., an active state), gate combiner 512 passes the input processing results (e.g., states and parameter values) received from gate combiner 510. More generally, when the right-hand child component gesture recognizer of the gated combiner is in the failed state, the gated combiner is in the failed state, and when the right-hand child component gesture recognizer of the gated combiner is in a state other than the failed state, the gated combiner passes on the input processing results received from its left-hand child component gesture recognizer.

[0183] Although FIG. 5A to FIG. 5F Only the leading pause component 508 is shown as having the spatial event listener 502 as a sub-component, but in some embodiments, the maximum duration component 504 and / or the distance component 506 also have a spatial event listener as a sub-component (e.g., an additional instance of the spatial event listener 502), or alternatively have an input port that is also coupled to the spatial event listener 502.

[0184] Figure 5B An input event is shown being received at tap gesture recognizer 500 (e.g., from event distributor module 174) in response to receiving touch input at the device. Specifically, the input event is received at gating combiner 512, which is the root node of tap gesture recognizer 500. Input events received at gesture recognizers are sometimes described as "flowing down the tree," i.e., input events are transmitted from each respective node at a higher level in the tree to its associated node at the next lower level in the tree. In some cases, a combiner with multiple child nodes transmits input events to its child nodes based on predefined information transmission rules, as described in more detail herein. Figure 5BIn the example shown, the gate combiner 512 transmits the input event to both its child nodes, namely the gate combiner 510 and the distance component 506. Similarly, the gate combiner 510 transmits the input event to both its child nodes, namely the leading pause component 508 and the maximum duration component 504. In addition, the leading pause component 508 transmits the input event to its child node, namely the spatial event listener 502.

[0185] Figure 5C The tap gesture recognizer 500 is shown as being processed by the component gesture recognizers in the tap gesture recognizer 500. Figure 5B The results of processing an input event received at a gesture recognizer are sometimes described as being passed "up the tree," i.e., the input event is transmitted from each corresponding node at a lower level in the tree to its associated node at the next higher level in the tree. A corresponding node at a higher level in the tree that has multiple associated nodes at the next lower level in the tree determines its own result based on a combination of the results from its multiple associated nodes.

[0186] exist FIG. 5B to FIG. 5C In the example shown, the received input event is a "spatial event", that is, it includes the location of the received touch input (information about it) and includes the initial location p of the touch input. i The received input event is also the first input event of a newly received touch input and therefore has a phase value of "touch start" and an initial timestamp t indicating the time when the touch input was first received or detected. i .like Figure 5C As shown, spatial event listener 502 has identified the input event as a spatial event of an ongoing touch input (as indicated by the "touch start" stage value), and is therefore in an "active" state. Spatial event listener 502 transmits its current "active" state and spatial input event to its parent node, namely leading pause 508.

[0187] exist Figure 5CIn the example shown, predecessor pause component 508 is in an "active" state because tap gesture recognizer 500 does not have an active predecessor gesture recognizer (either because no other gesture recognizer has a higher priority than tap gesture recognizer 500 or because every gesture recognizer with a higher priority than tap gesture recognizer 500 has failed). Predecessor pause component 508 communicates its current "active" state and the spatial input event to its parent, namely, gate combiner 510. In some embodiments, in the event that another gesture recognizer (such as a double-tap gesture recognizer) has a higher priority than tap gesture recognizer 500 (e.g., by using an exclusive combiner, as described herein), tap gesture recognizer 500 receives an indication, included with or in addition to the input event, as to whether the predecessor gesture recognizer is active. In such an event, predecessor pause component 508 outputs an indication that recognition of a tap gesture by tap gesture recognizer 500 should be suspended or "paused" until an indication is received that the predecessor gesture recognizer has failed. If such an indication is not received, the tap gesture recognizer remains in the suspended state and is unable to successfully recognize a tap gesture (e.g., the first tap gesture of a double-tap gesture will be recognized by the tap gesture recognizer even if the double-tap gesture recognizer has priority over the tap gesture recognizer and recognizes a double-tap gesture).

[0188] The longest duration component 504 identifies the current timestamp t of the input event. i In addition, the longest duration component 504 has identified the input event as an event with a phase value of "touch start". Therefore, the longest duration component 504 has identified the current timestamp t i The maximum duration component 504 uses the initial timestamp to determine whether the touch input has been on the touch-sensitive surface for more than a predefined threshold amount of time T th When the touch input is initially detected, when the timestamp is the initial timestamp t i When the touch input has not been on the touch-sensitive surface for more than a predefined threshold amount of time T th , so the longest duration component 504 is in the “active” state. The longest duration component 504 transmits its current “active” state to its parent node, namely the gate combiner 510.

[0189] As described above, the gate combiner 510 is controlled by the leading pause component 508 and the longest duration component 504. Since the longest duration component 504 is in the "active" state and therefore not in the "failed" state, the gate combiner 510 passes the input processing results received from the leading pause component 508. Therefore, the gate combiner 510 transmits the "active" state and the spatial event to its parent node, namely the gate combiner 512.

[0190] The distance component 506 has identified the current position p of the touch input from the input event. i In addition, the distance component 506 has identified the input event as an event with a phase value of "touch start". Therefore, the distance component 506 has identified the current position p i The distance component is used to determine whether the movement of the touch input (if any) satisfies the minimum movement amount and / or the maximum movement amount. In some embodiments, the distance component is used in gesture recognizers that require movement of the touch input to define a minimum threshold movement amount. In some embodiments, the distance component is used in gesture recognizers that require movement of the touch input to be limited to define a maximum threshold movement amount. In some embodiments, both the minimum threshold movement amount and the maximum threshold movement amount can be defined to require movement of the touch input to be within a predefined range.

[0191] exist Figure 5C In the example shown, the distance component 506 uses the initial position to determine whether the touch input has moved on the touch-sensitive surface beyond a predefined maximum threshold amount D max,th When the touch input is initially detected, when the position is the initial position p i , the touch input has not moved on the touch-sensitive surface, so distance component 506 is in an "active" state. Distance component 506 transmits its current "active" state to its parent node, namely, gate combiner 512.

[0192] As described above, gate combiner 512 is controlled by gate combiner 510 and distance component 506. Since distance component 506 is in the "active" state and therefore not in the "failed" state, gate combiner 512 passes the input processing results received from gate combiner 510. Therefore, gate combiner 512 transmits the "active" state and spatial events received from gate combiner 510. Since gate combiner 512 is the root node of tap gesture recognizer 500, the overall state of tap gesture recognizer 500 is the "active" state, and the overall output value of tap gesture recognizer 500 is the overall output value of spatial events. In some embodiments, the overall state and output value of an assembled gesture recognizer (such as tap gesture recognizer 500) are transmitted by the gesture recognizer to the parent node of the gesture recognizer, where the gesture recognizer is used as part of another gesture recognizer. For example, as described herein with reference to Figure 5G to Figure 5L As described, double-tap gesture recognizer 520 includes tap gesture recognizer 524 as an assembly of components, and thus the overall state and output value of tap gesture recognizer 524 is transmitted to other components in double-tap gesture recognizer 520 (e.g., Figure 5GIn some embodiments, the overall state and output value of the assembled gesture recognizer is transmitted by the gesture recognizer to the parent node of the gesture recognizer, where the gesture recognizer is a child node in the gesture recognition hierarchy. Fig. 6A As described, gesture recognition hierarchy 640 includes assembled tap gesture recognizer 614 as a child node, and thus the overall state and output value of tap gesture recognizer 614 is transmitted to other components in gesture recognition hierarchy 640 (e.g., Fig. 6A In some embodiments, the overall state and output value of the assembled gesture recognizer is transmitted by the gesture recognizer to an application executing on the device and including the gesture recognizer (e.g., Figure 1B application 136-1).

[0193] Figure 5D An example of a successful recognition of a tap gesture is shown. Specifically, Figure 5D FIG. 1 shows the processing and the processing of the gesture recognition with respect to the tap gesture 500 when all gesture recognition requirements are met. FIG. 5B to FIG. 5C Describes the result of a touch input being lifted off of an associated second input event. Figure 5B as shown) and processing (as Figure 5C shown) relative to FIG. 5B to FIG. 5C After the (first) input event described, a second input event is received. The second input event is another "space event", but has a phase value of "touch ended" and includes the position p of the touch input at the time of lift-off. c and a timestamp t indicating the lift-off time c .like Figure 5D As shown, the spatial event listener 502 has identified the input event as a spatial event of a completed touch input (as indicated by the "touch ended" stage value), and is therefore in the "ended" state. The spatial event listener 502 transmits its current "ended" state and the spatial input event to its parent node, the leading pause 508.

[0194] exist Figure 5D In the example shown, leading pause component 508 is in the "finished" state because tap gesture recognizer 500 has no active predecessor gesture recognizer. Leading pause component 508 transmits its current "finished" state and the spatial input event to its parent, strobe combiner 510.

[0195] The longest duration component 504 identifies the current timestamp t of the input event. c The longest duration component has the current timestamp t c The initial timestamp t stored iThe comparison is made to determine the amount of time (t c –t i ) is less than a predefined threshold amount of time T th Therefore, the longest duration component 504 has not yet transitioned to the "failed" state. Instead, the longest duration component 504 has identified the input event as an event with a phase value of "touch ended" and has not yet transitioned to the "failed" state for any reason, so it has transitioned to the "finished" state. The longest duration component 504 transmits its current "finished" state to its parent node, the gate combiner 510.

[0196] As described above, the gate combiner 510 is controlled by the leading pause component 508 and the longest duration component 504. Since the longest duration component 504 is in the "finished" state and therefore not in the "failed" state, the gate combiner 510 passes the input processing results received from the leading pause component 508. Therefore, the gate combiner 510 transmits the "finished" state and the spatial event to its parent node, namely the gate combiner 512.

[0197] The distance component 506 has identified the current position p of the touch input from the input event. c The distance component 506 converts the current position p c With the stored initial position p i is compared to determine the distance traveled by the touch input (p c –p i, |p c –p i |The absolute value of) is less than the predefined maximum threshold distance D max,th . Therefore, distance component 506 has not yet transitioned to the "failed" state. Instead, distance component 506 has identified the input event as an event with a phase value of "touch ended" and has not yet transitioned to the "failed" state for any reason, so has transitioned to the "ended" state. Distance component 506 transmits its current "ended" state to its parent node, namely, gate combiner 512.

[0198] As described above, gate combiner 512 is controlled by gate combiner 510 and distance component 506. Since distance component 506 is in the "finished" state and therefore not in the "failed" state, gate combiner 512 passes the input processing result received from gate combiner 510. Therefore, gate combiner 512 transmits the "finished" state and spatial event received from gate combiner 510. Since gate combiner 512 is the root node of tap gesture recognizer 500, the overall state of tap gesture recognizer 500 is the "finished" state, and the overall output value of tap gesture recognizer 500 is the overall output value of spatial event. As gate combiner 512 transitions to the "finished" state, tap gesture recognizer 500 has successfully recognized the touch input as a tap gesture.

[0199] Figure 5E An example of a tap gesture recognizer 500 failing to recognize a touch input is shown. Specifically, Figure 5E shows the result of processing the third input event, which is related to FIG. 5B to FIG. 5C The movement of the touch input is associated with, and instead of relative to Figure 5D After receiving (such as Figure 5B as shown) and processing (as Figure 5C shown) relative to FIG. 5B to FIG. 5C After the (first) input event described, a third input event is received. The third input event is another "spatial event", but has a phase value of "touch move" (or in some embodiments, "touch change") and includes the current position p of the touch input. c and a timestamp t indicating a time associated with the third input event c .like Figure 5E As shown, the spatial event listener 502 has identified the input event as a spatial event of an ongoing touch input (as indicated by the "touch move" stage value), and is therefore in the "active" state. The spatial event listener 502 transmits its current "active" state and the spatial input event to the leading pause 508.

[0200] exist Figure 5D In the example shown, leading pause component 508 is in the “active” state because tap gesture recognizer 500 has no active predecessor gesture recognizer. Leading pause component 508 transmits its current “active” state and the spatial input event to strobe combiner 510.

[0201] The longest duration component 504 identifies the current timestamp t of the input event. c The longest duration component has the current timestamp t c The initial timestamp t stored iThe comparison is made to determine the amount of time (t c –t i ) is less than a predefined threshold amount of time T th Therefore, the longest duration component 504 has not yet transitioned to the "failed" state, and remains in the "active" state because the input event has a phase value of "touch move" instead of "touch end". The longest duration component 504 transmits its current "active" state to the gate combiner 510.

[0202] Since the longest duration component 504 is in the “active” state and therefore not in the “failed” state, the gate combiner 510 passes the “active” state and spatial event received from the leading pause component 508 to the gate combiner 512 .

[0203] The distance component 506 has identified the current position p of the touch input. c , and the current position p c With the initial position p stored i is compared to determine the distance traveled by the touch input (|p c –p i |) is greater than the predefined maximum threshold distance D max,th Therefore, distance component 506 has transitioned to the “fail” state. Distance component 506 transmits its current “fail” state to gate combiner 512.

[0204] Gating combiner 512 is controlled by the "failed" state of distance component 506, so that gating component 512 is also in the "failed" state. Since gating combiner 512 is the root node of tap gesture recognizer 500, the overall state of tap gesture recognizer 500 is the "failed" state, and no tap gesture is recognized by tap gesture recognizer 500.

[0205] Fig. 5F Another example of a tap gesture recognizer 500 failing to recognize a touch input is shown. Specifically, Fig. 5F shows the result of processing the fourth input event, which is related to FIG. 5B to FIG. 5C The touch input is associated with the touch sensitive surface and is replaced with respect to FIG. 5D to FIG. 5E The second input event and the third input event described are received. Figure 5B as shown) and processing (as Figure 5C shown) relative to FIG. 5B to FIG. 5C After the (first) input event described, a fourth input event is received. The fourth input event is another "spatial event", but has a phase value of "touch stationary" and includes the current position p of the touch input. cand a timestamp t indicating a time associated with the third input event c .

[0206] The longest duration component 504 has identified the current timestamp t of the input event. c , and the current timestamp t c The initial timestamp t stored i The comparison is made to determine the amount of time (t c –t i ) has exceeded a predefined threshold amount of time T th Therefore, the longest duration component 504 has transitioned to the "failed" state. The longest duration component 504 transmits its current "failed" state to the gate combiner 510, which controls the gate combiner 510 to also be in the "failed" state (e.g., causes the gate combiner 510 to transition to the "failed" state).

[0207] The distance component 506 has identified the current position p of the touch input. c , and the current position p c With the initial position p stored i is compared to determine the distance traveled by the touch input (|p c –p i |) is less than the predefined maximum threshold distance D max,th Therefore, distance component 506 has not yet transitioned to the “failed” state and remains in the “active” state because the input event has a phase value of “touch stationary” instead of “touch ended.” Distance component 506 transmits its current “active” state to gate combiner 512 .

[0208] Since distance component 506 is in the "active" state and therefore not in the "failed" state, gated combiner 512 propagates the "failed" state and spatial events received from gated combiner 510. Since gated combiner 512 is the root node of tap gesture recognizer 500, the overall state of tap gesture recognizer 500 is the "failed" state, and no tap gesture is recognized by tap gesture recognizer 500. In some embodiments, since tap gesture recognizer 500 has failed, the transitive spatial events output by tap gesture recognizer 500 are ignored.

[0209] Figure 5G to Figure 5L An exemplary double-click gesture recognizer 520 is shown and its operation in processing input. Figure 5G Double-tap gesture recognizer 520 is shown to include multiple component gesture recognizers that define the requirements for a double-tap gesture, including:

[0210] Repeater 522, which requires its child gesture recognizer or sub-gesture recognizer to repeat successfully for (at least) the repeat count value (“count 重复 ”) a predefined number of times, optionally subject to a timeout period (“T 超时 ”), such that each additional success must occur before the amount of time that has elapsed since the most recent previous success reaches (or exceeds) a predefined timeout period; and

[0211] · Tap gesture recognizer 524, which is referred to herein as FIG. 5A to FIG. 5F A high-level representation of the various components of a tap gesture recognizer as described in exemplary tap gesture recognizer 500 of FIG.

[0212] exist Figure 5G In the example shown, in order for double-tap gesture recognizer 520 to succeed, repeater 522 requires that the recognitions be within 0.5 seconds of each other (by 超时 Set to value "0.5s" for two successful tap gestures (by setting count 重复 is set to a value of "2"). In some embodiments, Figure 5G to Figure 5L As shown, repeater 522 uses first instance 524a of the tap gesture recognizer to monitor for a first desired tap gesture, and uses second instance 524b of the tap gesture recognizer to monitor for a second desired tap gesture (after recognizing the first desired tap gesture). More generally, in some embodiments, a new / different instance of a child gesture recognizer of the repeater is used to monitor each successive occurrence of a child gesture recognizer's gesture recognition requirement being met. In some embodiments, a repeater in a gesture recognizer reuses a single child gesture recognizer (e.g., the same instance of the child gesture recognizer) by resetting the child gesture recognizer's state and value each time the child gesture recognizer's gesture recognition requirement is met. In some embodiments, a repeater tracks the number of times its child gesture recognizer succeeds.

[0213] Figure 5G Also shown is that in response to receiving a first touch input at the device, a first input event ("Event 1") is received at double-tap gesture recognizer 520 (e.g., from event dispatcher module 174). Specifically, the first input event is received at repeater 522, which is the root node of double-tap gesture recognizer 520. Repeater 522 transmits the first input event to its child node, namely tap gesture recognizer 524a. More specifically, since tap gesture recognizer 524a is itself a tree structure, repeater 522 transmits the first input event to the root node of tap gesture recognizer 524a (e.g., a gating combiner, such as Figure 5B), which in turn transmits the first input event to other nodes in the tap gesture recognizer 524a (e.g., as described herein with reference to Figure 5B In some embodiments, in response to receiving the first input event, repeater 532 adds first instance 524a of the tap gesture recognizer to double-click gesture recognizer 520, and then transmits the first input event to the root node of tap gesture recognizer 524a for processing. In some embodiments, "adding" an instance of a first gesture recognizer (e.g., a tap gesture recognizer) or a first component gesture recognizer to a second gesture recognizer (e.g., a double-click gesture recognizer) includes creating or generating an instance of the first gesture recognizer or the first component gesture recognizer, setting or initializing one or more parameters of the generated instance (e.g., setting thresholds, and / or setting one or more configuration parameters to control how the generated instance operates), and connecting the generated instance to the second gesture recognizer (e.g., connecting a port of the first gesture recognizer or the first component gesture recognizer to a port of the second gesture recognizer).

[0214] Figure 5H The double-click gesture recognizer 520 is shown as being processed by the component gesture recognizers in the double-click gesture recognizer 520. Figure 5G The result of the first input event is shown in . Figure 5H In the example shown, the first input event has been determined to be a spatial event (e.g., by a spatial event listener of tap gesture recognizer 524a). The first input event describes an ongoing touch input (e.g., the first input event has a phase value of "touch start" or "touch rest") and includes the location of the received first touch input. Tap gesture recognizer 524a has processed the first input event (e.g., as described herein with reference to Figure 5C ), and thus has transitioned to the "active" state. In addition, tap gesture recognizer 524a has transmitted its current "active" state and the first input event ("space event 1") up the tree to repeater 522.

[0215] Repeater 522 includes an input count value ("count 输入”) 526, which tracks the number of times tap gesture recognizer 524 (or a corresponding instance of tap gesture recognizer 524) has succeeded. Since tap gesture recognizer (instance 1) 524a has not yet succeeded (e.g., tap gesture recognizer 524a has not yet transitioned to the “ended” state), input count value 526 has a value of zero. In addition, repeater 522 has transitioned to the “active” state in response to receiving (but not yet ending) the first touch input. Repeater 522 transmits as its output its current “active” state and the first spatial event (“spatial event 1”) received from tap gesture recognizer 524a. Since repeater 522 is the root node of double-click gesture recognizer 520, the overall state of double-click gesture recognizer 520 is the “active” state, and the overall output value of double-click gesture recognizer 520 is the overall output value of the first spatial event.

[0216] Fig.5I 5 shows the result of processing a second input event ("Event 2"), which has been received at double-click gesture recognizer 520 and is associated with Figure 5G to Figure 5H The second input event is received after receiving and processing the first input event (e.g., as described herein with respect to Figure 5G to Figure 5H The second input event has been determined to be a spatial event ("spatial event 2") similar to the first input event, except that the second input event has a phase value of "touch ended". Fig.5I As shown, tap gesture recognizer 524a has processed the second input event (e.g., as described herein with reference to Figure 5D ) and recognizes the first touch input as a first tap gesture. Tap gesture recognizer 524a has accordingly transitioned to the "finished" state and has transmitted its current "finished" state and the second input event to repeater 522. In accordance with the successful recognition of the first tap gesture by tap gesture recognizer 524a, repeater 522 increases input count value 526 to a value of 1.

[0217] In some embodiments, after repeater 522 transitions to the “active” state in response to receiving touch input, repeater 522 remains in the “active” state until tap gesture recognizer 524 (or a corresponding instance of tap gesture recognizer 524) transitions to the “failed” state, or until tap gesture recognizer 524 has succeeded for a repeat count value (“count 重复 ”) a predefined number of times. Fig.5I, since tap gesture recognizer 524a has not yet transitioned to the "failed" state, and since only one tap input has been detected, repeater 522 remains in the "active" state. Repeater 522 transmits its current "active" state and the second spatial event ("spatial event 2") received from tap gesture recognizer 524a as its output. Therefore, the overall state of double-click gesture recognizer 520 is the "active" state, and the overall output value of double-click gesture recognizer 520 is the overall output value of the second spatial event.

[0218] Figure 5J 5. In response to receiving a second touch input at the device, a third input event ("Event 3") is received at double-tap gesture recognizer 520. Since double-tap gesture recognizer 520 (and specifically, tap gesture recognizer (instance 1) 524a) has recognized the first tap input, first instance 524a of the tap gesture recognizer has been replaced by second instance 524b of the tap gesture recognizer (e.g., Figure 5B In some embodiments, in response to receiving the processed input from first instance 524a, repeater 532 removes first instance 524a of tap gesture recognizer from double-click gesture recognizer 520, indicating that the first tap gesture has been recognized, and in response to receiving the third input event, repeater 532 adds second instance 524b of tap gesture recognizer to double-click gesture recognizer 520. Repeater 522 transmits the third input event to tap gesture recognizer (instance 2) 524b for processing (e.g., as described herein with reference to FIG. 1 ). Figure 5C as described above), thereby determining whether the second touch input is a tap gesture.

[0219] Figure 5K The double-click gesture recognizer 520 is shown as being processed by the component gesture recognizers in the double-click gesture recognizer 520. Figure 5J The result of the third input event is shown in . Figure 5J to Figure 5K In the example shown, the third input event has been determined to be a spatial event (e.g., by a spatial event listener of tap gesture recognizer 524b). The third input event describes an ongoing touch input (e.g., the third input event has a phase value of "touch start" or "touch stationary") and includes the location of the received second touch input and a current timestamp t associated with the third input event. c The tap gesture recognizer 524b has processed the third input event (eg, as described herein with reference to Figure 5CInstance 2) 524b has not yet succeeded (e.g., tap gesture recognizer 524b has not yet transitioned to the "Ended" state), and thus has transitioned to the "Active" state. In addition, tap gesture recognizer 524b has communicated its current "Active" state and the third input event ("Space Event 3") to repeater 522. Since tap gesture recognizer (Instance 2) 524b has not yet succeeded (e.g., tap gesture recognizer 524b has not yet transitioned to the "Ended" state), input count value 526 of repeater 522 remains to have a value of 1.

[0220] In addition, repeater 522 monitors the amount of time that has passed since the time corresponding to the most recently recognized tap gesture. In some embodiments, repeater 522 monitors the amount of time that has passed since the start of the touch input recognized as the most recently recognized tap gesture (e.g., by identifying and storing an initial timestamp associated with a “touch start” event for the touch input). In some embodiments, repeater 522 monitors the amount of time that has passed since the end of the touch input (e.g., by identifying and storing an end timestamp associated with a “touch end” event for the touch input). For example, in Figure 5K , the repeater 522 monitors the tap gesture recognizer 524a ( Fig.5I ) The end of the most recently recognized first tap gesture (“t 轻击1结束 ”) to the current time associated with the third input event (“t c ”). Since the time from the end of the first tap gesture (“t 轻击1结束 ”) to the current time associated with the third input event (“t c ”) is less than a predefined timeout period (“T 超时 ”), and because tap gesture recognizer 524 b has not transitioned to either the “Finished” state or the “Failed” state, repeater 522 remains in the “Active” state. Repeater 522 transmits its current “Active” state and the third spatial event (“Spatial Event 3”) received from tap gesture recognizer 524 b as both its own output and as the overall output of double-tap gesture recognizer 520.

[0221] Figure 5L 4 shows the result of processing a fourth input event ("Event 4"), which has been received at double-click gesture recognizer 520 and is associated with Figure 5J to Figure 5K The fourth input event is received after receiving and processing the third input event (eg, as described herein with respect to Figure 5J to Figure 5K The fourth input event has been determined to be a spatial event ("spatial event 4") similar to the third input event, except that the fourth input event has a phase value of "touch end". Figure 5L As shown, tap gesture recognizer 524b has processed the fourth input event (e.g., as described herein with reference to Figure 5D ) and recognizes the second touch input as a second tap gesture. Tap gesture recognizer 524b has accordingly transitioned to the "ended" state and has transmitted its current "ended" state and the fourth input event to repeater 522. In accordance with the successful recognition of the second tap gesture by tap gesture recognizer 524b, repeater 522 increases input count value 526 to a value of 2.

[0222] In addition, from the end of the first tap gesture ("t 轻击1结束 ”) to the end of the second tap gesture (“t 轻击2结束 ”) is less than a predefined timeout period (“T 超时 Since the timing requirements are still met, and since tap gesture recognizer 524 b has transitioned to the “Finished” state, repeater 522 transitions to the “Finished” state. Repeater transmits its current “Finished” state and the fourth spatial event (“Spatial Event 4”) received from tap gesture recognizer 524 b as both its own output and as the overall output of double-click gesture recognizer 520. The output of “Finished” by repeater 522 indicates that double-click gesture recognizer 520 has successfully recognized the double-click gesture.

[0223] In some embodiments, if any instance of tap gesture recognizer 524 has transitioned to the "failed" state, repeater 522 will transition to the "failed" state and will not recognize a double-tap gesture. In addition, if a predefined timeout period ("T 超时 ”), repeater 522 will transition to the “failed” state and will not recognize the double-tap gesture.

[0224] Figure 5M to Figure 5Q An exemplary (dual input) pinch gesture recognizer 530 and its operation in processing input is shown. In some embodiments, pinch gesture recognizer 530 or an instance thereof is included in a particular view of a user interface and is used to recognize a dual input pinch gesture performed using touch input in the particular view of the user interface. Figure 5M Pinch gesture recognizer 530 is shown to include multiple component gesture recognizers that define the requirements for a pinch gesture, including:

[0225] a first drag component ("drag 1") 532, which requires knowing the location of a touch input received on the touch-sensitive surface by monitoring input events that include location information (similar to a spatial event listener), except that the drag component also monitors the amount of movement of the touch input; and

[0226] A second drag component (“drag 2”) 534 , which behaves similarly to the first drag component 532 , but is a different drag component (or a different instance of a drag component) than the first drag component 532 .

[0227] Pinch gesture recognizer 530 also includes parallel combiner 536. Parallel combiner 536 controls the transmission of input events to first drag component 532 and second drag component 534. Specifically, parallel combiner assigns different touch inputs to each drag component so that each drag component receives and processes input events only for its assigned touch inputs. The state of parallel combiner 536 depends on the corresponding state of the drag component, and the output value of parallel combiner 536 is the output value of the drag component, as described in more detail herein.

[0228] Additionally, pinch gesture recognizer 530 includes pinch component 538. Pinch component 536 determines zoom (e.g., for controlling the zoom of a displayed user interface or user interface element when a pinch gesture is detected) and, optionally, zoom speed (e.g., for continuing to adjust the zoom of a user interface or user interface element after a detected pinch gesture has ended).

[0229] Figure 5M Also shown is that in response to receiving multiple touch inputs at the device, two input events are received at pinch gesture recognizer 530 (e.g., from event distributor module 174). Specifically, pinch gesture recognizer 530 receives a first input event ("Event 1") in response to a first received touch input ("Input 1"), and receives a second input event ("Event 2") in response to a second received touch input ("Input 2"). Pinch component 538, which is a root node of pinch gesture recognizer 530, receives the input events and transmits the two input events to parallel combiner 536. Parallel combiner 536 distributes the first touch input to first drag component 532, and transmits the first input event of the first touch input to first drag component 532. In addition, parallel combiner 536 distributes the second touch input to second drag component 532, and transmits the second input event of the second touch input to second drag component 532.

[0230] Figure 5N The process performed by the component gesture recognizers in pinch gesture recognizer 530 is shown. Figure 5M The result of the input event is shown in . Figure 5N In the example shown, the first input event is a spatial event describing a first touch input that is a newly received touch input (e.g., the first input event has a phase value of "touch start"). The first drag component 532 has identified and stored an initial position 540 ("p 1iIn addition, the first drag component 532 determines the current position of the first touch input (“p 1c ”) and the initial position of the first touch input (“p 1i ”) to monitor the amount of movement of the first touch input on the touch-sensitive surface. When the first touch input is initially detected, at the current position (“p 1c ”) is equal to the initial position (“p 1i ”), the touch input has not moved on the touch-sensitive surface, so the movement amount (“|p 1c –p 1i |”) is zero. Since the movement amount is less than the minimum threshold amount (“D min,th ”), so the first drag component 532 is in the “possible” state to indicate that dragging behavior (e.g., including movement of the input) may occur but has not yet been detected. In some embodiments, the “possible” state is an available state of a gesture recognition component that requires a minimum threshold amount of movement of the touch input. While the input is continuously detected and before the detected input has moved the minimum threshold amount (e.g., for subsequent input events with stage value “touch stationary”), such a component remains in the “possible” state, and transitions to the “active” state when the detected input has moved the minimum threshold amount (subject to the component’s other requirements, if any, being met).

[0231] The first drag component 532 transmits the current “possible” state of the first drag component 532 and a plurality of values ​​identified from the first input event (or determined based on information in the first input event) to the parallel combiner 536, including the initial timestamp of the first touch input (“t 1i ”), the instantaneous speed of the first touch input (“v 1i ”), the initial position of the first touch input (“p 1i ”) and the current position of the first touch input (“p” when the first touch input is initially detected). 1i ”).

[0232] Similarly, the second input event is also a spatial event describing an ongoing touch input. The second drag component has identified and stored the initial position 542 (“p 2i ”). The amount of movement due to the second touch input (“|p 2c –p 2i |”) is less than the minimum threshold amount (“D min,th ”), so the second drag component 534 is in the “possible” state. The second drag component 534 transmits the current “possible” state of the second drag component 532 and multiple values ​​identified from the second input event (or determined based on information in the second input event) to the parallel combiner 536, including the initial timestamp of the second touch input (“t 2i”), the instantaneous speed of the second touch input (“v 2i ”), the initial position of the second touch input (“p 2i ”) and the current position of the second touch input (“p” when the second touch input is initially detected). 2i ”).

[0233] The parallel combiner 536 receives the "possible" state and values ​​from the first drag component 532 and the second drag component 534. Since both drag components are in the "possible" state, the parallel combiner 536 is also in the "possible" state. The parallel combiner 536 transmits its current "possible" state and all values ​​received from the first drag component 532 and the second drag component 534 to the pinch component 538. In some embodiments, the parallel combiner 536 receives an array of values ​​from each drag component. In some embodiments, the parallel combiner 536 combines the received arrays into a larger array that includes all values ​​received from the drag components.

[0234] Pinch component 538 receives a “Maybe” state and value from parallel combiner 536. Since parallel combiner 536 is in the “Maybe” state, pinch component 538 is also in the “Maybe” state. Pinch component 538 determines and transmits a zoom value based on the values ​​received from parallel combiner 536 (e.g., based on the position of each touch input relative to its initial position), and optionally determines and transmits a zoom speed. Since pinch component 538 is the root node of pinch gesture recognizer 530, the overall state of pinch gesture recognizer 530 is the “Maybe” state, and the overall output value of pinch gesture recognizer 530 is the overall output value of pinch component 538.

[0235] Fig.5O An example of successful recognition of a pinch gesture is shown. Specifically, Fig.5O shows the processing relative to Figure 5M to Figure 5N The result of a second pair of input events received after the first input event and the second input event. The second pair of input events includes a third input event associated with the first touch input being maintained on the touch-sensitive surface and moving at least a minimum threshold distance on the touch-sensitive surface and a fourth input event associated with the second touch input being maintained on the touch-sensitive surface and moving at least a minimum threshold distance on the touch-sensitive surface. Fig.5O In the example shown, the third input event and the fourth input event are spatial events with a phase value of "touch move". The first drag component 532 has identified the third input event as a spatial event of the ongoing first touch input (as indicated by the "touch move" phase value), and has determined the movement amount ("p 1c –p 1i ”) is greater than the minimum threshold value (“D min,th”). Therefore, the first drag component 532 has transitioned to the “active” state. The first drag component 532 transmits the current “active” state of the first drag component 532 and the value identified from the third input event (or determined based on the information in the third input event) to the parallel combiner 536, including the current timestamp of the first touch input (“t 1c ”), the current instantaneous speed of the first touch input (“v 1c ”), the initial position of the first touch input (“p 1i ”) and the current position of the first touch input (“p 1c ”).

[0236] Similarly, the second drag component 534 has identified the fourth input event as a spatial event of the ongoing second touch input (as indicated by the "touch move" stage value), and has determined the movement amount ("p 2c –p 2i ”) is greater than the minimum threshold value (“D min,th ”). Therefore, the second drag component 534 has transitioned to the “active” state. The second drag component 534 transmits the current “active” state of the second drag component 534 and the value identified from the fourth input event (or determined based on the information in the fourth input event) to the parallel combiner 536, including the current timestamp (“t 2c ”), the current instantaneous speed of the second touch input (“v 2c ”), the initial position of the second touch input (“p 2i ”) and the current position of the second touch input (“p 2c ”).

[0237] Since both drag components have transitioned to the "active" state, parallel combiner 536 has also transitioned to the "active" state. In some embodiments, a pinch gesture can be performed by moving only one touch input while the other touch input remains stationary. Thus, in some embodiments, parallel combiner 536 transitions to the "active" state in response to at least one of its child drag components transitioning to the "active" state (e.g., even if another child drag component remains in the "possible" state). Fig.5O , parallel combiner 536 transmits its current “active” state and all values ​​received from first drag component 532 and second drag component 534 to pinch component 538 .

[0238] Since parallel combiner 536 is in the "active" state, pinch component 538 is also in the "active" state. Pinch component 538 determines and transmits a (current) zoom value based on the values ​​received from parallel combiner 536 (e.g., based on the current position of each touch input relative to its initial position), and optionally determines and transmits a (current) zoom speed. Therefore, the overall state of pinch gesture recognizer 530 is the "active" state, and the overall output value of pinch gesture recognizer 530 is the overall output value of pinch component 538.

[0239] Figure 5P Shows Fig.5O An example of continuous successful recognition of the pinch gesture recognized in . Specifically, Figure 5P Shows that relative to Fig.5O The processing results of the third pair of input events received after the second pair of input events described above. The third pair of input events includes a spatial event associated with the first touch input being maintained at a position at least a minimum threshold distance from its initial position on the touch-sensitive surface, and a spatial event associated with the second touch input being lifted off the touch-sensitive surface (e.g., with a phase value of "touch end"). Therefore, the first drag component 532 remains in the "active" state and transmits the current "active" state and associated value to the parallel combiner 536, as relative to Fig.5O described.

[0240] The second drag component 534 has identified the spatial event as being associated with a completed touch input (as indicated by the "touch ended" phase value), and has therefore transitioned to the "ended" state. The second drag component 534 transmits its current "ended" state and associated values ​​identified from (or determined based on information in) the "touch ended" input event, including the end timestamp ("touch ended" of the second touch input). 2e ”), the instantaneous velocity when the second touch input is lifted off (“v 2e ”), the initial position of the second touch input (“p 2i ”) and the end position of the second touch input when it is lifted off (“p 2e ”).

[0241] In some embodiments, a pinch gesture recognizer remains "active" as long as one touch input remains "active," even if the other touch input has "ended." That is, a pinch gesture recognizer that is "active" when a user performs a pinch gesture using two fingers in contact with the touch-sensitive surface remains "active" even after the user lifts one finger off the touch-sensitive surface (as long as the other finger remains on the touch-sensitive surface). The user can place the lifted finger back on the touch-sensitive surface to resume the pinch gesture. Thus, although second drag component 534 has transitioned to Figure 5P, but the parallel combiner 536 remains in the "active" state. The parallel combiner 536 transmits its current "active" state and all values ​​received from the first drag component 532 and the second drag component 534 to the pinch component 538.

[0242] Since parallel combiner 536 remains in the “active” state, pinch component 538 also remains in the “active” state and determines and transmits a (current) zoom value, and optionally a (current) zoom speed, based on the values ​​received from parallel combiner 536. Therefore, the overall state of pinch gesture recognizer 530 remains in the “active” state, and the overall output value of pinch gesture recognizer 530 is the overall output value of pinch component 538.

[0243] Figure 5Q Shows Figure 5O to Figure 5P An example of the completion of the pinch gesture identified in . Specifically, Figure 5Q 5 and is associated with the lift-off of the first touch input from the touch-sensitive surface (e.g., having a phase value of "touch end"). Thus, first drag component 534 has transitioned to the "end" state. First drag component 534 transmits its current "end" state and associated values ​​identified from (or determined based on information in) the "touch end" input event, including an end timestamp ("touch end" of the first touch input). 1e ”), the instantaneous velocity when the first touch input is lifted off (“v 1e ”), the initial position of the first touch input (“p 1i ”) and the end position when the first touch input is lifted off (“p 1e ”).

[0244] Since both drag components have transitioned to the "ended" state, parallel composer 536 has also transitioned to the "ended" state. Parallel composer 536 transfers its current "ended" state and all values ​​received from first drag component 532 and second drag component 534 to pinch component 538. Thus, the overall state of pinch gesture recognizer 530 is the "ended" state, indicating that the pinch gesture has been recognized and has ended, and the overall output value of pinch gesture recognizer 530 is the overall output value of pinch component 538. The zoom speed is optionally used to continue adjusting the zoom of the user interface or user interface element to which the pinch gesture is directed, even after the pinch gesture has ended.

[0245] In some embodiments, the corresponding view of the user interface includes a rotation gesture recognizer having the same component gesture recognizers as pinch gesture recognizer 530 described above, except that pinch component 538 is replaced by a rotation component (not shown). Additionally, the operation of the rotation gesture recognizer and its component recognizers is the same as described above for pinch gesture recognizer 530, except that the rotation component determines an angle or amount of rotation, and optionally an angular velocity or rotational velocity, based on position values ​​and, optionally, velocity values ​​received from parallel components 536 (when at least one of the drag components 532 of the rotation gesture recognizer is active).

[0246] FIG. 6A to FIG. 6C An example of a gesture recognition hierarchy generated from a view hierarchy of views in a user interface based on received input is shown.

[0247] Fig. 6A Device 100 is shown displaying user interface 600. User interface 600 includes multiple views 602 (labeled "View A"), 606 (labeled "View B"), 612 (labeled "View C"), and 616 (labeled "View D"). View 602 is the highest view in view hierarchy 630 and includes long press gesture recognizer 604. View 606 is a child view of view 602 in view hierarchy 630 and includes scroll gesture recognizer 608 and pinch gesture recognizer 610 (e.g., similar to Figure 5M 606 in view hierarchy 630 and includes a first tap gesture recognizer 614 (e.g., similar to Figure 5A 606 in view hierarchy 630 and includes a second tap gesture recognizer 618 (e.g., similar to Figure 5A In some embodiments, second tap gesture recognizer 618 includes instances of the same component gesture recognizer and first tap gesture recognizer 614, and thus those component gesture recognizers are reused two or more times in concurrently instantiated gesture recognizers. More generally, in some embodiments, two or more instances of a corresponding (or first) component gesture recognizer are included in concurrently instantiated gesture recognizers; and in some embodiments, two or more instances of each component gesture recognizer in a plurality of component gesture recognizers are included in concurrently instantiated gesture recognizers (e.g., in a gesture recognition hierarchy discussed below with respect to corresponding user interface 600).

[0248] Fig. 6AAlso shown is touch input 620 received at device 100. Device 100 (e.g., using hit view determination module 172) determines that view 612 is a hit view for touch input 620 (e.g., view 612 is the lowest view in view hierarchy 630 where touch input 620 has occurred, and therefore should receive priority for processing input events associated with touch input 620). Accordingly, view 612 is highlighted in view hierarchy 630. In some embodiments, gesture recognizers 604, 608, 610, 614, and 618 are created (e.g., instantiated) and executed on device 100 regardless of whether user input has been received. In some embodiments, in response to the device receiving user input (e.g., touch input 620), one or more of gesture recognizers 604, 608, 610, 614, and 618 are created and executed on the device.

[0249] In response to receiving touch input 620, device 100 generates gesture recognition hierarchy 640 based on view hierarchy 630. In some embodiments, as shown in FIG. Fig. 6A As shown, the generated gesture recognition hierarchy includes only gesture recognizers for views corresponding to the location of the touch input (e.g., the hit view and all of its predecessor views in the view hierarchy). Thus, gesture recognition hierarchy 640 includes only gesture recognizers for hit view 612, parent view 606 of view 612, and grandparent view 602 of view 612—namely, first tap gesture recognizer 614 of hit view 612, scroll gesture recognizer 608 and pinch gesture recognizer 610 of view 606, and long press gesture recognizer 604 of view 602. In some embodiments, the generated gesture recognition hierarchy includes gesture recognizers for all views in the view hierarchy (e.g., such a gesture recognition hierarchy would also include second tap gesture recognizer 618 of view 616, which does not correspond to the location of touch input 620).

[0250] In addition, because view 612 has been determined to be the hit view, the gesture recognizers of hit view 612 have the highest priority for processing touch input 620 over the gesture recognizers of other views in the view hierarchy. Therefore, gesture recognition hierarchy 640 assigns the highest priority to first tap gesture recognizer 614 using exclusive combiner 642. An exclusive combiner defines the relationship between multiple child (component) gesture recognizers by requiring the left-hand child gesture recognizer to fail in order for the right-hand child gesture recognizer to succeed. For an exclusive combiner with more than two child gesture recognizers, a corresponding child gesture recognizer is allowed to succeed only if every child gesture recognizer to its left fails. Therefore, because all gesture recognizers in gesture recognition hierarchy 640 except tap gesture recognizer 614 are on the right-hand side of exclusive combiner 542, the gesture recognition hierarchy requires first tap gesture recognizer 614 to fail in order for any other gesture recognizer in hierarchy 640 to succeed. It should be noted that the terms "right" and "left" in this discussion are different but interchangeable, and similarly, the terms "right hand" and "left hand" in this discussion are different but interchangeable. More generally, in some embodiments, each different input position of an exclusive combiner 642 or 644 is assigned a corresponding priority, and a sub-gesture recognizer connected to a particular input of an exclusive combiner 642 or 644 is allowed to succeed only if all sub-gesture recognizers of lower priority inputs connected to the exclusive combiner 642 or 644 have failed.

[0251] Because view 606 is the parent view of the hit view (e.g., the second-lowest view in view hierarchy 630 corresponding to touch input 620), the gesture recognizers of view 606 have the second-highest priority for processing touch input 620. Therefore, both gesture recognizers of view 606 (e.g., scroll gesture recognizer 608 and pinch gesture recognizer 610) have a priority defined by exclusive combiner 644 over other gesture recognizers in gesture recognition hierarchy 640 (e.g., long press gesture recognizer 604). In some embodiments, when a single view includes multiple gesture recognizers, a combiner is used to define a priority or other relationship between the multiple gesture recognizers. Fig. 6A In the example shown, simultaneous combiner 646 is used to define a relationship between scroll gesture recognizer 608 and pinch gesture recognizer 610 for view 606 such that scroll gesture recognizer 608 and pinch gesture recognizer 610 are allowed to succeed simultaneously.

[0252] Figure 6B Similar to Fig. 6A , the difference is Figure 6B 6. Touch input 622 is shown received at device 100, resulting in view 616 being determined to be the hit view (rather than Fig. 6A 612 in the hit view hierarchy 630). Therefore, view 616 is highlighted in view hierarchy 630. In response to receiving touch input 622, device 100 generates gesture recognition hierarchy 650 based on view hierarchy 630. Gesture recognition hierarchy 650 includes gesture recognizers of hit view 616, parent view 606 of view 616, and grandparent view 602 of view 616, namely, second tap gesture recognizer 618 that hits view 616 (instead of first tap gesture recognizer 614 of view 612), scroll gesture recognizer 608 and pinch gesture recognizer 610 of view 606, and long press gesture recognizer 604 of view 602. Because view 616 has been determined to be the hit view, second tap gesture recognizer 618 has the highest priority for processing touch input 622 over gesture recognizers of other views in the view hierarchy. Thus, gesture recognition hierarchy 650 is similar to gesture recognition hierarchy 640 ( FIG. A ), except that second tap gesture recognizer 618 of view 616 replaces first tap gesture recognizer 614 of view 612 .

[0253] Figure 6C Similar to Fig. 6A , the difference is Figure 6C A dual-contact touch input 624 is shown received at device 100. In some embodiments, a hit view for a multi-contact touch input is determined based on the centroids of the multiple contacts. Figure 6C In the example shown, the hit view determined for touch input 624 is view 606 (which corresponds to a position approximately halfway between the two contacts of touch input 624). Therefore, view 606 is highlighted in view hierarchy 630. In response to receiving touch input 624, device 100 generates gesture recognition hierarchy 660. Because view 606 has been determined to be the hit view, both the scroll gesture recognizer 608 and the pinch gesture recognizer 610 of view 606 (which are allowed to succeed simultaneously by simultaneous combiner 664) have the highest priority for processing touch input 624 over gesture recognizers of other views in the view hierarchy (e.g., long press gesture recognizer 604). Therefore, the simultaneous combiner is on the left-hand side of exclusive combiner 662, while the long press gesture recognizer 604 is on the right-hand side. In other words, gesture recognition hierarchy 660 is similar to gesture recognition hierarchy 640 (FIG. A), except that first tap gesture recognizer 614 and its associated exclusive combiner 642 are omitted.

[0254] In some embodiments, the hit view for touch input 624 is determined to be view 612 because at least one contact of touch input 624 corresponds to view 612 and because view 612 is below view 606. In some such embodiments, gesture recognition hierarchy 660 also includes first tap gesture recognizer 614 (e.g., a first tap gesture recognizer that is located adjacent to the first tap gesture recognizer in the gesture recognition hierarchy). Fig. 6A The gesture recognition hierarchy 640 is the same as above).

[0255] 7A to 7E is a flow chart illustrating a method 700 for processing input using a gesture recognizer assembled from component gesture recognizers according to some embodiments. The method 700 is performed in an electronic device (e.g., Figure 3 Device 300 or Figure 1A The portable multifunction device 100 is executed at the electronic device having an input device (e.g., Figure 1A Touch screen 112, Figure 3 touchpad 355 or keyboard / mouse 350), an optional display (e.g., Figure 1A Touch screen 112 or Figure 3 340), and optionally one or more sensors for detecting intensity of contact with an input device including a touch-sensitive surface (e.g., Figure 1A In some embodiments, the display is a touch screen display having a touch-sensitive surface on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 700 are optionally combined and / or the order of some operations is optionally changed.

[0256] As described below, method 700 provides an electronic device with an intuitive way to process detected inputs using a modular approach that divides a larger set of input event processing requirements into smaller subsets of event processing requirements that are associated with and processed by component gesture recognizers that make up a gesture recognizer, so that the smaller subsets of event processing requirements can be processed faster than the larger set. This modular approach increases the number, degree, and complexity of gestures that can potentially be recognized. Reusing (instances of) modules across multiple gesture recognizers provides consistency in gesture recognition capabilities. In addition, compared to methods that process inputs using monolithic gesture recognizers (i.e., gesture recognizers that are not modular), the modular approach provides design flexibility because modules can be added to or removed from gesture recognizers, or individual modules in a gesture recognizer can be replaced by modified versions without having to design a completely new monolithic gesture recognizer. The modular approach also reduces the computational load imposed on the electronic device by touch and other inputs and gestures, which in turn improves device responsiveness and reduces latency. Reducing latency in responding to inputs also reduces user errors when operating / interacting with the device, such as repeated or irrelevant inputs made by the user while waiting for the device to respond to the user's earlier input. Reducing the amount and / or extent of input from the user enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0257] The device detects ( 702 ) input via an input device (eg, touch input on a touch-sensitive surface, mouse input using one or more buttons of a mouse, etc.).

[0258] In response to detecting the input, the device monitors (704) the input using a gesture recognizer. The gesture recognizer includes (706) a plurality of component gesture recognizers that define requirements for recognizing a gesture (e.g., the gesture recognizer is an assembled gesture recognizer assembled using a plurality of component gesture recognizers from a set of available component gesture recognizers). A first component gesture recognizer from the plurality of component gesture recognizers defines a first requirement for recognizing a gesture (e.g., requirements with respect to location, contact strength, minimum duration, maximum duration, distance (optionally with minimum and / or maximum distance requirements), etc.), and a second component gesture recognizer from the plurality of component gesture recognizers defines a second requirement for recognizing a gesture (e.g., requirements with respect to location, contact strength, minimum duration, maximum duration, distance (optionally with minimum and / or maximum distance requirements), etc.). In some embodiments, a gesture recognizer is formed (e.g., created or instantiated) in response to detecting the input. In some embodiments, the gesture recognizer is associated with a view of a plurality of views of a software application executing on the electronic device, and the gesture recognizer is formed in accordance with determining that a location of the detected input corresponds to the view with which the gesture recognizer is associated. In some embodiments, the gesture recognizer specifically corresponds to the aforementioned view, or includes a component of an application that responds to touch input in the aforementioned view. FIG. 5A to FIG. 5Q An exemplary gesture recognizer assembled from multiple component gesture recognizers is shown and described.

[0259] Based on determining that the input satisfies the first gesture recognition criteria including the first requirement for the first component gesture recognizer and the second requirement for the second component gesture recognizer, the device recognizes (708) the first gesture using the gesture recognizer. In some embodiments, the set of available component gesture recognizers includes component gesture recognizers that recognize input events (e.g., touch events, mouse events, etc.) with specific attributes (e.g., including position information and / or translation (movement) information). Examples of component gesture recognizers that recognize input events include spatial event listeners (such as Figure 5A spatial event listener 502), translation event listener and drag event listener (such as Figure 5M Drag event listeners 532 and 534, etc.).

[0260] In some embodiments, the set of available component gesture recognizers includes a method for determining whether a particular attribute of an input (e.g., current position, current timestamp, current intensity, etc., optionally in combination with one or more previously determined attributes such as a starting position, a starting timestamp, etc.) satisfies a particular requirement for recognizing a gesture (e.g., a distance requirement (see Figure 5A 506), (timing) duration requirements (see Figure 5AIn some embodiments, the set of available component gesture recognizers includes a wrapper that includes one or more other component gesture recognizers, and the output of the wrapper depends on the included component gesture recognizers (e.g., a repeating component gesture recognizer (such as a Figure 5G Repeater 522 of the embodiment of the present invention) and / or depending on having more than the included component gesture recognizers (e.g., a pause component gesture recognizer (such as Figure 5A The leading pause 508)) is the result of other gesture recognizers or component gesture recognizers of higher priority.

[0261] In some embodiments, the set of available component gesture recognizers includes a combiner, wherein a respective combiner combines results generated by two or more component gesture recognizers (sometimes referred to as component gesture recognizer outputs or recognition results, or combiner inputs), and uses the component gesture recognizers combined by the combiner (e.g., gating a component gesture recognizer such as a gesture recognizer) based on the results thereof. Figure 5A strobe combiners 510 and 512), simultaneous component gesture recognizers such as Fig. 6A simultaneous combiner 646 of ), exclusive component gesture recognizers (such as Fig. 6A exclusive combiners 642 and 644 of ), sequential component gesture recognizers, or parallel component gesture recognizers (such as Figure 5M The relationship between the parallel combiner 536 ), etc., as described in more detail herein, defines the requirements for recognizing gestures.

[0262] In some embodiments, determining whether the input satisfies the first gesture recognition criteria includes: determining, using the first component gesture recognizer, whether the input satisfies first requirements defined by the first component gesture recognizer; determining, using the second component gesture recognizer, whether the input satisfies second requirements defined by the first component gesture recognizer; and determining whether the input satisfies the first gesture recognition criteria based on respective determinations made using the first component gesture recognizer and the second component gesture recognizer.

[0263] In some embodiments, a first component gesture recognizer has (710) a current state among a plurality of available states (e.g., determined based on whether the input satisfies a first requirement of the first component gesture recognizer); and a second component gesture recognizer has a current state among a plurality of available states (e.g., determined based on whether the input satisfies a second requirement of the second component gesture recognizer). In some embodiments, the available states of the respective component gesture recognizers include "possible", "active", "finished", and "failed". Examples of transitions between available states of the component gesture recognizers are described herein with respect to FIG. 5A to FIG. 5QHaving independent states for different component gesture recognizers is consistent with a modular design of gesture recognizers, which simplifies event handling and reduces the computational load on the device when processing input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0264] In some embodiments, the device determines (712) a current state of the gesture recognizer (e.g., a current state among the plurality of available states (e.g., "possible," "active," "ended," and "failed")) based on the current state of the first component gesture recognizer and the current state of the second component gesture recognizer. In some embodiments, the overall state of the gesture recognizer is determined based on a combiner rule. In one example, the gesture recognizer includes a first component gesture recognizer and a second component gesture recognizer that are associated using a gating component gesture recognizer (also referred to as a gating combiner). In this example, when the second component gesture recognizer is in a failed state, the gating combiner is in a failed state; and when the second component gesture recognizer is not in a failed state, the gating combiner passes the state of the first component gesture recognizer (e.g., the gating combiner operates similarly to a logical AND gate). As described herein with reference to FIG. 5A to FIG. 5F As described above, the current overall state of tap gesture recognizer 500 is determined based on the current state of gate combiner 512 (the state of the gate combiner is in turn determined based on the other component gesture recognizers 502, 504, 506, 508, and 510 in tap gesture recognizer 500). Figure 5G As described, the current overall state of double-tap gesture recognizer 520 is determined based on the current states of its component gesture recognizers 522 and 524. Determining the overall state of a gesture recognizer based on the states of its component gesture recognizers is consistent with the modular design of gesture recognizers, which simplifies event handling and reduces the computational load on the device when processing input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping users achieve intended results and by reducing user errors when operating / interacting with the device).

[0265] In some embodiments, monitoring input with a gesture recognizer includes (714): receiving, at the gesture recognizer, first information describing the input at a first point in time (e.g., corresponding to a touch event at the first point in time); processing the first information with a first component gesture recognizer to determine a current state of the first component gesture recognizer among the plurality of available states for the first point in time; and processing the first information with a second component gesture recognizer to determine a current state of the second component gesture recognizer among the plurality of available states for the first point in time, as described herein with reference to Figure 5C . In some embodiments, monitoring input with a gesture recognizer further includes receiving, at the gesture recognizer, second information describing a change in the input at a second time point that is later than the first time point. In some embodiments, the gesture recognizer receives and processes the second information at the second time point after the gesture recognizer has received and processed the first information. In some embodiments, monitoring input with a gesture recognizer further includes: processing the second information with a first component gesture recognizer to determine a current state of the first component gesture recognizer among the multiple available states for the second time point (e.g., updating its state from a previous state such as the state of the first component gesture recognizer at the first time point); and processing the second information with a second component gesture recognizer to determine a current state of the second component gesture recognizer among the multiple available states for the second time point (e.g., updating its state from a previous state such as the state of the second component gesture recognizer at the first time point), as described herein with reference to, respectively, FIG. 5D to FIG. 5F The processing result of the second input event, the third input event, or the fourth input event shown is described. Using independent component gesture recognizers whose states evolve over time to monitor input over time is consistent with the modular design of gesture recognizers, which simplifies event processing and reduces the computational load on the device when processing input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping users achieve intended results and by reducing user errors when operating / interacting with the device).

[0266] In some embodiments, one or more component gesture recognizers in the gesture recognizer include ( 716 ) one or more parameters describing the input (e.g., a point value describing the location of the input, or a timestamp describing a time associated with the input) (e.g., Figure 5CIn some embodiments, monitoring the input using the gesture recognizers includes determining values ​​of the one or more parameters using the one or more component gesture recognizers in the gesture recognizers. In some embodiments, the first gesture recognition criteria include a requirement that the corresponding parameter satisfies a corresponding threshold in order to recognize the first gesture. For example, the first gesture recognition criteria includes a requirement that the distance traveled by the input (e.g., the amount the touch input moves on the touch-sensitive surface) satisfies a corresponding distance threshold (e.g., the input must move at least a first threshold amount, the input must move no more than a second threshold amount, or the input must move at least the first threshold amount but no more than the second threshold amount) (defined by the corresponding component gesture recognizer).

[0267] In some embodiments, the gesture recognizer generates / sends an event to a software application based on a change in the one or more parameters. In some embodiments, the gesture recognizer generates / sends one or more events to a software application before determining whether the input satisfies the first gesture recognition criteria. In some embodiments, a corresponding event of the one or more events includes a phase value corresponding to the current state of the gesture recognizer (e.g., "possible", "active", "ended", or "failed"). Using the component gesture recognizers in the gesture recognizer to determine the values ​​of the parameters describing the input provides the gesture recognizer with information about the input needed to process the input, which simplifies event processing and reduces the computational load on the device when processing the input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve the desired results and by reducing user errors when operating / interacting with the device).

[0268] In some embodiments, the gesture recognizer is (718) a gesture recognition tree (e.g., a tree data structure) having a plurality of nodes, and each node of the gesture recognition tree includes a corresponding component gesture recognizer (e.g., as described herein with reference to FIG. 5A to FIG. 5Q In some embodiments, monitoring input using gesture recognizers includes transmitting information describing the input at the corresponding point in time from a component gesture recognizer at a corresponding node having a child node in a gesture recognition tree to a component gesture recognizer at the child node in the gesture recognition tree (e.g., as described herein with reference to Figure 5B , Figure 5G , Figure 5J and Figure 5MIn some embodiments, a (parent) component gesture recognizer (e.g., a combiner) with multiple child nodes transmits information describing an input (e.g., an input event) to its child component gesture recognizers based on the properties of the (parent) component gesture recognizer. Some combiners, such as simultaneous combiners and exclusive combiners, transmit information about an input (e.g., an input event) to all of its child component gesture recognizers (e.g., as described herein with reference to Fig. 6A In some embodiments, a combiner (e.g., an exclusive combiner) sends a particular input event to all of the sub-component gesture recognizers of the combiner by sending the particular input event to one sub-component gesture recognizer at a time for each sub-component gesture recognizer to process the input event in turn.

[0269] Some combiners, such as sequential combiners and parallel combiners, transmit information about an input to only a subset of its sub-component gesture recognizers. For example, a sequential combiner transmits corresponding information about an input (e.g., a corresponding input event) to (only) one corresponding sub-component gesture recognizer. Upon detecting a corresponding input, the sequential combiner first sends an input event about the corresponding input to its first sub-component gesture recognizer. After the first sub-component gesture recognizer succeeds, the sequential combiner sends subsequent input events to its second sub-component gesture recognizer, and so on for each additional sub-component gesture recognizer. If any sub-component gesture recognizer fails, the sequential combiner fails. In another example, a parallel combiner associates one or more of its sub-component gesture recognizers with one or more corresponding inputs, and transmits information about the corresponding input only to the associated sub-gesture recognizers. In the event that multiple touches are detected on the touch-sensitive surface, the first sub-gesture recognizer of the parallel combiner is associated with the first touch of the multiple touches, the second sub-gesture recognizer of the parallel combiner is associated with the second touch of the multiple touches, and so on for each touch of the multiple touches. In this example, input events describing the first touch are sent by the parallel combiner to the first child gesture recognizer (and not to the second child gesture recognizer or any other child gesture recognizer); similarly, input events describing the second touch are sent by the parallel combiner to the second child gesture recognizer (and not to the first child gesture recognizer or any other child gesture recognizer). Figure 5M to Figure 5Q Parallel combiner 536 describes an example of parallel combiner behavior.

[0270] In some embodiments, monitoring input using gesture recognizers further comprises determining, based on information describing the input at the corresponding point in time and based on determining that the corresponding node has one or more child nodes, a state of a component gesture recognizer at the corresponding node in the gesture recognition tree based on a corresponding state of the corresponding component gesture recognizers at the one or more child nodes, the corresponding state being determined based on the information describing the input at the corresponding point in time. In some embodiments, the state of the corresponding component gesture recognizer is further based on a corresponding value determined using the corresponding component gesture recognizers at the one or more child nodes. That is, the state of the corresponding component gesture recognizer is determined based on one or more of the information about the input (e.g., the input event), the determined state (if any) of its child component gesture recognizers (based on the information about the input), and / or the determined value (if any) of its child component gesture recognizers.

[0271] By defining relationships between component gesture recognizers, including, for example, setting priorities between component gesture recognizers, arranging the component gesture recognizers in a tree structure enables an organized and orderly flow of input events to be processed, which simplifies input event processing and reduces the computational load on the device when processing input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0272] In some embodiments, monitoring the input with the gesture recognizers further comprises (720) transmitting the determined state of the component gesture recognizer at the child node in the gesture recognition tree to the component gesture recognizer at the parent node of the child node in the gesture recognition tree. In some embodiments, determining whether the input satisfies first gesture recognition criteria for recognizing a first gesture comprises determining a state of the gesture recognition tree based on the determined states of the component gesture recognizers in the gesture recognition tree (e.g., as described herein with reference to FIG. 5C to FIG. 5F , FIG. 5H to FIG. 5I , Figure 5K to Figure 5L and Figure 5N to Figure 5Q Determining the state of a parent component gesture recognizer based on the state of a child component gesture recognizer enables more complex input processing without increasing the amount of processing required at the parent component gesture recognizer. Dividing a larger set of processing requirements into a smaller subset of processing requirements simplifies event processing and reduces the computational load on the device when processing input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0273] In some embodiments, monitoring input using gesture recognizers includes (722) determining values ​​of one or more component gesture recognizers at a corresponding node in a gesture recognition tree based on information describing the input at a corresponding point in time. In some embodiments, the values ​​are determined based on states and / or values ​​of child component gesture recognizers. In some embodiments, based on determining that the component gesture recognizers at the child nodes have determined values, determining the states of the corresponding component gesture recognizers at the corresponding nodes is further based on the determined values ​​of the component gesture recognizers at the child nodes of the corresponding node.

[0274] For example, a component gesture recognizer that defines a threshold requirement for a parameter has a child component gesture recognizer that determines a value for the parameter, and the (parent) component gesture recognizer determines its state based on whether the determined value from the child component gesture recognizer satisfies the threshold requirement. In some embodiments, the (parent) component gesture recognizer also determines a value for the parameter, and in such cases, the component gesture recognizer determines its value based on the determined value of its child component gesture recognizer (or based on the corresponding values ​​of two or more child component gesture recognizers). For another example, a component gesture recognizer (e.g., Figure 5N The pinch component 538 of the embodiment of the present invention is based on (directly or indirectly) receiving the gesture recognizer from the first subcomponent (e.g., Figure 5N The first drag component 532) receives the first position value and the gesture recognizer from the second subcomponent (eg, Figure 5N The zoom value is determined based on the second position value received by the second drag component 534 of the first drag component. In some embodiments, the first subcomponent gesture recognizer is the root node of the first pan gesture recognizer, and the second subcomponent gesture recognizer is the root node of the second pan gesture recognizer.

[0275] Determining the state of a parent component gesture recognizer based on information about the input determined by a child component gesture recognizer (e.g., the values ​​of parameters describing the input) enables more complex input processing without increasing the amount of processing required at the parent component gesture recognizer. Dividing a larger set of processing requirements into a smaller subset of processing requirements simplifies event processing and reduces the computational load on the device when processing input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0276] In some embodiments, the plurality of component gesture recognizers (724) in the gesture recognizers are selected from a plurality of available component gesture recognizers, the plurality of available component gesture recognizers comprising: a combiner that combines information from two or more component gesture recognizers and defines requirements for recognizing a gesture based on relationships between the component gesture recognizers combined using the combiner; and an input parameter recognizer that determines values ​​of parameters describing an input (e.g., location, timestamp, intensity) and / or defines threshold (e.g., lower and / or upper) requirements for parameters describing an input (e.g., threshold distance traveled by the input, threshold duration of the input, threshold contact intensity of the input). Defining and monitoring relationships between component gesture recognizers and input parameter recognizers using a combiner to monitor information about the input (e.g., values ​​of parameters describing the input) and define threshold requirements for the input enables more complex input processing while dividing a larger set of processing requirements into smaller subsets of processing requirements that can be processed more quickly than the larger set. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping users achieve intended results and by reducing user errors when operating / interacting with the device).

[0277] In some embodiments, a gesture recognizer is associated with an application (e.g., Figure 1B The gesture recognizers are associated with an application 136-1 (726), and the plurality of component gesture recognizers in the gesture recognizers are application-independent modules (e.g., an application-independent library of component gesture recognizers provided by an operating system or an application programming interface (API) on the device, such as Figure 1B An instance (e.g., an application-specific instance) of a corresponding component gesture recognizer from a plurality of available component gesture recognizers is provided by an application-independent software module 195). A framework of available component gesture recognizers is provided so that instances of available component gesture recognizers can be used to form gesture recognizers, which reduces the burden on the device to support and execute different gesture recognition processes across different applications and improves the consistency of gesture recognition functionality. This reduces the computational load imposed on the device, which in turn improves device responsiveness and reduces latency, thereby enhancing the operability of the device and making the user-device interface more effective (e.g., by helping the user achieve the desired results and by reducing user errors when operating / interacting with the device).

[0278] In some embodiments, the gesture recognizer includes (728) a third component gesture recognizer (e.g., a combiner) that defines a third requirement for recognizing a gesture based on a relationship between the first component gesture recognizer and the second component gesture recognizer (e.g., an exclusive combiner, a simultaneous combiner, a sequential combiner, a parallel combiner, or a gated combiner). In some embodiments, monitoring the input with the gesture recognizer includes monitoring a state of the first component gesture recognizer and a state of the second component gesture recognizer with the third component gesture recognizer to determine whether the first component gesture recognizer (e.g., the state of the first component gesture recognizer) and the second component gesture recognizer (e.g., the state of the second component gesture recognizer) satisfy the third requirement. In some embodiments, the first gesture recognition criteria includes the third requirement based on a predefined relationship between the first component gesture recognizer and the second component gesture recognizer. Reference is made to FIG. 5A to FIG. 5Q and FIG. 6A to FIG. 6C Exemplary combiners and their requirements / uses are described. Using a third component gesture recognizer to define and monitor a relationship between a first component gesture recognizer and a second component gesture recognizer increases the complexity of gestures that can potentially be recognized without increasing the computational load on the first component gesture recognizer and the second component gesture recognizer, wherein each of the first component gesture recognizer and the second component gesture recognizer defines and monitors a respective subset of event processing requirements for recognizing a gesture. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0279] In some embodiments, the third component gesture recognizer is (730) an exclusive combiner, and the third requirement specifies that the first component gesture recognizer must fail in order for the second component gesture recognizer to succeed. In some embodiments, when the first component gesture recognizer is not in a failed state or a canceled state, the state of the third component gesture recognizer is based on the state of the first component gesture recognizer (e.g., without regard to the state of the second component gesture recognizer). In some embodiments, when the first component gesture recognizer is in a failed state or a canceled state, the state of the third component gesture recognizer is based on the state of the second component gesture recognizer. For example, when the first component gesture recognizer is in a state other than a failed state or a canceled state, the state of the third component gesture recognizer is based on the state of the first component gesture recognizer, and the third component gesture recognizer also includes an indication that the state of the third component gesture recognizer corresponds to the first component gesture recognizer. In another example, when the first component gesture recognizer is in a failed state or a canceled state, the state of the third component gesture recognizer is based on the state of the second component gesture recognizer, and the third component gesture recognizer also includes an indication that the state of the third component gesture recognizer corresponds to the second component gesture recognizer. Reference is also made herein to FIG. 6A to FIG. 6C Exemplary exclusive combiners are described. Providing an exclusive combiner that requires its first (sub)component gesture recognizer to fail in order for its second (sub)component gesture recognizer to succeed allows different priorities to be assigned to different gesture recognizers or component gesture recognizers without increasing the computational load of the gesture recognizers or component gesture recognizers to which the priorities are assigned. This increases the diversity of gestures that can be recognized while reducing the additional computational load typically imposed by monolithic gesture recognizers for increasingly diverse and complex gestures, which in turn increases device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0280] In some embodiments, the third component gesture recognizer is (732) a simultaneous combiner, and the third requirement requires success of at least one of the first component gesture recognizer and the second component gesture recognizer at corresponding points in time in order for the third component gesture recognizer to succeed. In some embodiments, the simultaneous combiner allows both the first component gesture recognizer and the second component gesture recognizer to succeed simultaneously. For example, a simultaneous gesture recognizer that combines a tap gesture recognizer and a double-click gesture recognizer allows the tap gesture recognizer to succeed when a tap input (e.g., an input having a duration less than a predefined threshold duration and an amount of movement less than a predefined threshold amount of movement) is recognized, and also allows the double-click gesture recognizer to succeed when a second tap input that completes a double-click gesture is recognized. Providing a simultaneous combiner that requires at least one (sub) gesture recognizer or component gesture recognizer to succeed but allows multiple (sub) gesture recognizers or component gesture recognizers to succeed improves the device's ability to recognize and process multiple gestures simultaneously, and thus provides additional control options to the user without requiring additional inputs or cluttering the user interface with additional displayed controls. This reduces the additional computational load typically imposed by using a monolithic gesture recognizer to process multiple gestures simultaneously, which improves device responsiveness and reduces latency, and also enhances device operability and makes the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0281] In some embodiments, the third component gesture recognizer is a gated combiner, and the third requirement requires success of the first component gesture and success of the second component gesture recognizer at corresponding points in time in order for the third component gesture recognizer to succeed. In other words, the gated combiner requires that both the first component gesture recognizer and the second component gesture recognizer succeed at least at one point in time (e.g., one component gesture recognizer can succeed before the other, but there must be at least one overlapping moment when both succeed) (e.g., the gated combiner operates as a logical AND gate). Reference is also made herein to FIG. 5A to FIG. 5F An exemplary gate combiner is described.

[0282] In some embodiments, a corresponding component gesture recognizer is considered successful when its corresponding requirements for recognizing a gesture are met. FIG. 5A to FIG. 5FAs discussed, the tap gesture recognizer requires that: (a) the distance traveled by the input (e.g., the amount of movement of the input on the touch-sensitive surface) satisfies a predefined threshold distance (e.g., is within the predefined threshold distance), and (b) the duration of the input (e.g., the amount of time the input is detected anywhere on the touch-sensitive surface) satisfies (e.g., is less than) a predefined threshold duration. In the tap gesture recognizer example, a gating combiner (e.g., gating combiner 512) is used to require that both requirements (a) and (b) be met in order to recognize a tap gesture. In other words, if the second component gesture recognizer (e.g., distance component recognizer 506) fails, then the gating combiner (e.g., gating combiner 51) fails, and if the second component gesture recognizer has not yet failed, then the gating combiner passes as its output the state and any value output by the first component gesture recognizer (e.g., gating combiner 510).

[0283] In some embodiments, the third component gesture recognizer is a parallel combiner that requires the success of the first component gesture recognizer and the success of the second component gesture at corresponding points in time. In one example, a parallel combiner is used in a pinch gesture recognizer to define a relationship between the first component gesture recognizer as a first drag component gesture recognizer and the second component gesture recognizer as a second drag component gesture recognizer, each of which monitors movement of a corresponding input. In the pinch gesture recognizer example, the parallel combiner requires that the success of the first pan gesture recognizer (e.g., based on the detected first touch input) occur simultaneously with the success of the second pan gesture recognizer (e.g., based on the second touch input detected simultaneously with the first touch input). References herein Figure 5M to Figure 5Q An exemplary parallel combiner is described.

[0284] In some embodiments, the third component gesture recognizer is (734) a sequential combiner, and in order for the third component gesture recognizer to succeed, the third requirement requires the success of the first component gesture recognizer, and requires the success of the second component gesture recognizer after the first component gesture recognizer succeeds. In other words, the sequential combiner requires its child component gesture recognizers to succeed one after the other in order for the sequential component gesture recognizer to succeed. For example, a drag gesture recognizer for recognizing a long press then move input that requires a sequential combiner having a long press gesture recognizer as its first child component gesture recognizer and a pan gesture recognizer as its second child component gesture recognizer can be assembled using a sequential combiner that has a long press gesture recognizer as its first child component gesture recognizer and a pan gesture recognizer as its second child component gesture recognizer. In some embodiments, the success of the sequential combiner requires that the later component gesture recognizer succeed after the earlier component gesture recognizer has succeeded and ended. In some embodiments, the success of the sequential combiner still occurs if the later component gesture recognizer succeeds after the earlier component gesture recognizer succeeds but before the earlier component gesture recognizer has ended. In some embodiments, the third requirement further requires that the second component gesture recognizer succeed within a predefined amount of time (e.g., a timeout period) from the end of the first component gesture recognizer.

[0285] In some embodiments, a repeating component gesture recognizer is similar to a sequential component gesture recognizer in requiring its child gesture recognizers to succeed sequentially, except that a repeating component gesture recognizer has a single child-component gesture recognizer that is reused or re-instantiated. In some embodiments, the child-component gesture recognizers of a repeating combiner are created sequentially (e.g., because the two child-component gesture recognizers are the same type of component gesture recognizers (e.g., different instances of the same component gesture recognizer)). In some embodiments, a first child-component gesture recognizer of a repeating combiner is created and associated with the repeating combiner and used to process input, and after the first child-component gesture recognizer succeeds and ends, the first child-component gesture recognizer is destroyed or removed from the gesture recognizer, and a second child-component gesture recognizer (e.g., a second instance of the same type of component gesture recognizer as the first child-component gesture recognizer) is created and associated with the repeating combiner and subsequently used to process input). Figure 5G to Figure 5L Repeater 522 in double-click gesture recognizer 520 of FIG. 5 depicts an exemplary repeating component gesture recognizer.

[0286] Providing sequential or repeated combiners that require the sequential success of multiple (sub) gesture recognizers or component gesture recognizers allows composite gestures to be formed by reusing such gesture recognizers or component gesture recognizers (or using instances thereof). This increases the diversity of gestures that can be recognized while reducing the additional computational load typically imposed by monolithic gesture recognizers for increasingly diverse and complex gestures. This in turn increases device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0287] In some embodiments, the first requirement of the first component gesture recognizer requires (736) that the value of a parameter (e.g., any of a position, a distance moved, a timestamp, a duration, an intensity) describing the input satisfies a specified threshold (e.g., the first requirement requires that the parameter value does not exceed a maximum threshold, and / or the first requirement requires that the parameter value satisfies a minimum threshold, as described herein with reference to FIG. 5A to FIG. 5F In some embodiments, the gesture recognizer includes a fourth component gesture recognizer that determines a value of a parameter describing the input. In some embodiments, monitoring the input using the gesture recognizer includes monitoring the input using the fourth component gesture recognizer to determine a value of a parameter associated with the fourth component gesture recognizer, and the method includes transmitting the parameter value determined using the fourth component gesture recognizer to the first component gesture recognizer to determine whether the first requirement is met. Using another component gesture recognizer that extracts specific information about the input and transmits the extracted information to (e.g., based on the parameter value) to determine whether a corresponding requirement for recognizing a gesture is met makes the information needed to process the input available without increasing the computational load on the component gesture recognizer responsible for determining whether its corresponding requirement is met. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0288] In some embodiments, the gesture recognizer is (738) a tap gesture recognizer (e.g., Figure 5A The tap gesture recognizer 500 is referred to herein as FIG. 5A to FIG. 5F). In some embodiments, the first component gesture recognizer is a distance component gesture recognizer that stores an initial position of the input corresponding to a point in time when the input is first detected, and receives a current position of the input at the corresponding point in time. In some embodiments, the distance component gesture recognizer receives the current position of the input from a component gesture recognizer that recognizes information about the input (e.g., an input event) that includes position information. In some embodiments, the distance component gesture recognizer extracts the current position of the input from information about the input (e.g., an input event) transmitted to the distance component gesture recognizer. In some embodiments, the first requirement defined by the first component gesture recognizer requires that a distance between the initial position of the input and the current position of the input be within a threshold distance. In some embodiments, the first component gesture recognizer determines whether the input satisfies the first requirement.

[0289] In some embodiments, the second component gesture recognizer is a (maximum) duration component gesture recognizer that stores an initial timestamp corresponding to a point in time when the input was first detected and receives a current timestamp corresponding to a corresponding (e.g., current) point in time. In some embodiments, the duration component gesture recognizer receives the current timestamp of the input from a component gesture recognizer that recognizes information about the input (e.g., input event) that includes timestamp information. In some embodiments, the duration component gesture recognizer extracts the current timestamp of the input from the information about the input (e.g., input event) transmitted to the duration component gesture recognizer. In some embodiments, the second requirement defined by the second component gesture recognizer requires that an amount of time between the initial timestamp and the current timestamp be within a threshold (e.g., predefined threshold) amount of time. In some embodiments, the second component gesture recognizer determines whether the input satisfies the second requirement.

[0290] In some embodiments, the tap gesture recognizer further includes a position component gesture recognizer (e.g., a spatial event listener) that determines that the input has a current position at a corresponding point in time. In some embodiments, the tap gesture recognizer further includes a first gating combiner that requires the success of the second component gesture recognizer and the success of the position component gesture recognizer in order to succeed. In some embodiments, the tap gesture recognizer further includes a second gating combiner that requires the success of the first gating combiner and the success of the first component gesture recognizer in order to succeed. In some embodiments, the tap gesture recognizer further includes a suspend component gesture recognizer that prevents the tap gesture recognizer from succeeding when any other gesture recognizer or component gesture recognizer with a higher priority than the tap gesture recognizer is actively processing the input (and has not yet failed).

[0291] In some embodiments, the gesture recognizer is a double-tap gesture recognizer. In some embodiments, the double-tap gesture recognizer includes a repeating component gesture recognizer that includes a tap gesture recognizer (e.g., all component gesture recognizers of a tap gesture recognizer, as described herein). In some embodiments, the repeating component gesture recognizer transmits information to and receives processing results from a root node component gesture recognizer of the tap gesture recognizer. In some embodiments, the repeating component gesture recognizer requires repeated successes of the included gesture recognizer or component gesture recognizers. In some embodiments, the repeating component gesture recognizer defines the number of times (e.g., two, three, four, etc.) that the included gesture recognizer or component gesture recognizer must succeed in order for the repeating component gesture recognizer to succeed. In some embodiments, the repeating component gesture recognizer defines timing requirements, such as a maximum allowed delay between successes of the included gesture recognizers. In one example, a repeating component gesture recognizer in a double-tap gesture recognizer requires that the included tap gesture recognizer succeed twice; in such an example, the repeating component gesture recognizer may optionally require that the tap gesture recognizer succeed a second time within a predefined time since the tap gesture recognizer first succeeded.

[0292] In some embodiments, the gesture recognizer is a pan gesture recognizer, where the pan gesture recognizer includes a pan event listener (e.g., a component gesture recognizer that recognizes information about the input that indicates that the input has moved (e.g., does not require that the information about the input indicate a current location of the input)).

[0293] In some embodiments, the gesture recognizer is a long press gesture recognizer. In some embodiments, the long press gesture recognizer is similar to the tap gesture recognizer, except that the long press gesture recognizer includes a shortest duration component gesture recognizer instead of a longest duration component gesture recognizer. Specifically, in some embodiments, the long press gesture recognizer includes a position component gesture recognizer (e.g., a spatial event listener) as described herein. In some embodiments, the long press gesture recognizer also includes a distance component gesture recognizer that stores an initial position of the input corresponding to a point in time when the input was first detected, and receives a current position of the input at the corresponding point in time. The distance component gesture recognizer defines a requirement for recognizing a long press gesture that the distance between the initial position of the input and the current position of the input is within a threshold distance.

[0294] In some embodiments, the long press gesture recognizer also includes a shortest duration component gesture recognizer that stores an initial timestamp corresponding to a point in time when the input was first detected and receives a current timestamp corresponding to the corresponding point in time. In some embodiments, the shortest duration component gesture recognizer receives the current timestamp of the input from a component gesture recognizer that identifies information about the input that includes the timestamp information. In some embodiments, the shortest duration component gesture recognizer extracts the current timestamp of the input from the information about the input transmitted to the shortest duration component gesture recognizer. The shortest duration component gesture recognizer defines a requirement for recognizing a long press gesture that the amount of time between the initial timestamp and the current timestamp is at least a threshold (e.g., a predefined threshold) amount of time.

[0295] In some embodiments, the long press gesture recognizer includes a first gating combiner that requires success of the position gesture recognizer and success of the shortest duration component gesture recognizer in order to succeed. In some embodiments, the long press gesture recognizer includes a second gating combiner that requires success of the first gating combiner and success of the distance component gesture recognizer in order to succeed.

[0296] In some embodiments, the gesture recognizer is a pinch gesture recognizer, wherein the pinch gesture recognizer includes: a first drag component gesture recognizer that listens for events with position information and determines movement information based on the position information; and a second drag component gesture recognizer that listens for events with position information and determines movement information based on the position information. In some embodiments, the pinch gesture recognizer includes a parallel combiner that associates the first drag component gesture recognizer with a first input (e.g., a first contact on a touch-sensitive surface) and associates the second drag component gesture recognizer with a second input (e.g., a second contact on the touch-sensitive surface). In some embodiments, the parallel combiner transmits information about the first input to the first drag component gesture recognizer instead of the second drag component gesture recognizer, and receives state information and parameter values ​​describing the first input from the first drag component gesture recognizer. In some embodiments, the parallel combiner transmits information about the second input to the second drag component gesture recognizer instead of the first drag component gesture recognizer, and receives state information and parameter values ​​describing the second input from the second drag component gesture recognizer. In some embodiments, the pinch gesture recognizer includes a pinch component gesture recognizer that receives state information and parameter values ​​for both the first input and the second input and determines a zoom value (e.g., for zooming a displayed user interface element) based on the state information and the parameter values. Figure 5M to Figure 5Q An exemplary pinch gesture recognizer is shown and described.

[0297] In some embodiments, the gesture recognizer is a rotation gesture recognizer. In some embodiments, the rotation gesture recognizer is similar to the pinch gesture recognizer, except that the rotation gesture recognizer includes a rotation component gesture recognizer instead of a pinch component gesture recognizer. In some embodiments, the rotation component gesture recognizer receives state information and parameter values ​​(e.g., position and movement information) for both the first input and the second input, and determines an angle (e.g., for rotating a displayed user interface element) instead of a scale based on the state information and the parameter values. Similar to the pinch gesture recognizer, the rotation gesture recognizer also includes: a first drag component gesture recognizer that listens for events with position information and determines movement information based on the position information; a second drag component gesture recognizer that listens for events with position information and determines movement information based on the position information; and a parallel combiner that associates the first drag component gesture recognizer with the first input (e.g., a first contact on a touch-sensitive surface) and associates the second drag component gesture recognizer with the second input (e.g., a second contact on the touch-sensitive surface).

[0298] With respect to any gesture recognizer, including any of the aforementioned exemplary gesture recognizers, a user (e.g., an application developer) may add modules to or remove modules from a gesture recognizer, or develop a modified version of any corresponding component gesture recognizer in a gesture recognizer and replace the corresponding component gesture recognizer with the modified version, so as to customize the gesture recognizer and thereby customize the requirements for the gesture recognizer to recognize a gesture.

[0299] It should be understood that 7A to 7E The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., method 800 and method 900) are also applicable in a similar manner to the details of other processes described above with respect to 7A to 7E Method 700 described herein. For example, the inputs, states, values, user interfaces, component gesture recognizers, gesture recognizers, views, and view hierarchies described above with reference to method 700 optionally have one or more characteristics of the inputs, states, values, user interfaces, component gesture recognizers, gesture recognizers, views, and view hierarchies described herein with reference to other methods described herein (e.g., methods 800 and 900). For the sake of brevity, these details are not repeated here.

[0300] FIG. 8A to FIG. 8C8 is a flowchart illustrating a method 800 for generating a gesture recognition hierarchy from a view hierarchy of views in a user interface according to some embodiments. The method 800 is performed in an electronic device (e.g., Figure 3 Device 300 or Figure 1A The portable multifunction device 100 is executed at the electronic device having a display (e.g., Figure 1A Touch screen 112 or Figure 3 340), input devices (e.g., Figure 1A Touch screen 112, Figure 3 355 or keyboard / mouse 350), and optionally one or more sensors for detecting intensity of contact with an input device including a touch-sensitive surface (e.g., Figure 1A In some embodiments, the display is a touch screen display having a touch-sensitive surface on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 800 are optionally combined and / or the order of some operations is optionally changed.

[0301] As described below, method 800 provides an electronic device with an intuitive way to prioritize gesture recognizers for use in processing input directed to a user interface of an application based on a view hierarchy that defines relationships between views in the user interface to which the gesture recognizers are associated. Forming a gesture recognition hierarchy between gesture recognizers based on relationships between their associated views as defined in the view hierarchy facilitates processing input in a manner consistent with the displayed user interface and improves the responsiveness of the device, which in turn reduces latency. Reducing latency in responding to input reduces user errors in operating / interacting with the device, such as duplicate or irrelevant inputs made by a user while waiting for the device to respond to an earlier input by the user. Reducing the number and / or extent of inputs from a user enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0302] The device displays (802) the user interface of the application on the display (e.g., Fig. 6A The user interface includes (804) a view hierarchy (e.g., Fig. 6A A plurality of views (eg, Fig. 6A In some embodiments, the view hierarchy defines one or more relationships, such as parent-child relationships, between the multiple views in the view hierarchy. The view hierarchy defines a first view (e.g., Fig. 6A612) and a second view (eg, Fig. 6A The first view includes a first gesture recognizer (e.g., a first gesture recognizer that is a subview of the second view, the first view is a parent view of the second view, the first view and the second view are sibling views (two subviews of the same parent view), etc.). Fig. 6A ), and the second view includes a second gesture recognizer (e.g., Fig. 6A Scroll gesture recognizer 608).

[0303] The device detects ( 806 ), via an input device, an input (eg, a touch input on a touch-sensitive surface) at a first location corresponding to the displayed user interface.

[0304] The device processes (808) the input using a gesture recognition hierarchy (e.g., gesture recognition hierarchy 640) that includes the first gesture recognizer and the second gesture recognizer. A second relationship (e.g., a gesture dependency relationship) between the first gesture recognizer and the second gesture recognizer (in the gesture recognition hierarchy) is determined based on the first relationship between the first view and the second view in the view hierarchy. In some embodiments, in response to detecting the input (e.g., touch input 620), and based on a determination as to which view in the view hierarchy is a hit view (e.g., based on a determination that view 612 is the hit view), the gesture recognition hierarchy is formed. In some embodiments, relationships between gesture recognizers in the gesture recognition hierarchy are defined based on relationships between views in the view hierarchy (corresponding to each gesture recognizer) relative to the hit view, as described herein with reference to FIG. Fig. 6A described.

[0305] In some embodiments, one or more views other than the first view and the second view in the view hierarchy include (810) corresponding gesture recognizers, and the gesture recognition hierarchy includes corresponding gesture recognizers for the one or more other views. Specifically, in some embodiments, the view hierarchy includes a third view (e.g., view 602 with long press gesture recognizer 604, Fig. 6A ), and the gesture recognition hierarchy includes a third gesture recognizer (e.g., gesture recognition hierarchy 640 includes long press gesture recognizer 604, Fig. 6A In some embodiments, the gesture recognition hierarchy includes a respective gesture recognizer for a respective view in the view hierarchy, regardless of whether the respective view corresponds to the first position (e.g., as described herein with reference to Fig. 6AIn some embodiments, the view hierarchy includes a hit view determined based on the first position and a set of views that are parent views of the hit view, and the gesture recognition hierarchy includes (in some embodiments only) corresponding gesture recognizers in the hit view and the parent views of the hit view. In some embodiments, the gesture recognition hierarchy includes gesture recognizers for all views in the view hierarchy. Including gesture recognizers for (all) other views in the view hierarchy in the gesture recognition hierarchy facilitates comprehensive gesture processing that considers all possible gestures associated with displayed user interfaces of applications on the device when processing input to the device. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping users achieve intended results and by reducing user errors when operating / interacting with the device).

[0306] In some embodiments, the device identifies (812) a subset of views in the view hierarchy corresponding to the first position, wherein the identified subset includes the first view and the second view, and wherein the gesture recognition hierarchy includes corresponding gesture recognizers for the identified subset of views (e.g., as described herein with reference to Fig. 6AIn some embodiments, the view hierarchy includes a third view having a third gesture recognizer, wherein the third view does not correspond to the first position, and the gesture recognition hierarchy does not include the third gesture recognizer of the third view. In some embodiments, only gesture recognizers of the view corresponding to the first position in the view hierarchy are candidates for recognizing the input detected at the first position. In some embodiments, the gesture recognition hierarchy includes gesture recognizers of the view corresponding to the first position, and does not include gesture recognizers of views that do not correspond to the first position. In some embodiments, the view corresponding to the first position includes a hit view, which is the lowest level view in the view hierarchy corresponding to the first position, and optionally satisfies a predefined criterion (e.g., includes at least one gesture recognizer, or includes at least one gesture recognizer that is capable of recognizing the input). In some embodiments, the view corresponding to the first position also includes one or more parent views of the hit view, and in some embodiments, includes all parent views of the hit view. Including in the gesture recognition hierarchy (only) a subset of views in the view hierarchy associated with the location at which the input was detected facilitates processing of gestures that are considered more likely to be associated with the detected input without imposing the additional burden of considering gestures that are not associated with the location and are therefore less likely to be associated with the detected input. This reduces the computational load on the device when processing input, which improves device responsiveness and reduces latency, thereby enhancing device operability, and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0307] In some embodiments, the view hierarchy includes (814) one or more views that do not correspond to the first position, wherein the one or more views include one or more gesture recognizers, and the gesture recognition hierarchy does not include the one or more gesture recognizers for the one or more views that do not correspond to the first position (e.g., as described herein with reference to Fig. 6A Including in the gesture recognition hierarchy (only) a subset of views in the view hierarchy that are associated with the location where the input was detected, gesture recognizers facilitate processing of gestures that are more likely to be associated with the detected input without imposing the additional burden of considering gestures that are not associated with the location and are therefore less likely to be associated with the detected input. This reduces the computational load on the device when processing input, which improves device responsiveness and reduces latency, thereby enhancing device operability, and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0308] In some embodiments, the gesture recognition hierarchy (sometimes referred to as a gesture recognition tree) includes (816) component gesture recognizers (e.g., exclusive combiners such as Fig. 6A The component gesture recognizer defines a second relationship between the first gesture recognizer and the second gesture recognizer by requiring the first gesture recognizer to fail in order for the second gesture recognizer to succeed. Defining and monitoring the relationship between the first gesture recognizer and the second gesture recognizer in the gesture recognition hierarchy using independent component gesture recognizers (e.g., combiners) improves the ability of the electronic device to distinguish different gestures without increasing the computational load on the first gesture recognizer and the second gesture recognizer, which improves device responsiveness and reduces latency. In addition, using independent component gesture recognizers reduces the need for individual modifications to the first gesture recognizer and the second gesture recognizer, which provides consistency in gesture recognition. These benefits enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0309] In some embodiments, the gesture recognition hierarchy includes (818) a component gesture recognizer that defines a second relationship between a first gesture recognizer and a second gesture recognizer in the gesture recognition hierarchy. In some embodiments, processing the input using the gesture recognition hierarchy includes: receiving information describing the input at the gesture recognition hierarchy; communicating the information describing the input to the component gesture recognizer; and communicating the information describing the input from the component gesture recognizer to at least one of the first gesture recognizer and the second gesture recognizer based on the second relationship between the first gesture recognizer and the second gesture recognizer. In some embodiments, processing the input using the gesture recognition hierarchy further includes: processing the information describing the input using the first gesture recognizer based on determining that the information describing the input is communicated to the first gesture recognizer. In some embodiments, processing the input using the gesture recognition hierarchy further includes: processing the information describing the input using the second gesture recognizer based on determining that the information describing the input is communicated to the second gesture recognizer.

[0310] Alternatively, in some embodiments, the device transmits information describing the input from the component gesture recognizer to the first gesture recognizer but not to the second gesture recognizer at a first time, and at a second time after the first time, based on determining that the first gesture recognizer failed to recognize the input, the device transmits information describing the input from the component gesture recognizer to the second gesture recognizer. For example, as explained above with respect to operation 718 of method 700, some combiners transmit information describing the input (e.g., one or more input events) to all of their child component gesture recognizers based on information transmission rules for the combiner or combiner type. Other combiners transmit information to only some of their child component gesture recognizers based on information transmission rules for the combiner or combiner type by transmitting information about a first input of a plurality of inputs only to a first child component gesture recognizer and transmitting information about a second input of a plurality of inputs only to a second child component gesture recognizer, or by transmitting information about one or more inputs to a second child component gesture recognizer only after the first child component gesture recognizer successfully or otherwise completes processing the information.

[0311] Transmitting information from a combiner in the gesture recognition hierarchy to at least one of the first gesture recognizer and the second gesture recognizer based on a relationship between the first gesture recognizer and the second gesture recognizer defined by the combiner reduces unnecessary processing of input by gesture recognizers that are not currently associated with the input. This reduces computational load on the device, which improves device responsiveness and reduces latency, thereby enhancing device operability, and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors in operating / interacting with the device).

[0312] In some embodiments, the information describing the input is processed using the first gesture recognizer to determine (820) a current state of the first gesture recognizer, and the information describing the input is processed using the second gesture recognizer to determine a current state of the second gesture recognizer. In some embodiments, the device communicates the current state of the first gesture recognizer from the first gesture recognizer to the component gesture recognizer; communicates the current state of the second gesture recognizer from the second gesture recognizer to the component gesture recognizer; and processes the information describing the input, the current state of the first gesture recognizer, and the current state of the second gesture recognizer using the component gesture recognizer to determine the current state of the component gesture recognizer. In some embodiments, different component gesture recognizers have different information processing rules, and thus how the state is processed to determine the current state is determined by the processing rules for the component gesture recognizer type for the component gesture recognizer. FIG. 5A to FIG. 5QAn example of determining a state of a gesture recognizer based on the states of a component gesture recognizer is described. Determining the states of a component gesture recognizer that defines a relationship between two gesture recognizers based on the states of the gesture recognizers enables more complex input processing without increasing the amount of processing required at the gesture recognizers, which simplifies event processing and reduces the computational load on the device when processing input. This improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the device).

[0313] In some embodiments, the first view includes (822) a third gesture recognizer, and the gesture recognition hierarchy includes the third gesture recognizer and a component gesture recognizer that defines a third relationship between the first gesture recognizer and the third gesture recognizer in the gesture recognition hierarchy. In some embodiments, the device determines a gesture recognition state for the first view by performing a plurality of operations. In some embodiments, the operations include transmitting information describing the input to the component gesture recognizers, the first gesture recognizer, and the third gesture recognizer. In some embodiments, the operations include processing the information describing the input with the first gesture recognizer to determine a current state of the first gesture recognizer, and transmitting the current state of the first gesture recognizer from the first gesture recognizer to the component gesture recognizer. In some embodiments, the operations also include processing the information describing the input with the third gesture recognizer to determine a current state of the third gesture recognizer, and transmitting the current state of the third gesture recognizer from the third gesture recognizer to the component gesture recognizer. In some embodiments, the operations also include processing the information describing the input, the current state of the first gesture recognizer, and the current state of the third gesture recognizer with the component gesture recognizers to determine the gesture recognition state for the first view. For example, determining based on the (overall) processing results of the gesture recognition hierarchy 640 Fig. 6A The gesture recognition status of the hit view 612 in .

[0314] In some embodiments, if a view has only one gesture recognizer, the (current) gesture recognition state of the view is the (current) gesture recognition state of the one gesture recognizer. In some embodiments, if a view has two or more gesture recognizers, the two or more gesture recognizers are arranged in a gesture recognition hierarchy (e.g., a branch of a larger gesture recognition hierarchy that also includes gesture recognizers of other views) using one or more combiners that define priorities or simultaneity between the two or more gesture recognizers. In some embodiments, the gesture recognition state of the view is determined by the one or more combiners (e.g., by a combiner at the highest (root) node of the gesture recognition hierarchy of the view) based on the (current) states of the two or more gesture recognizers (e.g., the corresponding node determines its current state based on the corresponding states of its child nodes in the gesture recognition hierarchy). For example, Fig. 6A The gesture recognition state of middle view 606 is determined based on the processing results of simultaneous combiner 646, which defines the relationship between two gesture recognizers of view 616, namely scroll gesture recognizer 608 and pinch gesture recognizer 610.

[0315] Determining the gesture recognition state of a view based on the state of a component gesture recognizer that defines a relationship between two gesture recognizers in the view, where the state of the component gesture recognizer is in turn based on the state of the gesture recognizer, enables more sophisticated input processing and improves the device's ability to distinguish between different gestures without increasing the amount of processing required at the gesture recognizers. This simplifies event handling and reduces the computational load on the device when processing input, which in turn improves device responsiveness and reduces latency, thereby enhancing device operability and making the user-device interface more effective (e.g., by helping users achieve intended results and by reducing user errors when operating / interacting with the device).

[0316] It should be understood that FIG. 8A to FIG. 8C The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein in conjunction with other methods described herein (e.g., method 700 and method 900) are also applicable in a similar manner to the above with respect to FIG. 8A to FIG. 8CMethod 800 is described. For example, the inputs, states, values, user interfaces, component gesture recognizers, gesture recognizers, views, and view hierarchies described above with reference to method 800 optionally have one or more characteristics of the inputs, states, values, user interfaces, component gesture recognizers, gesture recognizers, views, and view hierarchies described herein with reference to other methods described herein (e.g., methods 700 and 900). For the sake of brevity, these details are not repeated here.

[0317] FIG. 9A to FIG. 9B is a flow chart illustrating a method 900 of processing an input using a gesture recognizer to which a value of a parameter describing the input has been associated according to some embodiments. Figure 3 Device 300 or Figure 1A The portable multifunction device 100 is executed at the electronic device having an input device (e.g., Figure 1A Touch screen 112, Figure 3 touchpad 355 or keyboard / mouse 350), an optional display (e.g., Figure 1A Touch screen 112 or Figure 3 340), and optionally one or more sensors for detecting intensity of contact with an input device including a touch-sensitive surface (e.g., Figure 1A In some embodiments, the display is a touch screen display having a touch-sensitive surface on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 900 are optionally combined and / or the order of some operations is optionally changed.

[0318] As described below, method 900 provides an electronic device with an intuitive way to monitor input using a gesture recognition tree (sometimes referred to as a gesture recognition hierarchy) in which a plurality of gesture recognizers are arranged. Transmitting parameter values ​​describing an input between gesture recognizers in the tree provides the gesture recognizers with information about the input needed to process the input without requiring each gesture recognizer to determine the information individually, which reduces the computational load on the device when processing the input, thereby improving device responsiveness and reducing latency. Reducing latency in responding to input reduces user errors in operating / interacting with the device, such as duplicate or irrelevant inputs made by a user while waiting for the device to respond to an earlier input by the user. Reducing the number and / or extent of inputs from a user enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0319] The device detects ( 902 ) input via an input device (eg, touch input on a touch-sensitive surface).

[0320] In response to detecting the input, the device uses a gesture recognition tree having a plurality of nodes (e.g., a gesture recognizer assembled from a plurality of component gesture recognizers) (e.g., as described herein with reference to FIG. 5A to FIG. 5Q and FIG. 6A to FIG. 6C The gesture recognition tree (e.g., any gesture recognizer or gesture recognition hierarchy described herein) monitors (904) the input. Each respective node of the gesture recognition tree corresponds (906) to a respective gesture recognizer (e.g., a gesture recognizer that includes multiple component gesture recognizers) or a respective component gesture recognizer (e.g., a combiner). In addition, one or more nodes include (e.g., store) one or more parameters describing the input (e.g., location of initial detection of the input, current location, distance traveled (determined based on the current location and the initial location), speed, contact strength, initial timestamp (e.g., when the input was first detected), current timestamp).

[0321] Monitoring input using the gesture recognition tree includes (908): processing the input using a first node of the plurality of nodes, including determining a value of a first parameter of the one or more parameters; transmitting the first parameter from the first node to a second node of the plurality of nodes; and processing the input using the second node, including determining whether the input satisfies a gesture recognition requirement defined by the second node based on the first parameter. In some embodiments, the gesture recognition requirement defined by the second node is one of a set of gesture recognition criteria defined by the gesture recognition tree (e.g., a gesture recognizer). References herein Figure 5M to Figure 5Q Examples are described of processing input to determine values ​​of parameters describing the input and transmitting those values ​​to other nodes for further processing of the input.

[0322] In some embodiments, the one or more parameters describing the input include (910) one or more of: a location where the input was initially detected, a current location of the input, a distance traveled by the input (e.g., a contact on a touch-sensitive surface), a velocity of the input (e.g., an instantaneous velocity, whether directly detected or derived from position and time information), intensity of the contact (e.g., of a touch input), an initial timestamp corresponding to when the input was first detected, and a current timestamp. For example, Figure 5M to Figure 5Q An example of transmitted parameters including current timestamp, current speed, initial position and current position is shown.

[0323] In some embodiments, the first parameter is (912) the location of the input, and the gesture recognition requirements defined by the second node require that the distance between the initial location of the input and the current location of the input satisfy a threshold distance. In some embodiments, the threshold distance is satisfied when the distance between the initial location and the current location exceeds the threshold distance (e.g., the input must move at least a minimum distance in order to satisfy the gesture recognition requirements defined by the second node, and in order to recognize a gesture associated with the second node). In some embodiments, the threshold distance is satisfied when the distance between the initial location and the current location is within the threshold distance (e.g., the input must not move more than a maximum distance in order to satisfy the gesture recognition requirements defined by the second node, and in order to recognize a gesture associated with the second node). In some embodiments, processing the input using the first node includes determining the current location of the input using the first node. In some embodiments, processing the input using the second node includes determining whether the distance between the initial location of the input (e.g., stored by the second node) and the current location of the input transmitted from the first node satisfies the threshold distance.

[0324] In some embodiments, the first parameter is (914) a timestamp corresponding to a corresponding point in time at which the input was detected, and the gesture recognition requirements defined by the second node require that an amount of time between an initial timestamp corresponding to a first point in time at which the input was first detected and a current timestamp corresponding to a second point in time at which the input was detected satisfy a threshold amount of time. In some embodiments, the threshold amount of time is satisfied when the amount of time between the initial timestamp and the current timestamp exceeds the threshold amount of time (e.g., the input must be continuously detected (e.g., for touch input, held on the touch-sensitive surface) for at least a minimum amount of time in order to satisfy the gesture recognition requirements defined by the second node, and in order to recognize a gesture associated with the second node). In some embodiments, the threshold amount of time is satisfied when the amount of time between the initial timestamp and the current timestamp is within the threshold amount of time (e.g., the input must not be continuously detected (e.g., held on the touch-sensitive surface) for more than a maximum amount of time in order to satisfy the gesture recognition requirements defined by the second node, and in order to recognize a gesture associated with the second node). In some embodiments, processing the input using the first node includes determining, using the first node, a current timestamp corresponding to the second point in time at which the input was detected. In some embodiments, processing the input using the second node includes determining, using the second node, whether an amount of time between an initial timestamp of the input (eg, stored by the second node) and a current timestamp of the input transmitted from the first node satisfies a threshold amount of time.

[0325] In some embodiments, the device processes (916) the input using one or more of the plurality of nodes in the gesture recognition tree to determine values ​​for the one or more parameters, regardless of whether the gesture has been recognized by the gesture recognition tree (e.g., as described herein with reference to Figure 5N, which shows the state and values ​​of pinch gesture recognizer 530 before a pinch gesture has been recognized. In some embodiments, the values ​​of the one or more parameters are determined before a gesture is recognized by the gesture recognition tree. For example, these values ​​are determined even when the state of the gesture recognizer tree (e.g., an overall state determined based on the corresponding states of the nodes in the gesture recognition tree, as described above with reference to method 700) is a state other than a gesture recognition state (e.g., a predefined "possible" state indicating that a particular gesture to be recognized by the gesture recognizer is possible). Determining the values ​​of the parameters describing the input when processing the input, regardless of whether the gesture has been recognized, reduces latency in processing the input and helps the device recognize gestures more quickly. Improving the responsiveness of the device enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user achieve desired results more quickly and by reducing user errors when operating / interacting with the device).

[0326] In some embodiments, in response to detecting the input, the device receives (918) first information describing the input at a gesture recognition tree, wherein a value of the first parameter is determined based on the first information, and wherein determining whether the input satisfies the gesture recognition requirements defined by the second node includes comparing the value of the first parameter to a previous value determined based on second information received at the gesture recognition tree prior to receiving the first information (e.g., as described herein with reference to Figure 5M to Figure 5Q As described above, the pinch gesture recognizer 530 compares the current position of the corresponding input with the initial position to determine whether the input satisfies the pinch gesture requirements). Determining whether the input satisfies a particular gesture recognition requirement by comparing the current value of a parameter describing the input with the previous (e.g., initial) value of the parameter provides an efficient way to monitor the input over time, which reduces the computational load on the device when processing the input. This improves device responsiveness and reduces latency, thereby enhancing the operability of the device and making the user-device interface more effective (e.g., by helping the user achieve the intended result and by reducing user errors when operating / interacting with the device).

[0327] It should be understood that FIG. 9A to FIG. 9B The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein in conjunction with other methods described herein (e.g., methods 700 and 800) are also applicable in a similar manner to the above with respect to FIG. 9A to FIG. 9BMethod 900 is described. For example, the inputs, states, values, user interfaces, component gesture recognizers, gesture recognizers, views, and view hierarchies described above with reference to method 900 optionally have one or more characteristics of the inputs, states, values, user interfaces, component gesture recognizers, gesture recognizers, views, and view hierarchies described herein with reference to other methods described herein (e.g., methods 700 and 800). For the sake of brevity, these details are not repeated here.

[0328] References 7A to 7E , FIG. 8A to FIG. 8C as well as FIG. 9A to FIG. 9B The described operations are optionally performed by Figure 1A to Figure 1B The components depicted are implemented. For example, detection operations 702, 806, 902, monitoring operations 704, 904, identification operations 708, and processing operations 806 are optionally implemented by event classifier 170, event identifier 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact on touch-sensitive display 112, and event distributor module 174 delivers event information to application 136-1. The corresponding event identifier 180 and one or more component event identifiers 181 of application 136-1 compare the event information with the corresponding event definition 184, and determine whether the first contact at the first position on the touch-sensitive surface (or whether the rotation of the device) corresponds to a predefined event or sub-event, such as selection of an object on a user interface, or rotation of the device from one orientation to another orientation. When a corresponding predefined event or sub-event is detected, event identifier 180 activates event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update application internal state 192. In some embodiments, event handler 190 accesses corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know that based on Figure 1A-1B How the components depicted in the figure can implement other processes.

[0329] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible based on the above teachings. The embodiments are selected and described in order to best illustrate the principles of the invention and its practical application, so as to enable others skilled in the art to best use the invention and the various described embodiments with various modifications suitable for the specific purposes contemplated.

Claims

1. A method comprising: At an electronic device having an input device: detecting input via the input device; In response to detecting the input, monitoring the input using a gesture recognizer, wherein the gesture recognizer includes a plurality of component gesture recognizers that define requirements for recognizing a gesture, wherein a first component gesture recognizer from the plurality of component gesture recognizers defines a first requirement for recognizing the gesture and has a current state from a plurality of available states for the first component gesture recognizer, and a second component gesture recognizer from the plurality of component gesture recognizers defines a second requirement for recognizing the gesture and has a current state from the plurality of available states for a second component gesture recognizer, wherein the current state of the second component gesture recognizer is a different state than the current state of the first component gesture recognizer; as well as recognizing a first gesture using the gesture recognizer based on determining that the input satisfies first gesture recognition criteria including the first requirement for the first component gesture recognizer and the second requirement for the second component gesture recognizer; wherein a first current state of the first component gesture recognizer is determined based on the input, a second current state of the second component gesture recognizer is determined based on the input, and a current state of the gesture recognizers is determined based on the first current state of the first component gesture recognizer and the second current state of the second component gesture recognizer.

2. The method of claim 1 , wherein monitoring the input using the gesture recognizer comprises: receiving, at the gesture recognizer, first information describing the input at a first point in time; processing the first information using the first component gesture recognizer to determine a current state of the first component gesture recognizer among the plurality of available states for the first point in time; processing the first information using the second component gesture recognizer to determine a current state of the second component gesture recognizer among the plurality of available states for the first point in time; as well as The current state of the gesture recognizer is determined based on the current state of the first component gesture recognizer and the current state of the second component gesture recognizer.

3. The method of claim 1 , wherein monitoring the input using the gesture recognizer comprises: receiving, at the gesture recognizer, first information describing the input at a first point in time; processing the first information using the first component gesture recognizer to determine a current state of the first component gesture recognizer among the plurality of available states for the first point in time; processing the first information using the second component gesture recognizer to determine a current state of the second component gesture recognizer among the plurality of available states for the first point in time; receiving, at the gesture recognizer, second information describing a change in the input at a second time point later than the first time point; processing the second information using the first component gesture recognizer to determine a current state of the first component gesture recognizer among the plurality of available states for the second point in time; as well as The second information is processed using the second component gesture recognizer to determine a current state of the second component gesture recognizer among the plurality of available states for the second point in time.

4. The method of any one of claims 1 to 3, wherein one or more component gesture recognizers of the gesture recognizers include one or more parameters describing the input; and monitoring the input using the gesture recognizer comprises determining values ​​of the one or more parameters using the one or more component gesture recognizers of the gesture recognizers.

5. The method of any one of claims 1 to 3, wherein the gesture recognizer is a gesture recognition tree having a plurality of nodes, wherein each node of the gesture recognition tree includes a corresponding component gesture recognizer, and wherein monitoring the input using the gesture recognizer comprises: transferring information describing the input at a corresponding point in time from a component gesture recognizer at a corresponding node having a child node in the gesture recognition tree to a component gesture recognizer at the child node in the gesture recognition tree; as well as Based on the information describing the input at the corresponding point in time, and in accordance with determining that the corresponding node has one or more child nodes, determining a state of a component gesture recognizer at a corresponding node in the gesture recognizer tree based on corresponding states of corresponding component gesture recognizers at the one or more child nodes, the corresponding states being determined based on the information describing the input at the corresponding point in time.

6. The method of claim 5, wherein monitoring the input using the gesture recognizer further comprises: transferring the determined state of the component gesture recognizer at a child node in the gesture recognition tree to a component gesture recognizer at a parent node of the child node in the gesture recognition tree; Wherein determining whether the input satisfies the first gesture recognition criteria for recognizing the first gesture comprises determining a state of the gesture recognition tree based on the determined states of the component gesture recognizers in the gesture recognition tree.

7. The method of claim 5, wherein monitoring the input using the gesture recognizer comprises: determining values ​​of one or more component gesture recognizers at corresponding nodes in the gesture recognition tree based on information describing the input at the corresponding point in time; Wherein, based on determining that the component gesture recognizer at a child node of the corresponding node has a determined value, determining the state of the corresponding component gesture recognizer at the corresponding node is further based on the determined value of the component gesture recognizer at the child node.

8. The method according to any one of claims 1 to 3, wherein: The gesture recognizer includes a gesture recognition tree having a plurality of nodes, wherein each node of the gesture recognition tree includes a corresponding component gesture recognizer from the plurality of component gesture recognizers; and the first component gesture recognizer and the second component gesture recognizer are located at child nodes of the gesture recognition tree relative to a parent node that includes a third component gesture recognizer from the plurality of component gesture recognizers.

9. The method according to claim 8, wherein: Monitoring the input using the gesture recognizers includes communicating states of the first component gesture recognizer and the second component gesture recognizer to the third component gesture recognizer, and determining a state of the third component gesture recognizer based on the states of the first component gesture recognizer and the second component gesture recognizer communicated to the third component gesture recognizer.

10. The method of any one of claims 1 to 3, wherein the plurality of component gesture recognizers in the gesture recognizers are selected from a plurality of available component gesture recognizers, the plurality of available component gesture recognizers comprising: a combiner that combines information from two or more component gesture recognizers and defines requirements for recognizing a gesture based on a relationship between the component gesture recognizers combined using the combiner; as well as An input parameter identifier determines values ​​of parameters describing the input and / or defines threshold requirements for parameters describing the input.

11. The method of any one of claims 1 to 3, wherein the gesture recognizer is associated with an application, and the plurality of component gesture recognizers in the gesture recognizer are instances of respective component gesture recognizers in a plurality of available component gesture recognizers provided by a module independent of the application.

12. The method according to any one of claims 1 to 3, wherein: the gesture recognizers comprising a third component gesture recognizer that defines a third requirement for recognizing a gesture based on a relationship between the first component gesture recognizer and the second component gesture recognizer; Monitoring the input using the gesture recognizer comprises monitoring a state of the first component gesture recognizer and a state of the second component gesture recognizer using the third component gesture recognizer to determine whether the first component gesture recognizer and the second component gesture recognizer satisfy the third requirement; and The first gesture recognition criteria includes the third requirement based on a relationship between the first component gesture recognizer and the second component gesture recognizer.

13. The method of claim 12, wherein the third component gesture recognizer is an exclusive combiner and the third requirement specifies that the first component gesture recognizer must fail in order for the second component gesture recognizer to succeed.

14. The method of claim 12, wherein the third component gesture recognizer is a simultaneous combiner, and the third requirement requires success of at least one of the first component gesture recognizer and the second component gesture recognizer at a corresponding point in time in order for the third component gesture recognizer to succeed.

15. The method of claim 12, wherein the third component gesture recognizer is a sequential combiner, and the third requirement requires success of the first component gesture recognizer in order for the third component gesture recognizer to succeed, and requires success of the second component gesture recognizer after success of the first component gesture recognizer.

16. The method of any one of claims 1 to 3, wherein the first requirement of the first component gesture recognizer requires that a value of a parameter describing the input satisfy a specified threshold.

17. The method of any one of claims 1 to 3, wherein the gesture recognizer is a tap gesture recognizer, wherein: the first component gesture recognizer being a distance component gesture recognizer that stores an initial position of the input corresponding to a point in time when the input is first detected and receives a current position of the input at the corresponding point in time, wherein the first requirement defined by the first component gesture recognizer requires that a distance between the initial position of the input and the current position of the input be within a threshold distance; as well as the second component gesture recognizer being a duration component gesture recognizer that stores an initial timestamp corresponding to the point in time when the input was first detected and receives a current timestamp corresponding to the corresponding point in time, wherein the second requirement defined by the second component gesture recognizer requires that an amount of time between the initial timestamp and the current timestamp be within a threshold amount of time; The tap gesture recognizer further comprises: a position component gesture recognizer that determines that the input has a current position at the corresponding point in time; a first gating combiner that requires success of the second component gesture recognizer and success of the position component gesture recognizer in order to succeed; and A second gating combiner that requires success of the first gating combiner and success of the first component gesture recognizer in order to succeed.

18. An information processing device, the information processing device being used in an electronic device having an input device, comprising: means for detecting input via said input device; means enabled in response to detecting the input for monitoring the input using a gesture recognizer, wherein the gesture recognizer comprises a plurality of component gesture recognizers that define requirements for recognizing a gesture, wherein a first component gesture recognizer from the plurality of component gesture recognizers defines a first requirement for recognizing the gesture and has a current state in a plurality of available states for the first component gesture recognizer, and a second component gesture recognizer from the plurality of component gesture recognizers defines a second requirement for recognizing the gesture and has a current state in the plurality of available states for a second component gesture recognizer, wherein the current state of the second component gesture recognizer is a different state than the current state of the first component gesture recognizer; as well as means for recognizing a first gesture using the gesture recognizer enabled based on a determination that the input satisfies first gesture recognition criteria including the first requirement for the first component gesture recognizer and the second requirement for the second component gesture recognizer; wherein a first current state of the first component gesture recognizer is determined based on the input, a second current state of the second component gesture recognizer is determined based on the input, and a current state of the gesture recognizers is determined based on the first current state of the first component gesture recognizer and the second current state of the second component gesture recognizer.

19. An information processing device according to claim 18, comprising means for performing a method according to any one of claims 2 to 17.

20. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device having an input device, cause the electronic device to: detecting input via the input device; In response to detecting the input, monitoring the input using a gesture recognizer, wherein the gesture recognizer includes a plurality of component gesture recognizers that define requirements for recognizing a gesture, wherein a first component gesture recognizer from the plurality of component gesture recognizers defines a first requirement for recognizing the gesture and has a current state from a plurality of available states for the first component gesture recognizer, and a second component gesture recognizer from the plurality of component gesture recognizers defines a second requirement for recognizing the gesture and has a current state from the plurality of available states for a second component gesture recognizer, wherein the current state of the second component gesture recognizer is a different state than the current state of the first component gesture recognizer; and recognizing a first gesture using the gesture recognizer based on determining that the input satisfies first gesture recognition criteria including the first requirement for the first component gesture recognizer and the second requirement for the second component gesture recognizer; in, A first current state of the first component gesture recognizer is determined based on the input, a second current state of the second component gesture recognizer is determined based on the input, and a current state of the gesture recognizers is determined based on the first current state of the first component gesture recognizer and the second current state of the second component gesture recognizer.

21. The computer-readable storage medium of claim 20, wherein the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method of any one of claims 2 to 17.

Images