Semantic Framework for Variable Tactile Output

Abstract

A haptic output system, method, and apparatus are provided that organizes multiple haptic output changes into a cohesive semantic framework that uses various information about alert conditions and triggers, application context, and other conditions, and shares characteristics between related events. [Solution] The event class or application class is the basis for the corresponding haptic output, and whether the alert salience setting is on is the basis for adding an increased salience haptic output to the standard haptic output for alerts, providing consistent haptics to categorize related application classes, applications and / or contexts.

Description

Technical Field

[0001] The present disclosure generally relates to methods and apparatuses for providing haptic feedback, and more specifically, to methods and apparatuses for a semantic framework of haptic feedback corresponding to conditions associated with a device or application.

Background Art

[0002] With a haptic feedback system, a user can interact with a subsystem by touch or contact. A haptic system facilitates these tactile interactions by using an actuator, a sensor, or both.

Summary of the Invention

[0003] The embodiments described herein organize changes in multiple haptic outputs into a cohesive semantic framework that uses various information regarding alert conditions and triggers, application context, and other conditions, to provide a haptic feedback method and apparatus that provides a system of haptic outputs that share features between related events. The present disclosure relates to systems and methods for providing haptic responses under various conditions. The disclosed approach to haptic response can be implemented using any suitable software, hardware, or both.

[0004] In some embodiments, the state associated with the application at the time associated with the detected alert condition associated with the application serves as a basis for providing a corresponding haptic output. For example, in response to the detection of an alert condition, the state associated with the application at the time associated with the alert condition is determined. If the application is in an active state at the time associated with the alert condition, a first haptic output representing the occurrence of the alert condition is provided. If the application is in an inactive state at the time associated with the alert condition, a second haptic output that represents the occurrence of the alert condition and is different from the first haptic output is provided.

[0005] In some embodiments, whether the detected alert condition is triggered manually or automatically serves as a basis for providing a corresponding haptic output. For example, in response to the detection of an alert condition, it is determined whether the alert condition is triggered by an event started manually. If the alert condition is triggered by an event started manually, a first haptic output corresponding to the notification of the event started manually is provided. If the alert condition is triggered by an event started automatically, a second haptic output corresponding to the notification of the event started automatically is provided, and the second haptic output is different from the first haptic output.

[0006] In some embodiments, whether the detected alert condition is associated with a user input or a predetermined system event serves as a basis for providing a corresponding haptic output. For example, in response to the detection of a first alert condition associated with the reception of a user input for an application, a first haptic output is provided corresponding to the user input. Following the first haptic output, in response to the detection of a second alert condition associated with the reception of a predetermined system event in the application, a second haptic output with an intensity greater than that of the first haptic output is provided corresponding to the predetermined system event.

[0007] In some embodiments, whether the detected alert condition is part of a multi-part operation serves as a basis for providing a corresponding haptic output. For example, in response to receiving an input corresponding to a first part of a multi-part operation, a continuous haptic output sequence is initiated. After initiation and in response to receiving an input corresponding to a second part of the multi-part operation, the continuous haptic output sequence is stopped.

[0008] In some embodiments, whether the detected request to perform an operation is a subset of another operation serves as a basis for providing a corresponding haptic output. For example, in response to detecting a first input corresponding to a request to perform a first operation, a first output including a haptic component is provided and the first operation is performed. After performing the first operation and in response to detecting a second input corresponding to a request to perform a second operation including the first operation and an additional operation, a second output including the first output with an additional output corresponding to the additional operation and including a haptic component is provided and the second operation is performed.

[0009] In some embodiments, whether the two detected alert conditions are of the same class of alert condition or application class, or different classes of alert condition or application class, serves as a basis for providing a corresponding haptic output. For example, in response to detecting the occurrence of a first alert condition, a first output including a first haptic component and a first non-haptic component is provided. After the first output and in response to detecting the occurrence of a second alert condition, a second output including a second haptic component and a second non-haptic component is provided. If the first alert condition and the second alert condition are different alert conditions of the same class of alert condition, the first output and the second output share one or more same components and have one or more different components, but if the first alert condition and the second alert condition are different alert conditions of different classes of alert condition, the first haptic component is different from the second haptic component and the first non-contact component is different from the second non-contact component. In another example,

[0010] In some embodiments,

[0011] Note that various embodiments of the methods described herein can be combined with any other embodiments described herein. The features and advantages described herein are not necessarily inclusive. Specifically, many additional features and advantages will be apparent to those skilled in the art in light of the drawings, the specification, and the claims. Further, note that the language used herein has been selected solely for readability and for the purpose of explanation and not to define or limit the subject matter of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012]

Figure 1A

[0013]

Figure 1B

[0014]

Figure 2

[0015]

Figure 3

[0016]

Figure 4A

[0017]

Figure 4B

[0018]

Figure 5

[0019]

Figure 6

[0020]

Figure 7

[0021]

Figure 8

[0022]

Figure 9

[0023]

Figure 10

[0024]

Figure 11

[0025]

Figure 12

[0026]

Figure 13

[0027]

Figure 14

[0028]

Figure 15

[0029]

Figure 16

[0030]

Figure 17

[0031]

Figure 18

[0032]

Figure 19

[0033]

Figure 20

[0034]

Figure 21A

[0035]

Figure 21B

[0036]

Figure 22A

[0037]

Figure 22B

[0038]

Figure 23A

[0039]

Figure 23B

[0040]

Figure 24

[0041]

Figure 25

[0042]

Figure 26

[0043]

Figure 27

[0044]

Figure 28

[0045]

Figure 29

[0046]

Figure 30

[0047]

Figure 31

[0048]

Figure 32

[0049]

Figure 33a

[0050]

Figure 33b

[0051]

Figure 33c

[0052]

Figure 34

[0053]

Figure 35

DETAILED DESCRIPTION OF THE INVENTION

[0054] Embodiments herein provide a haptic feedback method and apparatus for systematizing multiple haptic output changes in a binding semantic framework that provides a haptic output system that shares features between related events using various information regarding alert conditions and triggers, application context, and other conditions corresponding to haptic output.

[0055] In some embodiments, the state associated with the application at the time associated with the detected alert condition associated with the application serves as a basis for providing a corresponding haptic output. In some embodiments, whether the detected alert condition was manually triggered or automatically triggered serves as a basis for providing a corresponding haptic output. In some embodiments, whether the detected alert condition is associated with user input or a predetermined system event serves as a basis for providing a corresponding haptic output.

[0056] In some embodiments, whether the detected alert condition is part of a multi-part operation serves as a basis for providing a corresponding haptic output. In some embodiments, whether the detected requirement for performing an operation is a subset of another operation serves as a basis for providing a corresponding haptic output. In some embodiments, whether the two detected alert conditions are of the same class of alert conditions or different classes of alert conditions serves as a basis for providing a corresponding haptic output. Exemplary Devices

[0057] Here, reference is made in detail to the embodiments shown in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0058] In this specification, terms such as first, second, etc. are used in some cases to describe various elements, but it will also be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first contact could be referred to as the second contact, and similarly, the second contact could be referred to as the first contact. The first contact and the second contact are both contacts, but they are not the same contact.

[0059] The terms used in the description of the various embodiments described herein are for the purpose of describing particular embodiments only and are not intended to be limiting. When used in the description of the various embodiments described and in the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" refers to any and all possible combinations of one or more of the associated listed items and is to be construed as including the same. The terms "includes", "including", "comprises" and / or "comprising", as used herein, specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0060] As used herein, the term "if" is optionally construed to mean "when", "upon", "in response to determining", or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (a stated condition or event) is detected" are optionally construed to mean "upon determining", "in response to determining", "upon detecting (the stated condition or event)", or "in response to detecting (the stated condition or event)", depending on the context.

[0061] Embodiments of an electronic device, a user interface for such a device, and related processes for using such a device are described. In some embodiments, the device is a portable communication device such as a mobile phone that includes other functions such as PDA functionality and / or music playback functionality. Exemplary embodiments of portable multifunctional devices include, without limitation, the iPhone (registered trademark), iPod Touch (registered trademark), and iPad (registered trademark) devices manufactured by Apple Inc. (Cupertino, California). Other portable electronic devices such as laptop or tablet computers having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad) can also optionally be used. Also, in some embodiments, it should be understood that the device is a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad) rather than a portable communication device.

[0062] In the following description, an electronic device having a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, a mouse, and / or a joystick.

[0063] The device generally supports various applications such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a game application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music playback application, and / or a digital video playback application.

[0064] Various applications running on this device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or changed from one application to the next and / or within the corresponding applications. Thus, the common (such as touch-sensitive surface) physical architecture of the device optionally supports various applications with an intuitive and transparent user interface for the user.

[0065] Attention is now directed to an embodiment of a portable device equipped with a touch-sensing display. FIG. 1A is a block diagram showing a portable multifunctional device 100 having a touch-sensing display 112 according to some embodiments. The touch-sensing display 112 may be referred to as a "touch screen" for convenience, and is known as or may be referred to as a touch-sensing display system. The device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensing surface such as the touch-sensing display system 112 of the device 100). The device 100 optionally includes one or more haptic output generators 167 for generating haptic output on the device 100 (e.g., generating haptic output on a touch-sensing surface such as the touch-sensing display system 112 of the device 100 or the touch pad 355 of the device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0066] As used in this specification and the claims, the term "intensity" of a contact on a touch-sensing surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on the touch-sensing surface, or an alternative (substitute) for the force or pressure of a contact on the touch-sensing surface. The intensity of a contact has a range of numerical values that includes at least four different numerical values, and more typically, hundreds of different numerical values (e.g., at least 256). The intensity of a contact is optionally determined (or measured) using a variety of techniques and a variety of sensors, or combinations of sensors. For example, one or more sensors disposed under or adjacent to the touch-sensing surface are optionally used to measure the force at various points on the touch-sensing surface. In some implementations, the force measurements of multiple force sensors are combined (e.g., weighted average) to determine an estimated value of the contact force. Similarly, a pressure-sensing chip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensing surface. As another method, the size and / or change thereof of the contact area detected on the touch-sensing surface, the capacitance and / or change thereof of the touch-sensing surface in proximity to the contact, and / or the resistance and / or change thereof of the touch-sensing surface in proximity to the contact are optionally used as substitutes for the force or pressure of a contact on the touch-sensing surface. In some implementations, the alternative measurements of the force or pressure of a contact are used directly to determine whether the intensity threshold is exceeded (e.g., the intensity threshold is described in units corresponding to the alternative measurements). In some implementations, the alternative measurements of the force or pressure of a contact are converted to an estimated force or pressure, and this estimated force or pressure is used to determine whether the intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of a contact as an attribute of a user input, a user can access additional device functions that would normally be inaccessible to the user on a device of reduced size with a limited area for displaying affordances (e.g., on a touch-sensing display) and / or receiving user input (e.g., via a touch-sensing display, a touch-sensing surface, or a physical / mechanical control such as a knob or button).

[0067] As used in this specification and the claims, the term "haptic output" refers to a physical displacement of a device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensing surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device that will be detected by the user's sense of touch. For example, in a situation where the device or a component of the device is in contact with a surface of the user that is sensitive to touch (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical movement is interpreted by the user as a haptic sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, the movement of a touch-sensing surface (e.g., a touch-sensing display or a trackpad) can be optionally interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, the user can feel haptic sensations such as "down click" or "up click" even when there is no movement of the physical actuator button associated with the touch-sensing surface that has been physically pressed (e.g., displaced) by the user's action. As another example, the movement of a touch-sensing surface can be optionally interpreted or felt by the user as "roughness" of the touch-sensing surface even when there is no change in the smoothness of the touch-sensing surface. Such an interpretation of the touch by the user is based on the user's individual sensory perception, but there are many sensory perceptions of touch that are common to most users. Thus, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "up click", "down click", "roughness"), unless otherwise specified, the generated haptic output corresponds to a physical displacement of the device or a component of the device that produces the described sensory perception of a typical (or average) user.

[0068] Device 100 is merely an example of a portable multifunctional device, and it should be understood that device 100 may optionally have more or fewer components than those shown in the figures, optionally combine two or more components, or optionally have different configurations or arrangements of components. The various components shown in FIG. 1A are implemented in a combination of hardware, software, or both hardware and software, including one or two or more signal processing circuits and / or application-specific integrated circuits.

[0069] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory such as one or two or more magnetic disk storage devices, flash memory devices, or other non-volatile semiconductor memory devices. Access to memory 102 by other components of device 100, such as CPU 120 and peripheral device interface 118, is optionally controlled by memory controller 122.

[0070] Peripheral device interface 118 can be used to connect the input and output peripheral devices of this device to CPU 120 and memory 102. One or two or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and to perform data processing.

[0071] In some embodiments, peripheral device interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0072] The RF (Radio Frequency) circuit 108 transmits and receives RF signals, also called electromagnetic signals. The RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with a communication network and other communication devices via the electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing the above functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, etc. The RF circuit 108 optionally communicates wirelessly with networks such as the Internet, also called the World Wide Web (WWW), intranets and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and other devices. The wireless communication optionally uses any of a plurality of communication standards, protocols and technologies including but not limited to Global System for Mobile Communications (GSM) (registered trademark), Enhanced Data GSM Environment (EDGE) (registered trademark), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA) (registered trademark), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth (registered trademark), Wireless Fidelity (Wi-Fi) (registered trademark) (e.g., IEEE802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE802.11n). The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111.Speaker 111 converts an electrical signal into a sound wave audible to a human ear. Audio circuit 110 also receives an electrical signal converted from a sound wave by microphone 113. Audio circuit 110 converts the electrical signal into audio data and sends the audio data to peripheral device interface 118 for processing. The audio data is optionally taken in from and / or sent to memory 102 and / or RF circuit 108 by peripheral device interface 118. In some embodiments, audio circuit 110 further includes a headset jack (e.g., 212 of FIG. 2). The headset jack provides an interface between audio circuit 110 and a removable audio input / output peripheral device such as an output-only headset or a headset having both an output (e.g., headphones for one or both ears) and an input (e.g., a microphone).

[0073] The I / O subsystem 106 couples the input / output peripheral devices of device 100, such as the touch screen 112 and other input control devices 116, to the peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. One or two or more input controllers 160 receive electrical signals from other input or control devices 116 and transmit electrical signals to other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the (plural) input controllers 160 are optionally coupled to (or not coupled to any of) pointer devices such as a keyboard, an infrared port, a USB port, and a mouse. One or two or more buttons (e.g., 208 in FIG. 2) optionally include up / down buttons for adjusting the volume of the speaker 111 and / or the microphone 113. One or two or more buttons optionally include push buttons (e.g., 206 in FIG. 2).

[0074] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and transmits electrical signals to the touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects.

[0075] The touch screen 112 has a touch sensing surface, sensor, or set of sensors that receives input from a user based on tactile and / or haptic contact. The touch screen 112 and the display controller 156 (along with any associated modules and / or instruction sets within the memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and translate the detected contact into an interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In one exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0076] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although in some embodiments other display technologies are used. The touch screen 112 and the display controller 156 use any of a plurality of currently known or later developed touch sensing technologies, including but not limited to capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, and other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112, to optionally detect contact and any movement or interruption thereof. In one exemplary embodiment, projected mutual capacitance sensing technology, such as that found in the iPhone®, iPod Touch®, and iPad® made by Apple Inc. (Cupertino, California), is used.

[0077] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally contacts the touch screen 112 using any suitable object or implement such as a stylus, finger, etc. In some embodiments, the user interface is designed to function primarily based on finger-based contact and gestures, and due to the larger contact area of the finger on the touch screen, it may be less accurate than stylus-based input. In some embodiments, the device converts finger-based rough input into an accurate pointer / cursor position or a command to perform the user-desired action.

[0078] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touch pad (not shown) for activating or deactivating certain functions. In some embodiments, the touch pad, unlike the touch screen, is a touch-sensitive area of the device that does not display visual output. The touch pad is optionally a separate touch-sensitive surface from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0079] The device 100 also includes a power system 162 for powering various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power outage detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diode (LED)), and any other components associated with power generation, management, and distribution in a portable device.

[0080] Device 100 also optionally includes one or more optical sensors 164. FIG. 1A shows an optical sensor coupled to an optical sensor controller 158 within the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In combination with the imaging module 143 (also referred to as a camera module), the optical sensor 164 optionally captures a still image or video. In some embodiments, the optical sensor is disposed on the back of the device 100, which is opposite the touch screen display 112 on the front of the device, so as to enable the touch screen display to be used as a viewfinder for still image and / or video acquisition. In some embodiments, another optical sensor is disposed on the front of the device so as to optionally obtain an image of the user while the user is viewing other video conference participants on the touch screen display for a video conference.

[0081] Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 within the I / O subsystem 106. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric force sensors, optical force sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is co-located with or in proximity to a touch sensing surface (e.g., the touch sensing display system 112). In some embodiments, at least one contact intensity sensor is disposed on the back of the device 100, which is opposite the touch screen display 112 disposed on the front of the device 100.

[0082] Device 100 also optionally includes one or more proximity sensors 166. FIG. 1A shows a proximity sensor 166 coupled to the peripheral device interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 within the I / O subsystem 106. In some embodiments, when a multifunctional device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and the touch screen 112 becomes inactive.

[0083] Device 100 also optionally includes one or more haptic output generators 167. FIG. 1A shows a haptic output generator coupled to the haptic feedback controller 161 within the I / O subsystem 106. The haptic output generator 167 optionally includes one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal on the device into a haptic output). The contact intensity sensor 165 receives haptic feedback generation instructions from the haptic feedback module 133 and generates a haptic output on device 100 that can be sensed by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed with or adjacent to a touch sensing surface (e.g., the touch sensing display system 112) and, optionally, generates a haptic output by moving the touch sensing surface in a vertical direction (e.g., in / out of the surface of device 100) or a lateral direction (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is disposed on the back of device 100, opposite the touch screen display 112 disposed on the front of device 100.

[0084] Device 100 also optionally includes one or more accelerometers 168. FIG. 1A shows an accelerometer 168 coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 within the I / O subsystem 106. In some embodiments, information is displayed on the touch screen display in a portrait or landscape orientation based on analysis of data received from one or more accelerometers. Device 100, as described in U.S. Patent Application No. 11 / 969,800, filed January 4, 2008, the entire disclosure of which is incorporated herein by reference, optionally includes, in addition to the (plural) accelerometers 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the position and orientation (e.g., portrait or landscape) of Device 100.

[0085] In some embodiments, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Further, in some embodiments, as shown in FIGS. 1A and 3, the memory 102 stores a device / global internal state 157. The device / global internal state 157 includes one or more of the following. An active application state indicating which application is active if there is an active application, a display state indicating which application, view, or other information occupies various regions of the touch screen display 112, a sensor state including information obtained from various sensors and input control devices 116 of the device and position information regarding the position and / or altitude of the device.

[0086] The operating system 126 (e.g., an embedded operating system such as Darwin (registered trademark), RTXC (registered trademark), LINUX (registered trademark), UNIX (registered trademark), OS X (registered trademark), WINDOWS (registered trademark), or VxWorks (registered trademark)) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware components and software components.

[0087] The communication module 128 also includes various software components for facilitating communication with other devices via one or more external ports 124 and processing data received by the RF circuit 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE (registered trademark), etc.) are adapted to connect directly or indirectly via a network (e.g., the Internet, a wireless local area network, etc.) to other devices. In some embodiments, the external port is the same as, similar to, and / or compatible with the 30-pin connector used on iPod (trademark of Apple Inc.) devices, a multi-pin (e.g., 30-pin) connector.

[0088] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and contact with other touch-sensing devices (e.g., a touch pad or a physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to the detection of contact, such as determining whether contact has occurred (e.g., detecting a finger-down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to the force or pressure of the contact), determining whether there is movement of the contact and tracking of the movement across the touch-sensing surface (e.g., detecting one or more finger-drag events), and determining whether the contact has stopped (e.g., detecting a finger-up event or an interruption of the contact). The contact / motion module 130 receives contact data from the touch-sensing surface. Determining the movement of the contact point represented by a series of contact data optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., a single contact of a finger) or multiple simultaneous contacts (e.g., "multi-touch" / multiple contacts of fingers). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0089] In some embodiments, the contact / motion module 130 uses one or more sets of intensity thresholds to determine whether an action has been performed by the user (e.g., to determine whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, the mouse "click" threshold of a trackpad or touch screen display can be set to any of a wide range of default thresholds without changing the hardware of the trackpad or touch screen display. Further, in some implementations, the user of the device is provided with software settings for adjusting one or more of the sets of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once by a system-level click of an "intensity" parameter).

[0090] As used in this specification and the claims, the term "characteristic strength" of a contact refers to the characteristics of that contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on a plurality of strength samples. The characteristic strength is optionally based on a set of strength samples collected over a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) related to a predetermined number of strength samples, i.e., related to a predetermined event (e.g., after detecting a contact, before detecting lift-off of the contact, before or after detecting the start of movement of the contact, before detecting the end of the contact, before or after detecting an increase in the strength of the contact, and / or before or after detecting a decrease in the strength of the contact). The characteristic strength of a contact is optionally based on one or more of the maximum value of the strength of the contact, the mean value of the strength of the contact, the average value of the strength of the contact, the top 10 percentile value of the strength of the contact, a value that is half of the maximum value of the strength of the contact, a value that is 90 percent of the maximum value of the strength of the contact, etc. In some embodiments, the duration of the contact is used in determining the characteristic strength (e.g., if the characteristic strength is the average of the strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an operation has been performed by a user. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a first operation is performed as a result of a contact having a characteristic strength that does not exceed the first threshold, a second operation is performed as a result of a contact having a characteristic strength that exceeds the first threshold and does not exceed the second threshold, and a third operation is performed as a result of a contact having a characteristic strength that exceeds the third threshold. In some embodiments, the comparison between the characteristic strength and one or more thresholds is not used to determine which of a first operation or a second operation to perform, but rather is used to determine whether to perform one or more operations (e.g., whether to perform each option or not perform each operation).

[0091] In some embodiments, for the purpose of determining the characteristic intensity, a portion of the gesture is identified. For example, the touch sensing surface can receive a continuous swipe contact that moves from a starting position to reach an ending position, at which point the intensity of the contact increases. In this example, the characteristic intensity of the contact at the ending position may be based on only a portion of the continuous swipe contact (e.g., only the portion of the swipe contact at the ending position), rather than the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, this smoothing algorithm may optionally include one or more of a non - weighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. Depending on the situation, these smoothing algorithms eliminate small sharp increases or decreases in the intensity of the swipe contact for the purpose of determining the characteristic intensity.

[0092] The contact / motion module 130 optionally detects gesture inputs by the user. Different gestures on the touch sensing surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contact). Thus, a gesture is optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event and subsequently detecting a finger up (lift - off) event at the same position (or substantially the same position) as that finger down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch sensing surface includes detecting a finger down event, subsequently detecting one or more finger drag events, and then detecting a finger up (lift - off) event.

[0093] The graphic module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term "graphic" includes any object that can be displayed to the user, and examples of such objects include text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc., but are not limited thereto.

[0094] In some embodiments, the graphic module 132 stores data representing the graphics used. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives one or more codes specifying the graphic to be displayed, along with coordinate data and other graphic characteristic data as needed, from an application or the like, and then generates screen image data for output to the display controller 156.

[0095] The tactile feedback module 133 includes various software components for generating instructions used by the tactile output generator 167 to generate tactile output at one or more locations on the device 100 in response to user interaction with the device 100.

[0096] The text input module 134 is optionally a component of the graphic module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0097] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map / navigation widgets).

[0098] The application 136 optionally includes the following modules (or sets of instructions), or subsets or supersets thereof. · Contact module 137 (which may also be referred to as an address book or contact list), · Phone module 138, · Video conferencing module 139, · Email client module 140, · Instant messaging (IM) module 141, · Training support module 142, · Camera module 143 for still and / or video images, · Image management module 144, · Browser module 147, · Calendar module 148, · Widget module 149 that optionally includes one or more of weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widget 149-6, · Widget creator module 150 for creating user-created widget 149-6, · Search module 151, · Video and music playback module 152 optionally composed of a video playback module and a music playback module, · Memo module 153, · Map module 154, and / or · Online video module 155.

[0099] Examples of other applications 136 that are optionally stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA (registered trademark)-compatible applications, encryption, digital rights management, voice recognition, and voice replication.

[0100] In conjunction with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in the memory 102 or the memory 370), which includes adding (multiple) names to the address book, deleting (multiple) names from the address book, associating (multiple) phone numbers, (multiple) email addresses, (multiple) actual addresses, or other information with a name, associating an image with a name, classifying and sorting names, and providing a phone number or email address to initiate and / or facilitate communication via phone 138, video conference 139, email 140, or IM 141, etc.

[0101] In conjunction with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the phone module 138 is optionally used to input a series of characters corresponding to a phone number, access one or more phone numbers in the address book 137, modify the entered phone number, dial the corresponding phone number, conduct a conversation, and disconnect or hang up the phone when the conversation is complete. As described above, wireless communication optionally uses any of a plurality of communication standards, protocols, and technologies.

[0102] In combination with RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphic module 132, text input module 134, contact list 137, and telephone module 138, the video conferencing module 139 includes executable instructions for starting, conducting, and ending a video conference between the user and one or more other participants according to the user's instructions.

[0103] In combination with RF circuit 108, touch screen 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to the user's instructions. In combination with the image management module 144, the email client module 140 makes it very easy to create and send emails with still or video images captured by the camera module 143.

[0104] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact module 130, graphic module 132, and text input module 134, instant messaging module 141 includes executable instructions for inputting a string corresponding to an instant message, modifying the input characters, and transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone - based instant messages, or using XMPP, SIMPLE, or IMPS for Internet - based instant messages), receiving instant messages, and displaying the received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments so as to be supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" means both phone - based messages (e.g., messages transmitted using SMS or MMS) and Internet - based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

[0105] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and music playback module 146, training support module 142 includes executable instructions for devising a training (e.g., having time, distance, and / or calorie consumption goals), communicating with a training sensor (sports device), receiving training sensor data, calibrating sensors used to monitor the training, selecting and playing music for the training, and displaying, storing, and transmitting training data.

[0106] Together with the touch screen 112, the display controller 156, the (plural) optical sensors 164, the optical sensor controller 158, the touch module 130, the graphic module 132, and the image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams), storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0107] Together with the touch screen 112, the display controller 156, the touch module 130, the graphic module 132, the text input module 134, and the camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise operating on, labeling, deleting, presenting (e.g., within a digital slide show or album), and storing still images and / or video images.

[0108] Together with the RF circuit 108, the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with a user's instructions, including searching, linking, receiving, and displaying across web pages or portions thereof, and attached and other files linked to the web pages.

[0109] Together with touch screen 112, display system controller 156, touch module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for enabling a user to access, browse, receive (e.g., by streaming and / or downloading), play (e.g., on the touch screen or on an external display connected via external port 124), send an email including a link to a particular online video, and manage in other ways one or more file format online videos such as H.264. In some embodiments, instead of email client module 140, instant messaging module 141 is used to send a link to a particular online video.

[0110] Each of the identified modules and applications above corresponds to a set of executable instructions for performing one or more of the functions and methods above.

[0111] Together with RF circuit 108, touch screen 112, display system controller 156, touch module 130, graphic module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar items, ToDo lists, etc.) according to user instructions.

[0112] In combination with the RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphic module 132, text input module 134, and browser module 147, the widget module 149 is a mini-application (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) that is optionally downloaded and used by the user, or a mini-application (e.g., user-created widget 149-6) created by the user. In some embodiments, the widget includes a Hypertext Markup Language (HTML) file, a Cascading Style Sheets (CSS) file, and a JavaScript® file. In some embodiments, the widget includes an Extensible Markup Language (XML) file and a JavaScript® file (e.g., Yahoo!(R) widget).

[0113] In combination with the RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphic module 132, text input module 134, and browser module 147, the widget creator module 150 is optionally used by the user to create a widget (e.g., change a user-specified location on a web page to a widget).

[0114] In combination with the touch screen 112, display system controller 156, contact module 130, graphic module 132, and text input module 134, the search module 151 includes executable instructions for searching for text, music, voice, images, video, and / or other files in the memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to the user's instructions.

[0115] In combination with touch screen 112, display system controller 156, contact module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, video and music playback module 152 includes executable instructions that enable a user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (registered trademark of Apple Inc.).

[0116] In combination with touch screen 112, display controller 156, contact module 130, graphic module 132, and text input module 134, memo module 153 includes executable instructions for creating and managing memos, ToDo lists, etc. according to a user's instructions.

[0117] In combination with RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data regarding specific locations or stores and other target locations in the vicinity of a specific location, and other location-related data) according to a user's instructions.

[0118] Together with touch screen 112, display system controller 156, touch module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play (e.g., on the touch screen or on an external display connected via external port 124), send an email including a link to a particular online video, and manage in other ways one or more online videos in one or more file formats such as H.264. In some embodiments, instead of email client module 140, instant messaging module 141 is used to send a link to a particular online video.

[0119] Each of the modules and applications identified above corresponds to a set of executable instructions for performing one or more of the above functions and to the methods described in this application (e.g., methods executed by a computer and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may optionally be combined or rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the identified modules and data structures described above. Further, memory 102 optionally stores additional modules and data structures not described above.

[0120] In some embodiments, device 100 is a device in which the operation of a default set of functions on the device is executed exclusively via a touch screen and / or a touch pad. By using the touch screen and / or the touch pad as the primary input control device for the operation of device 100, the number of physical input control devices (such as push buttons, dials, and the like) on device 100 can be optionally reduced.

[0121] The set of default functions that are executed exclusively via a touch screen and / or a touch pad optionally includes navigation between user interfaces. In some embodiments, when touched by a user, the touch pad navigates device 100 from any user interface displayed on device 100 to a main menu, a home menu, or a root menu. In such embodiments, the "menu button" is implemented using the touch pad. In some embodiments, the menu button is a physical push button or other physical input control device instead of the touch pad.

[0122] FIG. 1B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 102 (in FIG. 1A) or 370 (in FIG. 3) includes an event sorter 170 (e.g., within operating system 126) and a corresponding application 136-1 (e.g., any one of applications 137-13, 155, and 380-390 described above).

[0123] Event sorter 170 receives event information and determines application 136-1 and the application view 191 of application 136-1 that distributes the event information. Event sorter 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates the current (plural) application view displayed on touch-sensitive display 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event sorter 170 to determine which (plural) application is currently active, and application internal state 192 is used by event sorter 170 to determine the application view 191 to which the event information is to be distributed.

[0124] In some embodiments, application internal state 192 includes additional information such as resume information used when application 136-1 resumes execution, user interface state information indicating whether information is being displayed or is ready to be displayed by application 136-1, a state queue for allowing the user to return to a previous state or view of application 136-1, and one or more of a redo / undo queue of previous actions performed by the user.

[0125] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral device interface 118 transmits information received from sensors such as I / O subsystem 106 or proximity sensor 166, (plural) accelerometers 168, and / or microphone 113 (via audio circuitry 110). Information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0126] In some embodiments, the event monitor 171 sends requests to the peripheral device interface 118 at a predetermined interval. In response, the peripheral device interface 118 sends event information. In an embodiment, the peripheral device interface 118 sends event information only when there is an important event (e.g., receiving an input that exceeds a predetermined noise threshold and / or is longer than a predetermined time).

[0127] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognition unit determination module 173.

[0128] The hit view determination module 172 provides a software procedure for determining where in one or more views a sub-event has occurred when the touch-sensitive display 112 displays one or more views. A view is composed of a control unit and other elements that a user can view on the display.

[0129] Another aspect of the user interface associated with an application is, in this specification, a set of views, sometimes referred to as application views or user interface windows, in which information is displayed and touch-based gestures are performed. The application view (of each application) where a touch is detected optionally corresponds to a program level within the program hierarchy or view hierarchy of the application. For example, the lowest level view where a touch is detected is optionally referred to as a hit view, and the set of events recognized as appropriate inputs is optionally determined at least in part based on the hit view of the first touch that initiates a touch-based gesture.

[0130] The hit view determination module 172 receives information associated with sub-events of a touch-based gesture. When the application has a plurality of views organized as a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view within the hierarchy in which the sub-event should be processed. In most situations, the hit view is the lowest-level view at which the originating sub-event (i.e., the first sub-event in the sequence of sub-events that form an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0131] The active event recognition unit determination module 173 determines which view(s) within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition unit determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In some embodiments, the active event recognition unit determination module 173 determines that all views that include the physical location of the sub-event are views that are actively involved, and thus, all views that are actively involved should receive a particular sequence of sub-events. In some embodiments, even if a touch sub-event is completely confined to an area associated with a particular one view, higher-level views within the hierarchy are still maintained as views that are actively involved.

[0132] The event dispatcher module 174 immediately sends event information to the event recognition unit (for example, the event recognition unit 180). In an embodiment including the active event recognition unit determination module 173, the event dispatcher module 174 distributes event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information obtained by each event receiver module 182 in the event waiting queue.

[0133] In some embodiments, the operating system 126 includes an event sorter 170. Alternatively, the application 136-1 includes the event sorter 170. However, in some embodiments, the event sorter 170 is a stand-alone module or part of another module stored in the memory 102 such as the contact / motion module 130.

[0134] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each including instructions for processing touch events that occur within respective views of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In some embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, each event handler 190 includes one or more of data update data 176, object update data 177, GUI update data 178, and / or event data 179 received from event sorter 170. The event handler 190 optionally utilizes or calls data update data 176, object update data 177, or GUI update data 178 to update the application internal state 192. Alternatively, one or more of the application views 191 include one or more respective event handlers 190. Also, in some embodiments, one or more of data update data 176, object update data 177, and GUI update data 178 are included within respective application views 191.

[0135] Each event recognition unit 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparing unit 184. In some embodiments, the event recognition unit 180 also includes at least a subset of metadata 183 and event distribution instructions 188 (optionally including sub-event distribution instructions).

[0136] The event receiving unit 182 receives event information from the event sorter 170. This event information includes information about sub-events, for example, information about a touch or a movement of a touch. Depending on the sub-event, the event information may also include additional information such as the location of the sub-event. When the sub-event is related to the movement of a touch, the event information may also optionally include the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the posture of the device).

[0137] The event comparison unit 184 compares event information with the definition of a default event or sub - event, and based on this comparison, determines an event or sub - event, or determines or updates the state of an event or sub - event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes, for example, definitions of events such as event 1 (187 - 1), event 2 (187 - 2), etc. (e.g., a default sequence of sub - events). In some embodiments, the sub - events within event 187 include, for example, touch start, touch end, touch movement, touch cancellation, and multiple touches. In one example, the definition of event 1 (187 - 1) is a double - tap on a displayed object. The double - tap includes, for example, a first touch (touch start) at a predetermined stage on the displayed object, a first lift - off (touch end) at a predetermined stage, a second touch (touch start) at a predetermined stage on the displayed object, and a second lift - off (touch end) at a predetermined stage. In another example, the definition of event 2 (187 - 2) is a drag operation on a displayed object. The drag operation includes, for example, a touch (or contact) at a predetermined stage on the displayed object, movement of the touch across the touch - sensitive display 112, and lift - off of the touch (touch end). In some embodiments, an event also includes information about one or more associated event handlers 190.

[0138] In some embodiments, event definition 187 includes the definition of events for each user interface object. In some embodiments, event comparator 184 performs a hit test to determine the user interface object associated with the sub - event. For example, in an application view where three user interface objects are displayed on touch - sensitive display 112, when a touch is detected on touch - sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub - event). If each of the displayed objects is associated with a respective event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub - event and the object that triggered the hit test.

[0139] In some embodiments, the definition of each event 187 also includes a delaying action that delays the delivery of event information until it is determined whether the sequence of sub - events corresponds to or is equivalent to the event type of the event recognition unit.

[0140] If each event recognition unit 180 determines that a series of sub - events does not match any of the events in event definition 186, each event recognition unit 180 enters a state of event - impossible, event - failed, or event - ended, and then ignores subsequent sub - events of the touch - based gesture. In this situation, other event recognition units that remain active with respect to the hit view, if any, continue to track and process the sub - events of the ongoing touch - based gesture.

[0141] In some embodiments, each event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists indicating how the sub - event distribution is to be performed for the event recognition units in which the event distribution system is actively involved. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating how the event recognition units can interact with each other or become capable of interacting. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating whether the sub - events are distributed at various levels within the view hierarchy or at various levels within the program hierarchy.

[0142] In some embodiments, each event recognition unit 180 activates an event handler 190 associated with an event when one or more specific sub - events of the event are recognized. In some embodiments, each event recognition unit 180 distributes event information associated with that event to the event handler 190. Activating the event handler 190 is different from sending (and deferring sending) sub - events to each hit view. In some embodiments, the event recognition unit 180 throws a flag associated with the recognized event, and the event handler 190 associated with that flag catches the flag and executes a predefined process.

[0143] In some embodiments, the event distribution command 188 includes a sub - event distribution command that distributes event information about sub - events without activating an event handler. Instead, the sub - event distribution command distributes the event information to an event handler associated with a series of sub - events or to a view actively involved. The event handler associated with the series of sub - events or the view actively involved receives the event information and executes a predetermined process.

[0144] In some embodiments, data update data 176 creates and updates data used in application 136-1. For example, data update data 176 updates a phone number used in contact module 137 or stores a video file used in video playback module 145. In some embodiments, object update data 177 creates and updates objects used in application 136-1. For example, object update data 176 creates a new user interface object or updates the position of a user interface object. GUI update data 178 updates the GUI. For example, GUI update data 178 prepares display information and sends the display information to graphic module 132 for display on a touch-sensitive display.

[0145] In some embodiments, (a plurality of) event handlers 190 include or have access to data update data 176, object update data 177, and GUI update data 178. In some embodiments, data update data 176, object update data 177, and GUI update data 178 are included within a single module of their respective application 136-1 or application view 191. In some embodiments, they are included within two or more software modules.

[0146] The above description regarding event handling of a user's touch on a touch-sensing display also applies to other forms of user input for operating the multifunctional device 100 using an input device, but it should be understood that not all of them are initiated on the touch screen. For example, movement of a mouse and pressing of a mouse button, contact movement such as tapping, dragging, and scrolling on a touch pad, pen stylus input, movement of the device, verbal instructions, detected eye movement, biometric input, and / or any combination thereof, optionally in association with single or multiple presses or holds of a keyboard, are optionally utilized as input corresponding to sub-events that define events to be recognized.

[0147] FIG. 2 shows a portable multifunctional device 100 having a touch screen 112 according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described hereinafter, the user can select one or more of the graphics by performing gestures on the graphics using, for example, one or two or more fingers 202 (not drawn to an exact scale in the figure) or one or more styluses 203 (not drawn to an exact scale in the figure). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, upward, and / or downward), and / or rolling of a finger in contact with the device 100 (from right to left, from left to right, upward, and / or downward). In some implementations or situations, accidental contact with a graphic does not select the graphic. For example, a swipe gesture that swipes over an application icon does not optionally select the corresponding application when the gesture corresponding to selection is a tap.

[0148] Device 100 also optionally includes one or more physical buttons, such as a "home" or menu button 204. As described above, menu button 204 is optionally used to navigate to any application 136 within a set of applications that are optionally executed on device 100. In some embodiments, menu button 204 includes a fingerprint sensor that identifies a fingerprint on the menu button 204. The fingerprint sensor is optionally used to determine whether a finger on the menu button 204 has a fingerprint that matches the fingerprint used to unlock device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in the GUI displayed on touch screen 112.

[0149] In one embodiment, device 100 includes touch screen 1I2, menu button 204, a push button 206 for turning the power to the device on / off and locking the device, (one or more) volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. Push button 206 is optionally used to turn the power on / off on the device, lock the device, and / or unlock the device or initiate an unlock process by pressing the button and holding the button in a pressed state for a predetermined time interval, or by pressing the button and releasing the button before a predetermined time interval has elapsed. In an alternative embodiment, device 100 also accepts verbal input through microphone 113 to activate or deactivate some functions. Also, device 100 optionally includes one or more contact intensity sensors I65 for detecting the intensity of contact with touch screen 112 and / or one or more haptic output generators I67 for generating haptic output for the user of device 100.

[0150] FIG. 3 is a block diagram of an exemplary multifunctional device with a display and a touch sensing surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia playback device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home or business controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more networks or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 that includes a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, a touch pad 355, a haptic output generator 357 (e.g., similar to the (one or more) haptic output generators 167 described above with reference to FIG. 1A) for generating haptic output on device 300, and sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensing sensors, and / or contact intensity sensors similar to the (one or more) contact intensity sensors 165 described above with reference to FIG. 1A). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices, and also optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from the (one or more) CPUs 310.In some embodiments, the memory 370 stores programs, modules, and program-like programs, modules, and data structures, or subsets thereof, stored in the memory 102 of the portable multifunctional device 100 (FIG. 1A). Further, the memory 370 optionally stores additional programs, modules, and data structures that do not exist within the memory 102 of the portable multifunctional device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while the memory 102 of the portable multifunctional device 100 (FIG. 1A) does not optionally store these modules.

[0151] Each of the elements of FIG. 3 identified above is optionally stored in one or more than one of the memory devices described above. Each of the modules identified above corresponds to a set of instructions that perform the functions described above. The identified modules or programs (i.e., sets of instructions) described above need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, various subsets of these modules are optionally combined or otherwise reorganized. In some embodiments, the memory 370 optionally stores a subset of the identified modules and data structures described above. Further, the memory 370 optionally stores additional modules and data structures not described above.

[0152] Now, focus on embodiments of a user interface ("UI") that are optionally implemented on the portable multifunctional device 100.

[0153] Figure 4A shows an exemplary user interface related to the menu of an application on the portable multifunctional device 100, according to some embodiments. A similar user interface is optionally implemented on the device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof.

[0154] One or more signal strength indicators 402 for (multiple) wireless communications such as cellular signals and Wi-Fi signals, time 404, Bluetooth® indicator 405, battery status indicator 406, tray 408 with icons for frequently used applications, for example, indicator 414 of the number of missed calls or voicemail messages, icon 416 for phone module 138 labeled "Phone", optionally including indicator 410 of the number of unread emails, icon 418 for email client module 140 labeled "Mail", icon 420 for browser module 147 labeled "Browser", icon 422 for video and music playback module 152, also referred to as iPod (trademark of Apple Inc.) module 152 labeled "iPod", and as icons for other applications, for example, icon 424 for IM module 141 labeled "Text", icon 426 for calendar module 148 labeled "Calendar", icon 428 for image management module 144 labeled "Photos", icon 430 for camera module 143 labeled "Camera", icon 432 for online video module 155 labeled "Online Video", icon 434 for stock price widget 149-2 labeled "Stock Price", icon 436 for map module 154 labeled "Maps", icon 438 for weather widget 149-1 labeled "Weather", icon 440 for alarm clock widget 149-4 labeled "Clock", icon 442 for training support module 142 labeled "Training Support", icon 444 for memo module 153 labeled "Memo". An icon 446 for a settings application or module that provides access to settings related to device 100 and its various applications 136.

[0155] Note that the labels of the icons shown in FIG. 4A are merely exemplary. For example, the icon 422 for the video and music playback module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to each application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0156] FIG. 4B shows an exemplary user interface on a device (e.g., device 300 of FIG. 3) having a touch sensing surface 451 (e.g., tablet or touch pad 355 of FIG. 3) separate from the display 450 (e.g., touch screen display 112). The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 357) for detecting the intensity of contact on the touch sensing surface 451, and / or one or more haptic output generators 359 for generating haptic output to the user of the device 300.

[0157] Some of the following examples are described with reference to inputs on a touch screen display 112 (where a touch sensing surface and a display are combined), but in some embodiments, the device detects inputs on a touch sensing surface separate from the display, as shown in FIG. 4B. In some embodiments, the touch sensing surface (e.g., 451 in FIG. 4B) has a major axis (e.g., 452 in FIG. 4B) that corresponds to a major axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to these embodiments, the device detects contacts (e.g., 460 and 462 in FIG. 4B) with the touch sensing surface 451 at positions (e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470) that correspond to respective positions on the display. Thus, when the touch sensing surface is separate from the display, user inputs (e.g., contacts 460 and 462 and their movement) detected by the device on the touch sensing surface (e.g., 451 in FIG. 4B) are used by the device to operate a user interface on the display (e.g., 450 in FIG. 4B) of the multifunctional device. It should be understood that a similar method is optionally used for other user interfaces described herein.

[0158] Furthermore, although the following description mainly refers to finger inputs (e.g., finger contact, finger tap gesture, finger swipe gesture), in some embodiments, one or more of those finger inputs may be replaced by inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of contact), and then the cursor may be moved along the path of the swipe (e.g., instead of the movement of the contact). As another example, a tap gesture may optionally be replaced by a mouse click while the cursor is placed over the location of the tap gesture (e.g., instead of stopping the detection of contact following the detection of contact). Similarly, it will be appreciated that when multiple user inputs are detected simultaneously, multiple computer mice may be optionally used simultaneously, or a mouse and finger contact may be optionally used simultaneously.

[0159] As used herein, the term "focus selector" refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations, including a cursor or other location marker, the cursor functions as a "focus selector" when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., the touchpad 355 of FIG. 3, or the touch-sensitive surface 451 of FIG. 4B), such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, including a touch screen display (e.g., the touch-sensitive display system 112 of FIG. 1A, or the touch screen 112 of FIG. 4A) that enables direct interaction with user interface elements on the touch screen display, the contact detected on the touch screen serves as the "focus selector", such that when an input (e.g., a press input by contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus is moved from one region of the user interface to another region of the user interface without a corresponding movement of the cursor or movement of the contact on the touch screen (e.g., by moving the focus from one button to another button using a tab key or arrow keys). In these implementations, the focus selector moves according to the movement of the focus between different regions of the user interface. Regardless of the particular form the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) controlled by the user to convey the user's intended (e.g., by indicating to the device the user interface element with which the user wishes to interact) interaction with the user interface.For example, when a press-and-hold input is detected on a touch-sensing surface (e.g., a touchpad or a touch screen), the position of a focus selector (e.g., a cursor, a contact, or a selection box) over each button will indicate (as opposed to other user interface elements shown on the device's display) that the user is attempting to activate each button. Haptic Output and Related Processes

[0160] Here, attention is drawn to embodiments of haptic output and related processes that can be implemented in an electronic device, such as device 300 or portable multifunctional device 100. The following description relates to device 100, but the embodiments herein can be applied to device 300.

[0161] FIG. 5 is a diagram of examples of various haptic waveform morphologies according to some embodiments. The various haptic types function as minimal haptic components used in various combinations to generate various haptic waveforms with distinct meanings. The following examples are described to establish terms for distinguishing haptic output types herein. The haptic output types described herein are not intended to be exhaustive, and other haptic waveforms may be used.

[0162] According to some embodiments, waveform 505 represents an example of a haptic waveform with a relatively large amplitude (e.g., significantly exceeding the absolute threshold of touch perceptibility of a given haptic device) or a "tap" type of haptic output that can be immediately perceived by a user of a device that outputs haptic even if it is a single tap. As seen in various tap haptic outputs 505a - 505e, the waveforms vary in amplitude, duration, and frequency. Typically, humans can perceive frequencies from 5 Hz to 29 Hz and amplitudes from 0 N to 1.0 N (equivalent to about 0 - 40 dBA), but it depends on kHz. Human hearing of audible amplitudes is most sensitive at 1 - 4 kHz.

[0163] According to some embodiments, waveform 510 represents an example of a haptic waveform, or "microtap" type of haptic output, that has a relatively small amplitude (and optionally, a shorter duration) compared to waveform 505 which is not immediately perceivable to a user of a device that outputs haptics for detection as a single microtap, but is easily perceivable when multiple microtaps are output in succession (e.g., near a low threshold of touch perceptibility of a given haptic device). Microtap waveforms 510a - 510c also vary in amplitude, duration, and frequency.

[0164] Waveform 515 represents an example of a combined haptic of two microtaps following a tap. Since the microtaps are not easily perceivable alone, a waveform having a tap preceding the microtaps can "prepare" the user to more quickly perceive the microtaps by drawing the user's attention with the tap.

[0165] Waveform 520 represents an example of a "fade" type of waveform having a longer duration wave with a gradually decreasing amplitude. Waveform 520a is a "tap fade" that starts with a larger tap - level amplitude and ends with a microtap - level amplitude, and waveform 520b is a microtap fade that starts with a smaller microtap - level amplitude and ends with an even lower amplitude.

[0166] Waveform 525 represents an example of a "buzz" type of waveform that is a continuous high - frequency, typically lasting for a relatively longer overall duration (e.g., between 0.5 seconds and 1 second). The buzz may have a higher or lower amplitude according to various embodiments and is perceivable by the user as a consistent vibration.

[0167] Various audio waveforms are optionally output in synchronization with various tactile waveforms so as to correspond to or distinguish from the frequency, amplitude, and duration characteristics of the tactile output in the audio domain. Various minimal tactile components as described above are used in various combinations to generate various tactile waveforms corresponding to the audio waveforms. The minimal tactile components are modulated using various parameters (e.g., amplitude, number / repetition, timing offset) to generate various audio patterns directly from the tactile device itself such that the audio waveforms correspond to the tactile waveforms. Additionally, some waveforms generated by tactile actuators (e.g., (plural) tactile output generators 167) generate audible outputs of mechanical sounds such as "clicking sounds" according to some embodiments. Alternatively, in some embodiments, separate audio signal generators and output devices (e.g., tone generators and speakers) can be used to output in synchronization audio waveforms having waveforms distinct from the tactile waveforms.

[0168] FIG. 6 is a diagram of examples of various audio waveform forms according to some embodiments. Waveform 605 represents an example of an audio waveform of various amplitudes, durations, and frequencies. Special waveforms may be generated to mimic normal sounds correlated with the activity to which the audio is to be paired. Waveform 610 is an example of a special waveform. Waveform 610a is a waveform mimicking the sound of a transceiver, and waveform 610b is a waveform mimicking a heartbeat sound.

[0169] In some embodiments, when the tactile waveform and the audio waveform have corresponding forms, for example, when the tactile waveform and the audio waveform have peaks of the same duration at the same time such that the two waveforms appear similar to each other, these waveforms are output synchronously. Waveforms 615 (tactile, dotted line) and 620 (audio, solid line) are shown separately and then aligned and time synchronized. In this example, the time alignment synchronizes the tactile waveform with a series of taps having the audio waveform of a heartbeat to generate a combined sound, for example, to generate the sensation of a heartbeat for use in conjunction with a heart monitor.

[0170] In some embodiments, the audio components share attributes but are still different. For example, the same incoming call tone or musical score (same amplitude, frequency, and duration) played at different pitches or using different instruments is used. Overlaying different audio (e.g., the same audio played at a high pitch and a low pitch, a metallic sound and a glass sound and a ceramic sound) on the same tactile output generates different semantic components. These examples are not intended to be exhaustive, and other audio waveforms may be used.

[0171] For example, each haptic output may be accompanied by a corresponding audio output, may be accompanied by at least a portion of each audio output that is used in combination with at least a portion of each haptic output, or each audio output may occur in temporal proximity to each haptic output such that each haptic output and each audio output are perceptually simultaneous or synchronized. The haptic waveform and the audio waveform need not be perfectly aligned, and the device may take into account the fact that, for a particular class of haptic and audio outputs, the user will perceive the haptic and audio outputs as being simultaneous or synchronized even if they are slightly temporally offset (e.g., because the audio output is processed more quickly than the haptic output, providing the haptic output before providing the audio output may, in some cases, cause the user to perceive the audio output and the haptic output as occurring simultaneously or in synchronization). The changes in the accompanying audio portion may also vary between embodiments that include an audio output. For example, each of the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more audio qualities (e.g., pitch, timbre, etc.) associated with the audio component associated with the first output may vary in a particular situation. Haptic Output Based on Application State

[0172] According to some embodiments, one or more states associated with an application at a point in time associated with an alert condition detected for the application serve as a basis for providing a corresponding haptic output. Providing a haptic output correlated with the state of the application creates a more efficient human-machine interface, thereby reducing the time taken by the user to perform an action, and as a result, reducing energy usage and increasing the battery life of the battery powering the device. FIG. 7 is a flowchart of a method 700 for detecting an alert condition associated with an application and providing a corresponding haptic output, according to some embodiments. Note that in some embodiments, steps different from those shown in FIG. 7 may be performed.

[0173] This method is initiated by device 100 that detects alert conditions 705 associated with an application running on computing device 100. For an application running on computing device 100, alert conditions take various forms according to various embodiments. An alert condition is any event, notification, or other alert directed to a user of the device. One type of alert condition corresponds to an event that is automatically triggered by the application or triggered from within the application. For example, alert conditions triggered by the application include predetermined or scheduled alerts such as reminders, scheduled notifications, or alarms generated by the application. A second type of alert condition is an automatically initiated event notification received by the application from an origin external to the device. For example, system-generated emails or messages such as bulk mass emails, spam, emails generated by a system to a mailing list, or communications generated by any other system. A third type of alert condition is a manually initiated event notification received by the application from a human user other than the user operating the device. For example, an incoming message or call from a known email address or phone number in the user's contact list. Various alert conditions can be detected, for example, detecting the receipt of an email or text message while the user interface of an email or messaging application is displayed on the device's display.

[0174] According to some embodiments, in response to an alert condition, device 100 determines 710 the state associated with the application at the time associated with the alert condition. In one example, the time associated with the alert condition is the time the alert is received. In a second example, the time associated with the alert condition is the time the alert is detected.

[0175] In some embodiments, determining, at 710, the state associated with the application at a time associated with an alert condition includes determining whether a user interface for the application was displayed on the device at or around the time of the alert condition. Depending on the situation, the time may be the time of the alert condition, and in other situations, the time of interest extends over a specific time interval around the time associated with the alert condition, for example, within a defined time window that extends before and / or after the time associated with the alert condition.

[0176] In accordance with some embodiments, the state can be determined at 710, at a time associated with an alert condition, based on whether the application was running on the computing device. Such a determination is made by determining that the application was in an active state based on the application running in the foreground. For example, user interface elements can be seen on the device's display while the application is running. In this case, based on the application not running or running in the background on the computing device, the application is determined to have been in an inactive state. For example, the device was not actively used, or was not performing calculations at the direction of the application, or was not displaying user interface elements indicating the active state of the application. The combination of both the user interface display situation and the active state of the application is optionally used for the determination at 710.

[0177] Referring to FIG. 8, an example of an email application 800 actively running in the foreground on device 100 according to some embodiments is shown, and the user interface for that application can be viewed. In this example, the email application 800 is in an active state. In contrast, an example of an email application that is inactive on device 100 is shown in FIG. 4A, and the email application 800 is not visible on device 100, so it is either not running or running in the background.

[0178] Referring again to FIG. 7, in this example, at the time associated with the alert condition, the state associated with the application is determined at 710 based on whether each user interface window corresponding to the application is displayed in the foreground of the user interface of the multi-application window at the time associated with the alert condition. For example, when multi-application windows are displayed overlapping, the application at the frontmost position in the z-layer is considered the active application. As is well known in the art, each user interface window may be referred to as an application window that has focus and is configured to receive and / or accept user input or user interaction.

[0179] According to some embodiments, one or more other user interface windows are simultaneously displayed in the user interface of the multi-application window on the device at the time associated with the alert condition. In this case, determining the state associated with the application at 710 at the time associated with the alert condition includes determining whether each user interface window corresponding to the application was displayed in the user interface of the multi-application window at the time associated with the application.

[0180] According to some embodiments, another way to determine, at 710, the state associated with the application at a time associated with an alert condition is to determine, using a user interaction different from the alert condition, whether user input to the application was detected at a time associated with the alert condition. For example, user input in various situations includes user touch input, voice input, or vision / gaze-based input, or any means of receiving user input to the application. In some cases, the time associated with the alert is the time of the alert condition, and in other situations, it includes a specific time interval around the time associated with the alert condition, e.g., within a defined time window that extends before and / or after the time associated with the alert condition.

[0181] In some embodiments, to determine, at 710, that the application was in an active state based on user interaction detected at a time associated with the alert condition, device 100 further determines the level of user engagement associated with the user interaction at the time associated with the alert condition. User engagement in some situations is the range of possible user interactions with the application, ranging from a minimum level of engagement such as just a line of sight, through a medium level of engagement such as voice or touch only, to a greater level of engagement such as a combination of touch, voice, and line of sight to focus on the application (e.g., either the current level of engagement, the level of engagement up to a predetermined time, or a combination of both).

[0182] Determining, at 710, the state associated with the application at a time associated with the alert condition includes, in some embodiments, determining the state of the computing device at the time associated with the alert condition. In this example, determining that the application was in an active state includes determining that the computing device was active at the time associated with the alert condition. According to some embodiments, for example, an active device includes a device that is powered on (e.g., the display is on) and / or in a state of active use. Similarly, determining that the application was in an inactive state includes determining that the computing device was inactive in some form at the time associated with the alert condition. For example, a device that is powered off, a device in sleep mode or hibernation mode, a device not in active use, a device with the display turned off, and corresponds to an inactive device in various situations, respectively.

[0183] According to some embodiments, in accordance with the determination that the application was in an active state at the time associated with the alert condition, device 100 provides, at 715, a first haptic output having a first set of output characteristics.

[0184] The characteristics of the first set for the first tactile output include one or more of the amplitude, duration, regularity, repetition frequency of the first tactile output, or changes in minimal tactile features. Depending on the situation, the first tactile output is accompanied by the first audio output. For example, each tactile output may be accompanied by a corresponding audio output, may be accompanied by at least a portion of each audio output used in combination with at least a portion of each tactile output, or each audio output may occur in temporal proximity to each tactile output such that each tactile output and each audio output are perceptually simultaneous or synchronized. The tactile waveform and the audio waveform do not need to be perfectly aligned, and the device 100 may consider the fact that for a particular class of tactile and audio outputs, even if the tactile and audio outputs are slightly temporally misaligned, they will be perceived by the user as occurring simultaneously or in synchronization (e.g., since the audio output is processed more quickly than the tactile output, providing the tactile output before providing the audio output may, depending on the situation, cause the user to perceive the audio output and the tactile output as occurring simultaneously or in synchronization). Changes in the accompanying audio portion can also vary between embodiments that include an audio output. For example, each of the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more audio qualities (e.g., pitch, timbre, etc.) associated with the audio component associated with the first output may vary in a particular situation.

[0185] According to some embodiments, the first and second sets of characteristics correspond to the device type of the computing device. For example, for the same event, device 100 optionally provides different haptic outputs based on the type of device (e.g., phone vs. wristwatch vs. laptop computer vs. other handheld device). However, in some embodiments, the difference between the haptic components of the first output and the haptic components of the second output is maintained independent of the device on which these outputs are executed. In this situation, the same pattern of haptic output is provided on all types of devices, but the difference in the amplitude of the haptic output is based on the device on which the output is executed.

[0186] Changes in the device itself, and thus the details of the haptic output, vary according to the various embodiments. For example, in some situations, the device has a touch-sensitive display. In some embodiments, the first haptic output and the second haptic output are provided via a touch-sensitive display on the computing device. In this example, device 100 is capable of both receiving the user's touch input via the touch-sensitive display and providing haptic output via the same touch-sensitive display. According to some embodiments, this two-way touch sensitivity enables device 100 to provide feedback based on the received user input.

[0187] As described above, device 100 optionally determines a level of user engagement associated with user interaction at a time associated with an alert condition. In these situations, device 100 further determines one or more of a first set of output characteristics for a first haptic output based on the determined level of engagement. In some embodiments, the amplitude, duration, and repetition frequency of the audio component and / or haptic component may vary inversely with the degree of user interaction / engagement with the application. Thus, for a higher level of user interaction, the first set of output characteristics is selected to give the first output a lower intensity so that it is more difficult to capture and more difficult to perceive. Thus, the user is not exposed to extremely strong signals when the user is already highly engaged or interacting with the application. According to some embodiments, when the user is less engaged, a higher intensity signal is provided to the user to draw further attention to the device or application. For example, if the detected user interaction includes a touch input or touch gesture (and / or the user's gaze) that reflects a high level of user interaction / engagement, the user is likely already aware of the alert condition because they are more engaged with the application (by interacting with the application via vision or touch), so a weaker or smaller voiced output is provided to the user. If the detected user interaction includes voice input but no touch or gaze that reflects a lower level or degree of engagement with the application, a stronger and louder voiced output is optionally provided to draw further attention to the alert condition.

[0188] In accordance with a determination that the application was in an inactive state at a time associated with an alert condition, device 100 optionally provides, at 720, a second haptic output indicative of the occurrence of the alert condition, the second haptic output having a second set of output characteristics, the second haptic output being different from the first haptic output.

[0189] According to some embodiments, the second haptic output is different from the first haptic output and has a greater intensity than the first haptic output. For example, optionally, the signal amplitude is greater, the frequency is higher, and / or the duration is longer than the first haptic output. The second haptic output has a greater intensity than the first haptic output depending on the situation. In some embodiments, the haptic component of the first output is composed of a first set of haptic features or minimal haptic components (e.g., micro taps) that are less pronounced during the active state of the device than a second set of parameters during the second state, having a first set of parameters (e.g., smaller amplitude, less periodicity / steadiness, lower repetition frequency or number of events, shorter duration of the output, less intrusive / less perceived / less vocalized). Similarly, when each audio component is associated, the audio component of the first output has a first set of parameters (e.g., smaller amplitude, shorter duration of the output, less intrusive / less perceived / less strong audio memo) during the active state and a second set of parameters during the non-active state. According to some embodiments, the purpose of the higher intensity output is to draw the user's attention to an alert condition when the application is in the non-active state.

[0190] As discussed above, the state of device 100 is optionally used to modulate the haptic output or to determine the output characteristics. For example, according to some embodiments, when the device is in a power saving state (e.g., because the battery life of the device has dropped below a predetermined threshold, such as 5%, 10%, 15%, or 20%), haptic outputs that save power (e.g., haptic outputs with a smaller amplitude, a shorter duration, and / or a lower frequency) are used even if these haptic outputs are difficult for the user to perceive. When the device is not in a power saving state, the initial haptic output settings are used depending on the situation even if these haptic outputs do not save power (e.g., haptic outputs with a relatively larger amplitude, a longer duration, and / or a higher frequency compared to the power saving haptic outputs). In some embodiments, the difference between the haptic components of the first output and the haptic components of the second output is maintained even when the power saving haptic output is used because the device is in a power saving state.

[0191] Similar to the first haptic output, the second haptic output is optionally accompanied by an audio output and has a waveform of the second haptic output that is generated based on, reflects, and is synchronized with the waveform of the accompanying audio output. In some embodiments, the waveform of the haptic component is generated from and matches, mimics, reflects, or is synchronized with the waveform of each audio component, as discussed in conjunction with FIG. 6.

[0192] Various combinations of haptic and audio can be output by the device to provide different semantic information to the user. As a first example, the first haptic output is not accompanied by an audio output, while the second haptic output is accompanied by an audio output. As a second example, the first haptic output is accompanied by a first audio output, and the second haptic output is accompanied by a second audio output that is different from the first audio output.

[0193] A given tactile waveform can generate different perceptions, so semantic information can be generated when different audio waveforms are associated. For example, the output of a tactile component accompanied by a high-frequency audio sound results in different perceivable units, so it provides semantic information when compared to the output of the same tactile component accompanied by a low-frequency audio sound. High-frequency sounds may be used to draw the user's attention to important occurrences in the class of events represented by the tactile output, while on the other hand, low-frequency sounds may be used to indicate state changes in the class of events. For example, a given tactile output may be used to indicate the receipt of a text message, and either a high-frequency sound for an urgent message (or a message from a specified sender) or a low-frequency sound for a non-urgent message or a message received from a non-specified sender may be associated. Additionally, for two related alerts (e.g., an alert corresponding to the receipt of a text message and an alert corresponding to the receipt of an email message), since they have audio components with the same rhythm corresponding separately, the same tactile component is used (e.g., the shared tactile component provides information that the two alerts correspond to message receipt alerts, while on the other hand, different audio components provide information about what type of message was received).

[0194] In some embodiments, the first tactile output is similar, identical, or substantially identical to the second tactile output. In one example, the first tactile output has an associated audio output, and the second tactile output has the same associated audio output. In this example, the first tactile output is different from the second tactile output. Refer again to the above examples of the active email application 800 in FIG. 8 and the non-active email application 800 in FIG. 4A. The following is a table showing exemplary alert condition information for the email application 800 and the outputs corresponding to various conditions, states, modes, and conditions. [Table 1]

[0195] Table 1 shows the possible changes in output based on the change of a single alert condition for the received email. The first seven rows represent the various modes that the application will take when an email is received, i.e., using the application to display the mailbox screen, display the inbox, display the selected message, display the sent message box, display the draft being created, the email search field, and an active application with no new messages when the application is running in the background. Since no new messages have been received in the first eight rows, there is no output (tactile or otherwise).

[0196] However, the remaining eight rows show the conditions under which a new message has been received and the corresponding output. As shown in the table, for most active modes, the tactile output is a micro tap and the audio output is the default audio for email reception. However, for the last row where the application is inactive and the second last row where the application is active but running in the background, the tactile output is a fade tap and the audio output remains the default for email reception. Thus, the audio output is maintained regardless of whether the application is active or not, but the tactile output has a stronger intensity (fade tap versus micro tap) when the application is inactive or running in the background. Table 1 provides the conditions and corresponding output according to some embodiments, which are merely illustrative. Other combinations of alerts, states, modes, conditions, and outputs may be used in different embodiments.

[0197] In one example, at a first time point, while the application is in an active state, device 100 detects a first alert condition associated with the application at 705 and provides a first output representing the first alert condition at 715 in response to detecting the first alert condition while the first application is in the active state. At a second time point, while the first application is in an inactive state, device 100 detects a second alert condition and provides a second tactile output representing the second alert condition at 720 in response to detecting the second alert condition while the application is in the inactive state. The second tactile output is different from the first tactile output.

[0198] It should be understood that the particular order described for the operations in FIG. 7 is merely exemplary and is not intended to indicate that the described order is the only order in which the operations can be performed. One of ordinary skill in the art will recognize various ways to reorder the operations described herein. Additionally, it should be noted that the details of the other processes described herein with respect to other methods (e.g., methods 900, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) are also applicable in a manner similar to the method 700 described above in connection with FIG. 7. For example, the input, alert condition, application, state, and tactile output described above with reference to method 700 optionally have one or more of the characteristics of the input, alert condition, application, state, and tactile output described herein with reference to the other methods (e.g., methods 900, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200). For the sake of brevity, these details are not repeated here. Tactile Output Based on Trigger of Alert Condition

[0199] Alert conditions can be triggered manually or automatically. According to some embodiments, whether a detected alert condition is triggered manually or automatically serves as a basis for providing a corresponding tactile output. Providing a tactile output that correlates with whether an alert condition is triggered manually or automatically creates a more efficient human-machine interface, thereby reducing the time it takes for a user to perform an action, and as a result, reducing energy consumption and increasing the battery life of the battery that powers the device. FIG. 9 is a flowchart of a method 900 for detecting alert conditions associated with an application and providing a tactile output depending on whether the condition is triggered by an event that was manually initiated or an event that was automatically initiated, according to some embodiments. Note that in some embodiments, different steps than those shown in FIG. 9 may be performed.

[0200] The method is initiated by a device 100 that detects an alert condition associated with an application at 905. For an application running on the computing device 100, the alert condition takes various forms according to various embodiments. The alert condition is any event, notification, or other alert for the user of the device. One type of alert condition corresponds to an event triggered by an event that was manually initiated, and a second type of alert condition corresponds to an event triggered by an event that was automatically initiated. Depending on the situation, the device 100 has a touch-sensitive display.

[0201] In response to detecting the alert condition, the device 100 determines at 910 whether the alert condition was triggered by an event that was manually initiated. This step includes determining whether the alert condition corresponds to an event initiated by a human user. Various situations can contribute to the device 100 determining whether an event was initiated by a human user, as will be further discussed below in conjunction with Table 2.

[0202] In some embodiments, determining at 910 that an alert condition characteristic indicates that the alert condition was triggered by an event manually initiated involves determining whether the alert condition corresponds to an event initiated by a human user other than the user of the computing device. Some exemplary conditions for this determination are also shown in Table 2 as described below.

[0203] Determining that an alert condition was triggered by an event manually initiated, in some embodiments, involves determining whether the alert condition corresponds to an input received from a human user, identifying the user of the computing device, and including an instruction from the human user to send an alert to the user of the computing device regarding the input. In some embodiments, the input received from the human user is from the user of device 100, but in some embodiments, the input is received from another user on another device (e.g., the sender of a personal message from another user to the device user).

[0204] Table 2 shows some exemplary situations under which device 100 has information used at 910 to determine whether an alert condition was triggered by an event manually initiated, whether the event was initiated by a human user, and whether the user is a different user than the user of device 100. The applications shown in Table 2 include messages, emails, phone calls, calendars, timers, activity monitors, and social networking, but any other application that provides similar information to device 100 for making this determination is also contemplated.

Table 2

[0205] For the examples shown in the table, the "manual / automatic" column indicates whether a particular alert, condition, and application corresponds to an alert that is considered to be initiated manually (artificially) or an alert that is considered to be initiated automatically, i.e., whether the alert is initiated by artificial means or by automatic means. For example, rows 2, 4, 6, and 7 each show a manual action by a human user. In these examples, the actions are clearly performed by the user of the device (the "user" column) through user input to an application on the device (e.g., a message, email, or phone application), so the device 100 can easily determine that the user was a human user and more specifically, the device user. However, in other situations, some alert conditions belong to a known type on device 100 as being generable either manually, by a human user, or automatically. For example, rows 1, 3, and 5 have alert conditions of this category. In particular, a new message received in a messaging application is usually a manual (artificially initiated) event, e.g., a text or other message sent by a different device user. However, the message can also be automatically generated by a system, such as a spam or advertising text message. Similarly, a new email message can be initiated either by a human user or automatically by a system (e.g., mass emails, spam emails, or listserv (registered trademark) emails for regular subscribers that are considered to be generated by bots). In the last example, an incoming call in a phone application is usually an artificially initiated event, but can also be an automatically initiated call by a system automatically dialing from a list of callers, e.g., as part of a donation solicitation or political campaign. In each of these situations, additional information regarding the origin of the alert condition is needed to more deterministically determine whether the event was initiated manually or automatically.

[0206] For example, for an email or message, device 100 can determine whether the alert condition applies to a personal communication from a sender in a contact list that is managed, created, controlled, or even associated with the user. The simplest example is an actual contact file such as a contact card with an email address or phone number, but other contact lists may also include email addresses or phone numbers that the user has previously communicated with or received communications from, depending on the situation. These examples are simply to show that there are many other means by which device 100 can check whether the communication appears to have been manually initiated by a human user.

[0207] Other examples are shown in the second through last rows of Table 2. In this example, the social networking application sends a notification that the user has tagged a post. While the communication itself arrives from the social networking server, the content of the notification indicates that the post was manually initiated by a human user connected to the device user within the social networking application.

[0208] According to some embodiments, in accordance with the determination that the alert condition is triggered by a manually initiated event, device 100 provides a first haptic output at 915 in response to the notification of the manually initiated event.

[0209] When the device 100 determines that an alert condition has been initiated by a manual event, haptic output is provided corresponding to the alert condition. According to some embodiments, the first haptic output is provided via a touch-sensitive display of the computing device. In some embodiments, this determination further includes determining a degree of urgency associated with the manually initiated event and modulating the intensity of the first haptic output at 912 based on the degree of urgency. For example, the device 100 analyzes the content of an incoming message from another end user to determine the urgency of the message. Thus, the device 100 verifies the date information in the email or the subject / contents of the message, the subject or keywords used in the message (e.g., "urgent"), or in some embodiments, metadata included in the message such as an "important" flag. According to some embodiments, the device 100 determines whether the sender or recipient of the message includes an individual flagged by the user as an important person or an individual with some other special notation. Thereafter, the device 100 provides at 915 a haptic output and / or an audio output whose intensity (e.g., amplitude, duration, frequency) is proportional to the urgency associated with the content of the incoming message. According to some embodiments, typically, notifications / outputs that are expected or not urgent have one set of output characteristics, and unexpected notifications have a stronger second set of output characteristics in an attempt to draw the user's attention and raise it.

[0210] Depending on the situation, device 100 determines the attributes of one or more contexts associated with a manually initiated event and modulates the intensity of the haptic output based thereon to 912. For example, various factors can be considered to evaluate the attributes of the context associated with an incoming manual notification and the intensity of the haptic or audio notification to be provided. Attributes of the context can include the identity of the recipient sending the message, the inclusion and identification of other recipients of the message, the timing of message reception, the user's location when the notification is received, the activity being performed by the user when the notification is received, the media used to convey the message, etc., but this enumeration is not exhaustive.

[0211] According to some embodiments, at this time, a first haptic output corresponding to a notification of a manually initiated event is stronger than a second haptic output corresponding to a notification of an automatically initiated event. For example, the stronger output is thought to draw the user's attention more to the manually initiated event. Depending on the situation, the haptic for a manually initiated alert may be useful for quickly drawing the user's attention to an event determined by a system that may be more personal, more targeted, and more likely to be directly relevant to the user of the device, as it requires the user to notice it immediately.

[0212] For example, according to some embodiments, the first haptic output is characterized by a first set of characteristics including one or more of the amplitude of the first haptic output, the duration of the first haptic output, the regularity associated with the first haptic output, the repetition frequency of the haptic features in the first haptic output, and the selection of the haptic features constituting the first haptic output. For example, the second haptic output is characterized by a second set of characteristics including one or more of the amplitude of the second haptic output, the duration of the second haptic output, the regularity associated with the second haptic output, the repetition frequency of the haptic features in the second haptic output, and the selection of the haptic features constituting the second haptic output.

[0213] Depending on the situation, two manually initiated events can be received consecutively. The first alert condition can be detected corresponding to an incoming email message from a first human sender, and in response to a determination that the incoming email message corresponds to a manually initiated event, device 100 provides a first tactile output. The second alert condition is detected corresponding to an incoming text message from a second human sender, and in response to a determination that the incoming text message corresponds to a manually initiated event, device 100 provides a first tactile output. According to some embodiments, numerous other variations are possible.

[0214] In response to the detection of an alert condition associated with an application, in accordance with a determination that the alert condition is triggered by an automatically initiated event, device 100 provides, at 920, a second tactile output corresponding to a notification of the automatically initiated event. The second tactile output is different from the first tactile output.

[0215] There are numerous examples of alert conditions triggered by events that are automatically initiated. Referring again to Table 2, the alert conditions for the calendar, timer, and activity monitor applications are each examples of various automations and are generated by the application itself. In a slightly different example, the last row shows an alert condition that is an overview of activities for social networking in the form of an email. In this example, the alert condition is scheduled and occurs periodically. Thus, while the overview may include some event information related to an action initiated by the user (such as the above example of being tagged in a post by another user), the email from the social network aggregates the activity over a day or a week including the specific post. However, since the email was not triggered by a request or instruction from a human user, it is not considered a manually initiated event (the email contains little information about manually initiated events). According to some embodiments, the second tactile output is provided via the touch-sensitive display of the computing device.

[0216] According to some embodiments, device 100 determines that the alert condition was automatically triggered by the application (e.g., a predefined or scheduled alert such as a reminder notification or automated alarm generated by the application), or a notification of an automatically initiated event received by the application from a source external to the device (e.g., an automatically / system-generated email or message). For example, the sender email address noreply@domain.com suggests that the email was sent by automated means since there is no actual human sender for replying to the email.

[0217] In accordance with some embodiments, based on the determination that an automatically initiated event corresponds to an event occurring outside device 100, device 100 provides a first variation of a second haptic output corresponding to a notification of the automatically initiated event occurring outside. Similarly, based on the determination that an automatically initiated event corresponds to an event initiated within the device, device 100 provides a second variation of a second haptic output that is generated internally and corresponds to a notification of the automatically initiated event. Thus, in accordance with some embodiments, different notifications are provided for the same application depending on whether the event was generated by an incoming message.

[0218] In accordance with some embodiments, it is shown that an alert condition has an automatic trigger when device 100 determines that the alert condition occurs at a predetermined time or reports that a predetermined trigger condition is met. For example, meeting some predetermined trigger conditions (e.g., achieving one's own activity goal for the day, leaving a geo-fenced area, etc.) is an automatic type of trigger.

[0219] Similar to a manually initiated alert trigger, device 100 analyzes the characteristics associated with the alert condition to determine one or more contextual attributes associated with the automatically initiated event and modulates the intensity of the second output based on the one or more contextual attributes by 912. Various factors are considered when evaluating the contextual attributes associated with an automatic notification and the intensity of the haptic or audio notification to be provided depending on various situations. For example, the type of notification (e.g., one-time event vs. recurring event such as a periodic / repeating alarm), the user's location at the time the event occurs, whether the notification was triggered externally or internally, the activity being performed by the user at the time the notification is received, the activity in response to an external notification, the medium used to convey the message.

[0220] Similar to manually-triggered events, for automatically-initiated events, device 100 determines the degree to which the urgency associated with the automatically-initiated event changes over time, and modulates a second tactile output at 912 over a specific time window to indicate the degree to which the urgency changes over time. For example, for an automatically-initiated (e.g., internally, pre-scheduled) trigger, a set of tactile parameters is modulated at 912 based on the urgency or timing of the event associated with the notification. For example, according to some embodiments, for a camera application, device 100 provides a series of microtap outputs that increase until a photo is taken, and the interval between these microtaps gradually decreases as the time to take the photo approaches, providing an output that changes over time to increase the awareness of urgency or to help the user understand the timing factor.

[0221] In some embodiments, a first set of characteristics for the first tactile output is different from a second set of characteristics for the second tactile output. Similarly, when each audio component is associated, the audio component of the first output has a first set of parameters (larger amplitude, longer duration of the output, stronger / more perceivable / more pronounced audio signature) for notifications of manually-initiated events rather than for notifications of automatically-initiated events.

[0222] Specific examples include detecting an alert condition associated with an application and determining whether the alert condition corresponds to a personal communication from a sender within a list of contacts associated with the user when a first tactile output corresponding to a personal alert notification is provided, or determining whether the alert condition does not correspond to a personal communication from a sender within a list of contacts associated with the user when a second tactile output corresponding to an automatic alert notification is provided. Here, the first tactile output has a greater intensity than the second tactile output.

[0223] The tactile output is optionally accompanied by a corresponding audio output, with at least a portion of each audio output being used in combination with at least a portion of each tactile output, or each audio output occurs in temporal proximity to each tactile output such that each tactile output and each audio output are perceptually simultaneous or synchronized. The tactile waveform and the audio waveform need not be perfectly aligned, and the device 100 may consider the fact that, for a particular class of tactile and audio outputs, the user will perceive them as occurring simultaneously or in synchronization even if the tactile and audio outputs are slightly out of temporal alignment (e.g., since the audio output is processed more quickly than the tactile output, providing the tactile output before providing the audio output may, depending on the situation, cause the user to perceive the audio output and the tactile output as occurring simultaneously or in synchronization). Changes in the accompanying audio portion may also vary between embodiments that include an audio output. For example, each of the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more audio qualities (e.g., pitch, timbre, etc.) associated with the audio component associated with the first output may vary in a particular situation.

[0224] According to some embodiments, similar to the first tactile output, the second tactile output is accompanied by an audio output and has a waveform of the second tactile output that is generated to be similar and synchronized based on the waveform of the accompanying audio output. In some embodiments, the waveform of the tactile component is generated from and matches, mimics, reflects, or is synchronized with the waveform of each audio component, as discussed in conjunction with FIG. 6.

[0225] According to some embodiments, various combinations of tactile waveforms and audio waveforms are output. According to some embodiments, a first tactile output is not accompanied by an audio output, while a second tactile output is accompanied by an audio output. In one example, the first tactile output is accompanied by a first audio output, and the second tactile output is accompanied by a second audio output. In this case, the first tactile output is the same as the second tactile output, and the first audio output is different from the second audio output. In some embodiments, the same tactile waveform generates different perceptions when different audio waveforms are associated. For example, if an audio (incoming call) tone is played at a high pitch instead of a low pitch and the same tactile component (or slightly similar tactile components that a normal user would not be able to distinguish between them) is associated, the perception of the tactile component is different even though the tactile is the same. The higher or more shrill version of the incoming call tone is thought to be used with the second output to attract and raise the user's attention. In some embodiments, the first tactile output is similar to, the same as, or substantially the same as the second tactile output.

[0226] As another example, the first tactile output is accompanied by an audio output, and the second tactile output is accompanied by the same audio output. In this example, the first tactile output is different from the second tactile output.

[0227] The specific order described for the operations in FIG. 9 is merely exemplary and is not intended to indicate that the described order is the only order in which the operations can be performed. One of ordinary skill in the art will recognize various ways to reorder the operations described herein. Additionally, it should be noted that the details of the other processes described herein with respect to other methods described herein (e.g., methods 700, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) are also applicable in a manner similar to method 900 described above in connection with FIG. 9. For example, the input, alert condition, event, application, state, and haptic output described above with reference to method 900 optionally have one or more of the characteristics of the input, alert condition, event, application, state, and haptic output described herein with reference to the other methods described herein (e.g., methods 700, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200). For the sake of brevity, these details are not repeated here. Haptic output based on feedback pair notification

[0228] Alert conditions can be triggered by user input or by a predetermined system event (e.g., reminder, notification). According to some embodiments, whether the detected alert condition is associated with user input or with a predetermined system event serves as a basis for providing a corresponding tactile output. Providing a tactile output that correlates with the type of alert condition creates a more efficient human-machine interface, thereby reducing the time it takes for the user to perform an action, and as a result, reducing energy consumption and increasing the battery life of the battery that powers the device. FIG. 10 is a flowchart of a method 1000 for detecting a first alert condition from user input and a second alert condition from a predetermined event associated with an application and providing a corresponding tactile output, according to some embodiments. Note that in some embodiments, steps different from those shown in FIG. 10 may be performed.

[0229] The method begins by detecting, at 1005, a first alert condition on a computing device 100 associated with the receipt of user input for an application. According to some embodiments, the first alert condition can be any of the various alert conditions described elsewhere in this specification, and the application can be any application running on the device 100. According to some embodiments, the computing device 100 includes a touch-sensitive display for receiving user input.

[0230] According to some embodiments, in response to detecting a first alert condition, device 100 provides, at 1010, a first haptic output having a first intensity and corresponding to a user input to an application. The first haptic output is optionally feedback that is provided directly in response to the user input and as a result of the user input. According to some embodiments, a stronger haptic (e.g., a larger amplitude or a longer haptic) is used for an unexpected alert that is not feedback directly responsive to the user input and not in response to the user input, rather than for feedback directly responsive to the user input. In some embodiments, an unexpected alert is an alert that occurs outside of the context in which the user's attention is directed to the device (e.g., an alert that occurs while the user is not actively using the device or while the user is not looking at the device). For example, an electronic message alert generated by the device while the user of the device is wearing the device but not looking at the device and not interacting with the messaging application is an unexpected alert, while the same alert received while the user is looking at the device and / or while the messaging application is open on the device is considered a predicted alert. Similarly, feedback for a particular user input (e.g., a "click" when the user places a pin on a map) is predicted by the user because the user is actively engaged with the device and thus a less intense haptic alert is more noticeable. According to some embodiments, the first haptic output is provided via a touch-sensitive display on computing device 100. The first intensity can be a predetermined or adjustable combination of the amplitude, frequency, and / or duration of the haptic waveform, as described in other parts of this specification.

[0231] Device 100 also detects, at 1015, a second alert condition associated with the receipt of a predetermined system event in an application. For example, the second alert condition corresponds to an event that is automatically triggered by or within the application. Such events include, according to various embodiments, an alert that occurs at a predetermined time or reports meeting a predetermined criterion, a reminder notification or alert generated by the application, a notification of an automatically initiated event received by the application (e.g., an auto / system-generated email or message such as a bulk email, spam, or an email generated by the system to a mailing list sent by the system), or a notification of a manually initiated event received by the application (e.g., an event initiated by a human user such as an incoming message or call from a known email address or phone number in the user's contact list).

[0232] In response to detecting the second alert condition, device 100 provides, at 1020, a second tactile output having a second intensity corresponding to the predetermined system event, the second intensity being greater than the first intensity. The second intensity is greater than the first intensity in one or more of amplitude, frequency, and / or duration. When computing device 100 includes a touch-sensitive display, the second tactile output is provided via the touch-sensitive display on the computing device.

[0233] The greater second intensity of the second haptic output is designed to draw the user's attention to alert conditions that do not result from user input. For example, user feedback while using an application (in this case, the finger may already be touching the device 100) may require a weaker haptic output for the user to notice the feedback compared to a user not touching the device with their finger. Feedback while in contact with the device is expected by the user and may optionally be less obtrusive and smaller than an unexpected output. According to some embodiments, direct manipulation should result in short, quick haptic feedback (< 100 milliseconds). Haptic feedback should not be too strong (e.g., lower amplitude, fewer haptic features, longer periodicity / repetition, shorter duration, weaker haptic features, etc.) when it is a response to user input. On the other hand, notifications are often unexpected outputs and may therefore need to be stronger in some situations to draw the user's attention. Thus, notifications are optionally longer (about 1.5 seconds) and stronger.

[0234] As an example, according to some embodiments, device 100 detects, at 1005, a first alert condition corresponding to a user's selection of a user interface element displayed on an application user interface associated with a first application, and in response to the first alert condition, provides, at 1010, respective first tactile outputs representing the user's selection of the user interface element, each first tactile output including a first tactile component including first tactile features of a first intensity and a first duration. According to some embodiments, after providing each first tactile output, device 100 detects, at 1015, a second alert condition corresponding to an alert notification received by the first application, and in response to the second alert condition, provides, at 1020, respective second tactile outputs representing the receipt of the alert notification, each second tactile output including a second tactile component including the following. (1) Second tactile features of a second intensity and a second duration, and (2) being more pronounced than the first tactile output, different from each first tactile output based on the second tactile features being more pronounced than the first tactile features, the second intensity being greater than the first intensity, or the second duration being longer than the first duration. For example, an incoming phone call is considered to have a more pronounced tactile output for answering or hanging up the incoming call on the phone application.

Table 3

[0235] Table 3 shows the telephone application and the outputs corresponding to three different alert conditions: incoming call, response to an incoming call, and hang up an incoming call. In this example, since the action is predicted by the user who responds to or hangs up an incoming call, the corresponding haptic output associated with the action is considered feedback to the user. For these two alert conditions, the haptic output is not very strong (e.g., a micro tap). In contrast, an incoming call is a more unexpected type of alert condition and, in fact, something that exists for a limited time, so it has a stronger haptic (e.g., a double buzz) with the intention of attracting the user's attention. According to some embodiments, a stronger haptic (e.g., a larger amplitude or a longer haptic) is used for unexpected alerts that are not a response to user input rather than for feedback that directly responds to user input.

[0236] Table 3 provides conditions and corresponding outputs according to some embodiments, which are merely illustrative. Other combinations of alerts, states, modes, conditions, and outputs are used in different embodiments.

[0237] Depending on the situation, the haptic output may vary based on whether the haptic output results from a particular user input. In some embodiments, device 100 first detects a first alert condition associated with a first application, and the first alert condition results from each user input. For example, a user who gives a touch types by entering an email message in an email application. According to some embodiments, device 100 provides each first haptic output representing the first alert condition. Continuing with the above example, the device provides a short interval buzz at a low intensity for each user touch input. After providing each first haptic output, device 100 detects a second alert condition associated with the first application. The second alert condition does not result from the receipt of user input. In the context of an email application, this alert condition is optionally an incoming email within the email application. According to some embodiments, device 100 then provides each second haptic output representing the second alert condition, and each second haptic output has a greater intensity, different from each first haptic output. For example, the second haptic output optionally has a greater intensity and / or a longer tap to notify the user of the device of the incoming message.

[0238] According to some embodiments, the first and second haptic outputs can have various characteristics that affect their intensity and their ability to draw the user's attention. For example, the first haptic output can have a first set of characteristics that include one or more of the amplitude of the first haptic output, the duration of the first haptic output, the regularity associated with the first haptic output, the repetition frequency of the tactile features in the first haptic output, the selection of the tactile features that make up the first haptic output, and the second haptic output can have a second set of characteristics that include one or more of the amplitude of the second haptic output, the duration of the tactile components associated with the second haptic output, the regularity associated with the second haptic output, the repetition frequency of the tactile features in the second haptic output, the selection of the tactile features that make up the second haptic output. According to various embodiments, any of the characteristics can differ between the first and second haptics, or all of the characteristics can differ.

[0239] According to some embodiments, various combinations of haptic and audio are output. According to some embodiments, the first haptic output has no accompanying audio output, while the second haptic output has an accompanying audio output. In one example, the first haptic output has an accompanying first audio output, and the second haptic output has an accompanying second audio output. In this case, the first haptic output is the same as the second haptic output, and the first audio output is different from the second audio output. In some embodiments, the same haptic waveform produces different perceptions when different audio waveforms are associated with it. For example, if an audio (incoming call) tone is played at a higher pitch than a lower pitch, and the same tactile component (or a slightly similar tactile component that an average user would not be able to distinguish between them) is associated with it, the perception of the tactile component will be different even though the haptics are the same. The higher or shriller version of the incoming call tone is thought to be used with the second output to draw and increase the user's attention. In some embodiments, the first haptic output is similar, the same as, or substantially the same as the second haptic output.

[0240] According to some embodiments, any of the haptic outputs can be accompanied by an audio output, and is accompanied by a waveform of a second haptic output that is generated based on, reflected by, and synchronized with the waveform of the accompanying audio output. In some embodiments, the waveforms of the haptic components are generated from and match, mimic, reflect, or are synchronized with the waveforms of the respective audio components, as discussed in conjunction with FIG. 6. The haptic waveform and the audio waveform need not be perfectly aligned, and the device 100 may take into account the fact that for a particular class of haptic and audio outputs, the user will perceive them as occurring simultaneously or in synchronization even if the haptic and audio outputs are slightly out of time with each other (e.g., because the audio output is processed more quickly than the haptic output, providing the haptic output before providing the audio output may, depending on the situation, cause the user to perceive the audio and haptic outputs as occurring simultaneously or in synchronization). Changes in the accompanying audio portion can also vary between embodiments that include an audio output. For example, each of the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more audio qualities (e.g., pitch, timbre, etc.) associated with the audio component associated with the first output can vary in a particular situation.

[0241] In some embodiments, there is a linkage between the haptic intensity and the volume of the device. For example, the first intensity of the first haptic output and the second intensity of the second haptic output are adjusted inversely proportional to the volume setting on the device such that they are optionally increased either independently of the haptic output or in addition to the initial set intensity / predetermined intensity if the audio channel is turned off or attenuated.

[0242] In some embodiments, before providing the first haptic output, the device 100 determines at 1025 whether there are modulation conditions based on the input, and in response to determining that there are modulation conditions based on the input, before providing the first haptic output, the device 100 modifies the first haptic output at 1030.

[0243] For example, one modulation condition based on the input is user engagement. In this case, the device 100 detects each alert condition and determines that each alert condition originated from each user input received on the computing device. Then, according to some embodiments, the device 100 identifies the degree of user engagement associated with the detected user input and modulates the first signal strength of the first haptic output based on the degree of user engagement. For example, in some embodiments, the intensity (e.g., amplitude, duration, repetition frequency) of the audio component and / or the haptic component may vary inversely with the degree of user input interaction / user engagement with the application / device. For a higher degree of user interaction / engagement (such as a touch or direct operation), the output characteristics are selected to make the first output more difficult to capture, or weaker / less perceivable. Thus, the user does not receive an extremely strong feedback signal when the user is engaged in a high degree of interaction or dialogue with the application / device. For example, the weakest output is provided for a touch / direct operation. For example, when the user is in contact with the device while providing input, the user should not receive an extremely strong feedback signal. On the other hand, when the user is less engaged, a stronger signal is provided to the user to draw further attention.

Table 4

[0244] Table 4 shows various levels of user engagement with alert conditions involving changes in the user's focus, and the resulting haptic output levels. The haptic output levels shown in the table can be considered as a sliding scale of the level of haptic intensity required to draw the user's attention, taking into account the primary and secondary user focus. For example, the highest level of user engagement is the direct manipulation of the application that generates the alert condition when the alert condition is in the form of feedback. For example, when the user is inputting information into an application, the application is already the primary focus of the user, and only a slight haptic is needed to draw the user's attention, and the lowest level of haptic "Level 1" can be output. This can be the smallest haptic possible, for example, even a single micro tap.

[0245] Jumping to the third line, when the user is touching the device but inputting into a different application, the user is somewhat engaged as they are focused on the device, but they are using an application different from the one providing the alert, so they are less engaged than if they were inputting into the application providing the alert. In this case, the appropriate output is a haptic output of "Level 2". For example, the output of Level 2 can be considered as two consecutive micro taps.

[0246] Jumping from the second line to the last line, the user can have an application that actively issues alerts but operates in the background, and may not be touching the device at the time of the alert condition. As a result, the device may choose a greater haptic in this situation to gain the user's attention. Therefore, the haptic output of "Level 3" can be appropriate, for example, a priming tap, and can be followed by a single micro tap or consecutive micro taps.

[0247] Finally, the last row indicates a user having an application that alerts in an inactive state. Depending on the situation, the application may or may not be running in the background, or the device may be in a sleep state. In this example, the maximum haptic "level 4" is appropriate to gain the user's attention. For example, a double-buzz haptic output may be considered appropriate depending on the situation. According to some embodiments, a stronger haptic (e.g., a larger amplitude or a longer haptic) is used for unexpected alerts that are not in response to user input, rather than for feedback that directly responds to user input.

[0248] This is merely an example using a limited number of situations and haptic output examples, assuming that all applications and alerts are treated the same way, and this may not be the case in other situations. Additionally, a particular level of haptic output may be the "default" output, e.g., level 2, in which case the device may adjust the intensity of the haptic output up or down depending on the user's focus. Table 4 provides conditions and corresponding outputs according to some embodiments, but this is merely illustrative. Other combinations of alerts, states, focus, conditions, and outputs are used according to various embodiments.

[0249] In some embodiments, the haptic output is modified based on the location of the user input. In this example, device 100 detects each alert condition and determines that each alert condition results from the receipt of a tactile user input on the device. Device 100 modulates the spatial haptic energy profile of the first haptic output by reducing the amplitude of the haptic energy of the first haptic output so that the haptic output can be perceived at the location on the device where the tactile user input is received. According to some embodiments, the amplitude of the haptic energy can be reduced so that it cannot be perceived at locations remote from the location of the user's finger / tactile input. For example, a low amplitude can be perceived by the finger at the point of contact but not by the rest of the hand and / or wrist. Similarly, in some embodiments, device 100 detects each alert condition and determines that each alert condition results from the receipt of a tactile user input on the device, but in that case, specifically identifies the location on the device where the tactile user input was received and provides more than a specific percentage of the first haptic output within a specific radius threshold of the location on the device where the tactile user input was received, thereby modulating the spatial haptic energy profile of the first haptic output. According to some embodiments, since the location on the device where the tactile user input is received changes over time, the spatial haptic energy also changes over time corresponding to the temporally changing location on the device where the tactile user input is received.

[0250] According to some embodiments, the morphological characteristics of the tactile output mimic the tactile input. For example, device 100 detects each alert condition, determines that each alert condition has resulted from the reception of a tactile user input, and then identifies the temporally varying morphological attributes of the tactile user input. According to various embodiments, the temporally varying morphological attributes of the tactile input include monitoring the detection of contact on the touch sensing surface, whether the contact is maintained with the sensing surface, how far apart, and in which direction it is moving, determining the amount of pressure placed at the contact point, and the like. The device 100 then modifies the morphological characteristics of the first tactile output over time to mimic the temporally varying morphological attributes of the tactile user input. For example, the morphological characteristics of the first tactile output for mimicking include varying the amplitude, duration, frequency, etc. of the tactile output to reproduce the tactile input.

[0251] In some embodiments, the waveform of the haptic component need not be perfectly aligned with the user haptic input, and the device 100 may take into account the fact that the haptic outputs will be perceived by the user as occurring simultaneously or in sync even if the haptic outputs are slightly out of time (e.g., the haptic outputs are processed faster than the user can perceive them as a delayed response, so the user will perceive the haptics as occurring simultaneously or in sync). In some embodiments, a particular haptic feature among a given set of available haptic features is selected to compose the haptic component of the first output based on the haptic feature best mimicking the user input. Some examples include a microtap for a simple touch, a pen click tap for a longer / deeper touch, a buzz for a swipe, and a growing buzz at the end for a swipe that ends with a finger lift. For example, in some embodiments, the time-varying morphological attribute includes a time-dependent contact pressure profile between the haptic user input and the touch sensing surface, and modifying includes modulating the energy profile of the first haptic output over time to reproduce the time-dependent contact pressure profile between the haptic user input and the touch sensing surface of the computing device.

[0252] According to some embodiments, the haptic output can mimic the duration of continuous user input, gesture, mechanical adjustment, or other device operations by the user. For example, device 100 first detects each alert condition and determines that each alert condition has arisen from continuous user input. According to some embodiments, continuous user input includes tactile user input on a touch-sensitive surface on a computing device, rotation of a knob, depression of a button, or any other user input of a device operation with a detectable start / end point. In some embodiments, the device can determine that the alert condition has arisen from a gesture, mechanical adjustment, etc. In response to that determination, device 100 starts a first haptic output at the start point of the continuous user input, detects the end of the continuous user input, and then ends the first haptic output (e.g., mimicking the user input) at the end point of the continuous user input. In some embodiments, the haptic output is started by the user input at the start point of the user input and ends at the end point of the user input. Thus, the haptic feedback / haptic output continues almost throughout the duration of the user input. The haptic output is not limited to the touch-sensitive surface and is considered to be provided at the location of the user input. For example, when turning a rotary knob on the side of a wristwatch, it is considered to provide continuous feedback as the user turns the knob, with a tactile sensation like a click.

[0253] In some embodiments, before providing the second haptic output, device 100 determines at 1035 whether there is a modulation condition based on ambient conditions, and in response to the determination that there is a modulation condition based on ambient conditions, before providing the second haptic output, the device modifies the second haptic output at 1040.

[0254] For example, device 100 determines whether ambient conditions are present as potential interferences to the user's perception of the haptic output. In this example, device 100 determines whether one or more ambient conditions are present with respect to the device as potential interferences to the haptic output at the time of occurrence of each alert condition. According to some embodiments, ambient conditions include noise (including measured ambient noise and identifying that the user is in a library, museum, movie theater, hospital, or other location generally known to be quiet), movement / vibration that affects the user's physical receptivity to haptic sensations, the level of engagement with the device (e.g., viewing, touching, conversing, etc. as discussed above), the proximity of the user to the device (e.g., being located further away from the device on the user's body than a specific proximity threshold from the device).

Table 5

[0255] Table 5 shows, according to some embodiments, various ambient conditions for various application states that may exist during alert conditions, and possible modifications to the haptic output provided based on these conditions. The ambient conditions shown in the table include low noise level and high noise level, low vibration level and high vibration level, short distance and long distance to the device, and high user engagement level and low user engagement level related to the above sections and Table 4. This table provides examples of various ambient conditions and shows how the haptic output might be modified to adjust to the ambient conditions for a given application state (active or inactive). Table 5 only shows modifications to the haptic output, but similar adjustments are considered to be made to non-haptic outputs such as audio. The column for haptic modifications lists possible modifications of none (maintain the initial haptic for the alert), increase by 1, 2, or 3, and decrease by 1, although many variations are possible. These are just examples. For some conditions such as high noise, low vibration (inactive), high vibration (active), and long distance to the device (rows 3, 4, 6, 7, and 11), the device can modify the initial haptic in one or two steps, for example, from a single microtap to a tap or two microtaps, or from a tap to a buzz, etc., in ambient conditions that may make the user somewhat distracted. For other conditions such as high vibration (inactive) or long distance to the device (rows 8 and 12), the device can modify the initial haptic by a larger amount, for example, by increasing from a microtap to a buzz, etc., in ambient conditions that may make the user more distracted. Depending on the situation, the haptic may be accompanied by an audio output as part of the modification. Alternatively or additionally to modifying the intensity of the haptic, the device may delay the haptic output for a user receiving the haptic output for ambient conditions marked, for example, as increase 3, until the ambient conditions improve. In this example, the device can periodically determine the ambient conditions to find out if it can output the haptic now that the interfering ambient conditions have subsided. In other situations, the delay is not applied.These examples are very limited, using a limited number of situations and haptic output examples, assuming that all applications and alerts are treated the same, which may not be the case in other situations.

[0256] In some embodiments, the ambient conditions do not require modification because they do not distract the user very much, if at all (e.g., the rows marked "none" in the modification column). Finally, depending on the ambient conditions, the device can reduce the initial haptics associated with the alert condition for a high level of engagement, for example, if a smaller output is required because the user is involved. Table 5 provides conditions and corresponding outputs according to some embodiments, which are merely illustrative. Other combinations of ambient conditions, alerts, states, and outputs are used in various embodiments.

[0257] According to some embodiments, device 100 delays the haptic output. First, device 100 makes a determination that there are no interfering ambient conditions present for the device. In this example, device 100 provides each second haptic output based on the user's receptivity state, and according to the determination that one or more ambient conditions are present for the device as potential interference to the haptic output, device 100 delays the provision of each second output to the user. For example, the delay may continue until the interfering ambient conditions no longer exist in a subsequent instance (e.g., device 100 determines again at some later point in time whether interfering ambient conditions are present). For example, when the user is very active (e.g., during running or training) when receiving a notification and thus unable to perceive the haptic sensation, the device can delay the haptic output until the activity is completed, such as waiting until the user finishes the activity, and then provide the haptic output corresponding to the notification. In other situations, the alert may be of sufficient urgency that the device cannot delay the alert regardless of the ambient conditions. In these examples, the determination of the ambient conditions functions as a surrogate for estimating the user's body receptivity to the haptic sensation, based on, for example, the characteristics of the user's current surroundings.

[0258] According to some embodiments, device 100 provides a variant of haptic output, for example, a modification described in conjunction with Table 5. In this example, device 100 detects each alert condition associated with each application and determines that each alert condition did not result from the receipt of user input. In response to the determination that each alert condition did not result from the receipt of user input, device 100 determines whether one or more ambient conditions exist for the device as potential interference with the haptic output at the time of occurrence of each alert condition, and according to the determination that there are no interfering ambient conditions for the device, device 100 provides the user with a first variant of a second haptic output, and the first variant of the second haptic output has a first set of output characteristics. For example, according to some embodiments, the variant can be a haptic output that includes softer, shorter-duration, more difficult-to-catch haptic characteristics (e.g., micro taps). According to the determination that one or more ambient conditions exist for the device as potential interference with the haptic output, device 100 provides the user with a second variant of a second haptic output, and the second variant of the second haptic output has a second set of output characteristics, the second set of characteristics being different from the first set of characteristics, and the second variant of the second haptic output has a greater intensity than the first variant of the second haptic output.

[0259] In some embodiments, device 100 can modify the haptic output to “prepare” the user. For example, device 100 first detects each alert condition associated with each application and then determines that each alert condition did not result from the receipt of user input. In accordance with some embodiments, in response to determining that each alert condition did not result from the receipt of user input, the device provides a priming haptic output as a precursor to a second haptic output, the priming haptic output being designed to increase the level of engagement of the user operating the device with respect to the second haptic output, and at least a portion of the priming haptic output is more pronounced than the second haptic output. Device 100 optionally provides the second haptic output following the provision of the priming haptic output and within a particular time interval of the provision.

[0260] According to some embodiments, device 100 can provide a variant of priming. In one example, the priming tactile output includes a priming tactile component with a temporally varying waveform pattern characterized by an intensity of the tactile component that gradually increases over time. In some embodiments, the priming tactile output includes an intensity of a gradually increasing tactile waveform to unconsciously prepare the user for a notification. According to some embodiments, after priming, a significant increase in audio volume occurs in response to the tactile component. In another example, the priming tactile output optionally includes a priming tactile component with a temporally varying waveform pattern characterized by an emphasized tactile feature before the intensity of the tactile component gradually decreases. Alternatively, according to some embodiments, device 100 can intentionally prepare the user's sensitivity by first providing a stronger priming tactile (and optionally audio) output, and then, when the user's perception is prepared, continue with a moderate output for that output. According to some embodiments, there is a priming tactile output prior to a second tactile output, and the first tactile output is provided directly in response to a corresponding alert condition resulting from a received user input without a corresponding preceding priming tactile output. In various embodiments, the characteristics of the priming tactile output are selected based on the urgency or context associated with the alert condition, with an intensity of the output reflecting the urgency of the alert condition.

[0261] It should be understood that the specific order described for the operations in FIG. 10 is merely exemplary and is not intended to indicate that the described order is the only order in which the operations can be performed. One of ordinary skill in the art will recognize various ways to reorder the operations described herein. Additionally, it should be noted that the details of the other processes described herein with respect to the other methods described herein (e.g., methods 700, 900, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) are also applicable in a manner similar to method 1000 described above in connection with FIG. 10. For example, the input, alert condition, application, and tactile output described above with reference to method 1000 optionally have one or more of the characteristics of the input, alert condition, application, and tactile output described herein with reference to the other methods described herein (e.g., methods 700, 900, 1100, 1200, 1300, 2500, 2600, 3000, and 3200). For the sake of brevity, these details are not repeated here. Tactile Output for Multipart Operations

[0262] In accordance with some embodiments, whether a detected alert condition is part of a multipart operation serves as a basis for providing a corresponding tactile output. Providing a tactile output related to whether an alert condition is part of a multipart operation creates a more efficient human-machine interface, thereby reducing the time it takes for a user to perform the operation, and as a result, reducing energy consumption and increasing the battery life of the battery that powers the device. FIG. 11 is a flowchart of method 1100 for detecting an alert condition associated with a multipart operation and providing a corresponding tactile output, according to some embodiments. Note that in some embodiments, different steps than those shown in FIG. 11 may be performed.

[0263] This method begins by receiving, at 1105, an input corresponding to a first part of a multi-part operation being performed by an application running on computing device 100. In some embodiments, the input corresponding to the first part of the multi-part operation is detected on the touch-sensitive surface of device 100.

[0264] In response to receiving the input corresponding to the first part of the multi-part operation, device 100 begins, at 1110, a continuous haptic output sequence. The continuous haptic output sequence can be maintained, constant, vary over time while continuing as long as the sequence continues until a second input is received, or continue in the sense of being intermittent, according to various embodiments. In some embodiments, in response to detecting the input corresponding to the first part of the multi-part operation, the device also begins a continuous audio output sequence to accompany the continuous haptic output sequence.

[0265] According to some embodiments, device 100 has a touch-sensitive surface and the haptic output sequence is provided via the touch-sensitive surface. For example, the haptic output includes haptic vibrations of the touch-sensitive surface in a predetermined pattern that is perceptible to a user of the device touching the touch-sensitive surface.

[0266] In some embodiments, in addition to the continuous haptic output sequence, device 100 provides, at 1115, a first haptic feedback in response to receiving the input corresponding to the first part of the multi-part operation. The first haptic feedback is selected based on an event, indicates the start of the event, and in some cases, is different from the continuous haptic output sequence. For example, according to some embodiments, the first haptic feedback has a haptic output of greater intensity than the continuous haptic output sequence.

[0267] According to some embodiments, after the start of the continuous haptic output sequence, device 100 receives an input at 1120 that corresponds to the second part of the multi-part operation. For a device 100 having a touch-sensing surface, the input corresponding to the second part of the multi-part operation is detected on the touch-sensing surface.

[0268] In some embodiments, two devices are related. Thus, inputs corresponding to both the first part of the multi-part operation and the second part of the multi-part operation are detected by device 100. However, in some embodiments, the input corresponding to the first part of the multi-part operation is detected by device 100, and the input corresponding to the second part of the multi-part operation is detected by a second device.

[0269] According to some embodiments, additional user input can be received by device 100. For example, according to some embodiments, after the start of the continuous haptic output sequence at 1110 in response to the first part of the multi-part operation and before the reception of the input corresponding to the second part of the multi-part operation at 1120, device 100 receives one or more additional user inputs at 1125 that are different from the input corresponding to the second part of the multi-part operation. In these situations, according to some embodiments, the device continues to provide continuous haptic output independent of the additional user input until it receives the input corresponding to the second part of the multi-part operation.

[0270] According to some embodiments, in response to receiving the input corresponding to the second part of the multi-part operation, device 100 ends the continuous haptic output sequence at 1130. In some embodiments, in response to detecting the input corresponding to the second part of the multi-part operation, device 100 also ends any ongoing audio output sequence.

[0271] The input corresponding to the second part of the multi-part operation follows the input corresponding to the first part of the multi-part operation and, in accordance with some embodiments, is eventually separated from the input corresponding to the first part of the multi-part operation by a specific interval of time corresponding to the duration of the event. Further, depending on the situation, the continuous tactile output sequence is provided over the entire duration of the event, indicating the occurrence of the event.

[0272] In accordance with some embodiments, the device also provides, at 1135, a second tactile feedback in response to receiving an input corresponding to the second part of the multi-part operation. The second tactile feedback is different from the continuous tactile output sequence. Depending on the situation, the second tactile feedback indicates the end of the event and is different from the continuous tactile output sequence. For example, in accordance with some embodiments, the second tactile feedback has a tactile output of greater intensity than the continuous tactile output sequence.

[0273] In accordance with various embodiments, the multi-part operation can take various forms. In accordance with some embodiments, the multi-part operation is a secure transaction and involves a first part of the multi-part operation that includes protecting the device to authenticate the secure transaction. In this example, depending on the user providing a credential, such as a password or biometric authentication such as a fingerprint, the device is moved to a payment mode where payment information can be provided to a payment terminal via a Near Field Communication (NFC) subsystem. Thus, protecting the device is the first part and is considered to initiate the corresponding continuous tactile output. Additionally, the second part of the multi-part operation, in this example, includes authenticating the secure transaction. Thus, in accordance with some embodiments, the authentication is a second step and includes, for example, providing payment information to the payment terminal when the NFC subsystem enters the range of the NFC area of the payment terminal while the NFC subsystem is protected from the first step.

[0274] In some embodiments of the NFC multipart secure transaction, the first part of the multipart operation includes determining that user authentication is required to complete a secure transaction. For example, in response to the NFC subsystem entering the range of the NFC area of the payment terminal, the payment terminal prompts the user to provide a credential such as a password or biometric authentication such as a fingerprint to authenticate a secure transaction. In this example, the second part of the multipart operation includes receiving user authentication for the secure transaction. For example, a tactile output is provided to alert the user that they need to provide user authentication information before the payment process can begin, and the user authentication information optionally includes authentication based on a passcode or biometric. In some embodiments, a first tactile output pattern is provided by the device to alert the user to provide user authentication information between when it is determined that user authentication is required and when user authentication is received (e.g., between the request for user authentication and the receipt of user authentication), and a second tactile output pattern is provided by the device between when user authentication information is received and when the secure transaction is approved (e.g., between the receipt of user authentication and the provision of payment information to the payment terminal). In this example, according to some embodiments, the second tactile output pattern is different from the first tactile output pattern to alert the user that the NFC subsystem is placed near the payment terminal.

[0275] According to other payment embodiments, the input is compatible with secure transactions that do not use NFC. In this example, the first part of the multi-part operation is the selection of the first user interface element to initiate the transaction. For a financial transaction such as an online purchase, by selecting an element corresponding to the activation of online payment, the user can input into the user interface for entering secure or confidential information. As another example, the user requests to log in to a secure location such as a secure email account or an online store. Continuing with the above example, the multi-part operation includes one or more inputs for the entry of information required to process the transaction. For example, the entry of credit card information or other verification information required for the authentication of a financial transaction and the corresponding continuous tactile output sequence are provided throughout the duration of the information entry. Similarly, the input may be the entry of a username and password to log in to a secure web account. In some embodiments, the event corresponds to the number of one or more data entry fields that receive the information required to process the transaction, and the input corresponding to the second part of the multi-part operation is the selection of a user interface element that authenticates the completion of the transaction, for example, a payment authentication user interface element.

[0276] According to some embodiments, the first part of the multi-part operation includes closing a draft of a document, and the second part of the multi-part operation includes returning to the draft of the document. Examples of closing a document include, according to various embodiments, closing the draft of an email or text message to switch to another task or open another application, and returning to the document includes switching back to the draft of the email or text message from another task or another application.

[0277] In some embodiments, the inputs are individual inputs that occur close to each other in time. For example, according to some embodiments, the input corresponding to the first part of the multi-part operation is a first user input that interacts with an application running on the device, and the input corresponding to the second part of the multi-part operation is a second user input that interacts with the application, and the second user input is different from the first user input. For example, the first user input to the application is considered to be a first tap and lift-off, and the second user input is considered to be a second tap and lift-off within the same application.

[0278] Depending on the situation, the multi-part operation is a multi-part gesture. For example, according to some embodiments, the operation includes first detecting an input corresponding to the first part of the multi-part operation as a contact on the touch-sensing surface of the device 100, and detecting an input corresponding to the second part of the multi-part operation includes detecting a movement of the contact on the touch-sensing surface. In the example of a swipe gesture, the first part is the user touching the touch-sensing surface, and the second part is the user maintaining contact with the touch-sensing surface while moving the contact across the touch surface. In another example, the operation includes first detecting an input corresponding to the first part of the multi-part operation as a movement of a contact on the touch-sensing surface and detecting an input corresponding to the second part of the multi-part operation as a detection of a lift-off of the contact from the touch-sensing surface.

[0279] According to some embodiments, the inputs corresponding to the first and second parts of the multi-part operation include a single gesture, the input corresponding to the first part of the multi-part operation is the initial part of the gesture (e.g., touch and hold by finger contact), and the input corresponding to the second part of the multi-part operation is the subsequent part of the gesture (e.g., lift-off of finger contact). For example, in a text editing or viewing application, placing a finger on a word and holding it down first highlights the word, and then continuing to place a finger on the highlighted word and holding it down launches the cut-copy-paste menu. In this example, the lift-off of the contact is considered to represent the selection of the displayed aspect corresponding to the touch-and-hold.

[0280] In some embodiments, the gesture is initiated on a portion of the touch-sensing surface corresponding to a first position within a user interface displayed on the display of device 100 and ends on a second portion of the touch-sensing surface corresponding to a second position within the user interface that is different from the first position. For example, according to some embodiments, a touch-swipe-lift-off sequence can be used to perform a drag and drop or to move a focus region within a camera viewer.

[0281] According to some embodiments, the input corresponding to the first part of the multi-part operation starts an event associated with each application, and the event persists until an input corresponding to the second part of the multi-part operation is received, at which point the input corresponding to the second part of the multi-part operation ends the event associated with each application. For example, according to some embodiments, the first input is the selection of a user interface element that starts an event associated with an application (e.g., provided on the application user interface of the application), and the second input is the selection of a different user interface element that ends the event associated with the application (e.g., on the application user interface of the same application).

[0282] According to some embodiments, the application is a text editing application, and the input corresponding to the first part of the multi-part operation is the selection of a first user interface element that enables the text entry mode of the text editing application. In this example, the multi-part operation includes one or more inputs for entering text into the text editing application, the event corresponds to text operations within the text editing document in response to the input for text entry, and the input corresponding to the second part of the multi-part operation is the selection of a user interface element that disables the text entry mode of the text editing application. According to some embodiments, the text editing application is a communication application, such as an email or messaging application, that enables the device user to communicate with other users. According to some embodiments, the text editing application is a non-communication text editing application, such as a word processor application, a notepad or sticky note application, or other application that enables text entry but does not provide its own communication function.

[0283] According to some embodiments, the characteristics of the continuous haptic output sequence are selected based on an event initiated by an input corresponding to a first part of a multi-part operation. For example, when a call is put on hold, a haptic output sequence of lower intensity (e.g., lower energy, weaker, less perceptible) than a financial transaction event or a secure login is associated. For a financial transaction event or a secure login, the user is providing confidential or security information, so it is a transaction that should draw more attention. In this example, over the duration of the multi-part operation (e.g., after the confidential information has been entered but before the transaction is complete), the user is made aware of the pre-entry, and as a result, completes the transaction promptly so as not to compromise the confidentiality of the information being entered. In some situations, a time-out function is used so as not to leave the confidential information available. Additionally, different haptic output patterns optionally provide the user with information as to which task the pattern is for midway through its completion.

[0284] According to some embodiments, an input corresponding to a first part of a multi-part operation pauses a continuous event associated with an application, and an input corresponding to a second part of the multi-part operation resumes the continuous event associated with the application. For example, depending on the situation, the first part initiates a pause in the continuous event. In some embodiments, the first input corresponds to the selection of a user interface element that pauses an action such as the playback of a multimedia file, a video clip, a song, or an ongoing phone call, and the second input is the selection of a different user interface element that resumes the action.

[0285] According to some embodiments, the intensity of the continuous haptic output sequence increases over time until a subsequent input corresponding to the multi-part operation is received. For example, the haptic energy (e.g., amplitude, frequency, haptic characteristics) of the continuous haptic output sequence is modulated over time (e.g., by increasing the intensity or haptic energy) to alert the user that the urgency to provide further input corresponding to the multi-part operation is increasing. Such increasing output can, according to various embodiments, correspond to a reminder to end an event started by the first part of the multi-part operation or to resume an event paused by the first part of the multi-part operation.

[0286] In some embodiments, the temporally varying haptic energy profile of the haptic output sequence mimics the temporally varying acoustic energy profile of the audio output sequence. As discussed in conjunction with FIG. 6, according to some embodiments, the temporally varying waveform patterns of the haptic sequence and the audio sequence mimic, reflect, align, and / or synchronize with each other.

[0287] According to some embodiments, the output includes an audio output that depends on whether the first part of the multi-part operation starts a new event or pauses an ongoing event. According to some embodiments, if the input corresponding to the first part of the multi-part operation does not pause an ongoing event but instead starts an event associated with each application, device 100 provides a haptic output sequence that does not have an accompanying audio output sequence. If the input corresponding to the first part of the multi-part operation pauses an ongoing event associated with each application, device 100 provides an ongoing audio output sequence that accompanies the ongoing haptic output. In some embodiments, the audio output sequence accompanies the haptic output sequence only when the first part of the multi-part operation pauses or stops an ongoing event or action. During the pause, the user is likely not interacting with the device and / or application. For example, the user can passively wait for the call to be transmitted and, therefore, tolerate additional stimuli associated with the audio output. However, in some embodiments, the audio sequence does not accompany the haptic sequence when an actual event, such as when the user is actively typing an email message or entering sensitive financial or login information, is initiated by the first part of the multi-part operation according to some embodiments.

[0288] According to some embodiments, the application is a voice communication application. For example, according to some embodiments, the application can be a normal telephone application, a voice over internet application such as Skype (registered trademark), or a voice communication application embedded in an email application such as Gchat. According to some embodiments, the event is an ongoing voice communication that occurs when an input corresponding to a first part of a multi-part operation is received, and the input corresponding to the first part of the multi-part operation is a selection of a first user interface element of the voice communication application, and the selection pauses the ongoing voice communication. For example, a voice call is interrupted for call transfer or call hold requests. According to some embodiments, the input corresponding to the second part of the multi-part operation is a selection of a second user interface element that resumes the voice communication.

[0289] It should be understood that the specific order described for the operations in FIG. 11 is merely exemplary and is not intended to indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. Additionally, it should be noted that the details of the other processes described herein with respect to the other methods described herein (e.g., methods 700, 900, 1000, 1200, 1300, 2500, 2600, 3000, and 3200) are also applicable in a manner similar to method 1100 described above in relation to FIG. 11. For example, the inputs, operations, applications, and tactile outputs described above with reference to method 1100 optionally have one or more of the characteristics of the inputs, operations, applications, and tactile outputs described herein with reference to the other methods described herein (e.g., methods 700, 900, 1000, 1200, 1300, 2500, 2600, 3000, and 3200). For the sake of brevity, these details are not repeated here. Tactile Output for Subset Operations

[0290] According to some embodiments, whether a detected requirement to perform an action is related to an action that is a subset of another action serves as a basis for providing a corresponding tactile output. Providing a tactile output that correlates with whether a requirement to perform an action is associated with an action that is a subset of another action creates a more efficient human-machine interface, thereby reducing the time it takes for a user to perform an action, and as a result, reducing energy consumption and increasing the battery life of the battery that powers the device. FIG. 12 is an exemplary method flow diagram of method 1200 for detecting first and second inputs for performing first and second actions, performing the actions, and providing corresponding outputs, according to some embodiments. Note that in some embodiments, different steps than those shown in FIG. 12 may be performed.

[0291] The method begins by detecting at 1205 a first input corresponding to a requirement to perform a first action. According to some embodiments, computing device 100 includes a touch-sensitive display, and the first input is received via the touch-sensitive display according to some embodiments.

[0292] According to some embodiments, in response to detecting the first input, device 100 provides at 1210 a first output that includes a tactile component. When the touch-sensitive display is provided, the tactile component of the first output is provided via the touch-sensitive display according to some embodiments.

[0293] Also, in response to the detection of a first input according to some embodiments, device 100 performs a first operation at 1215. The first operation can be any operation performed by device 100 in response to the input. Examples described later include, according to various embodiments, capturing an image using a camera, a transaction enabling operation to enable the device to authenticate a secure transaction, a saving operation to save content within an existing file, and a sending operation to send a reply to a message in a message inbox, but this enumeration is not intended to be exhaustive. Other operations by device 100 are envisioned as steps of the present method below.

[0294] After performing the first operation, device 100 detects a second input at 1220 in response to a request to perform a second operation that includes the first operation and an additional operation according to some embodiments. According to some embodiments, when computing device 100 includes a touch-sensitive display, the second input is received via the touch-sensitive display.

[0295] In response to the detection of the second input according to some embodiments, device 100 provides a second output at 1225 that includes a tactile component, and the second output includes a first output in conjunction with the provision of an additional output corresponding to the additional operation. When a touch-sensitive display is provided, the tactile component of the second output is provided via the touch-sensitive display according to some embodiments.

[0296] In some embodiments, the tactile component corresponding to the second output includes a first tactile component corresponding to the first output and a second tactile output component corresponding to the additional output, and the second tactile output component is different from the first tactile component. For example, the second tactile component has a greater intensity (e.g., amplitude, duration, frequency) than the first tactile component according to some embodiments.

[0297] According to some embodiments, the additional output includes non-tactile components. According to various embodiments, the non-tactile components can be either audio or visual (e.g., graphic, text). In some embodiments, the additional output includes an audio component, while the first output does not include an audio component. For example, the audio output is provided during a particular interval of time that a camera application measures for counting down to an image capture, while the act of capturing the image itself is not accompanied by an audio sound.

[0298] For additional output that includes an audio output, the tactile output is, in some embodiments, accompanied by a corresponding audio output, and the audio portion and the tactile portion are aligned when at least a portion of the audio output is simultaneous with at least a portion of the tactile output, or the audio output occurs at approximately the same time as the tactile output in some embodiments. The tactile waveform and the audio waveform need not be perfectly aligned, and the device 100 may consider the fact that, for a particular class of tactile output and audio output, the user will perceive them as occurring simultaneously or in sync even if the tactile output and the audio output are slightly out of time with each other (e.g., since the audio output is processed more quickly than the tactile output, providing the tactile output before providing the audio output may, depending on the situation, cause the user to perceive the audio output and the tactile output as occurring simultaneously or in sync). Changes in the accompanying audio portion can also vary between embodiments that include an audio output. For example, each of the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more sound qualities (e.g., pitch, tone color, etc.) associated with the audio component associated with the first output can vary in a particular situation.

[0299] In some embodiments, the same haptic waveform produces different perceptions when different audio waveforms are associated with it. For example, if an audio (incoming) tone is played at a high pitch versus a low pitch, and the same haptic component (or a substantially similar haptic component that an average user would not be able to distinguish between them) is associated, the perception of the haptic component will be different even though the haptic is the same. The higher pitch or brighter version of the incoming tone is thought to be used with the second output to draw the user's attention more. In some embodiments, the first haptic output is similar, identical, or substantially identical to the second haptic output.

[0300] According to some embodiments, the second output includes a text component that specifies that the additional action is different from the first action. For example, the text component may describe supplementary information. For example, an output corresponding to a send action and an archive action provides text information indicating that the message has been archived. As another example, an output corresponding to a payment success message indicates that the security information has been verified.

[0301] According to some embodiments, the additional action has various values and the additional output is based on the value of the variable attribute. For example, the time interval before taking a photo by a camera may be 5 seconds, 10 seconds, 15 seconds, etc. according to various embodiments. According to some embodiments, for a 10 - second camera timer, the additional output correspondingly lasts for 10 seconds.

[0302] According to some embodiments, then, device 100 performs the second action at 1230. The timing of the performance of the first and second actions can each vary. According to some embodiments, the first action is performed simultaneously with the provision of the first output, and the additional action is performed simultaneously with the provision of the additional output. For example, for a shutter self - timer, the first output corresponding to the first action is provided when the timer counts down, and the additional output corresponding to the additional action is provided when the shutter takes a photo.

[0303] The first operation can be performed before the first output provided before the additional operation and the provision of additional output according to some embodiments. For example, for the first operation corresponding to a message sending (e.g., email) function, the second operations are a sending operation and an archiving operation, and the sending operation is performed before the additional operation which is the archiving part.

[0304] According to some embodiments, at least a part of the first operation is performed simultaneously with a part of the additional operation, and at least a part of the first output is provided simultaneously with a part of the additional output.

[0305] The following are some examples useful for illustrating the steps of method 1200. In the first example, the first operation corresponds to the capture of an image using a camera. According to various embodiments, the camera is integrated into device 100 or is remote from the device but remotely controlled by the device. The second operation in this example corresponds to the capture of an image after a specific interval of time. In particular, the method of this example starts by detecting a first input corresponding to a request to capture a first image using the camera, and in response to the detection of the first input, provides a first output including a tactile component according to some embodiments. Also, in response to the detection of the first input, device 100 captures the first image. After capturing the first image, device 100 detects a second input corresponding to a request to capture a second image after a specific interval of time, and in response to the detection of the second input, provides a second output including a tactile component according to some embodiments. In this example, the second output includes the first output related to providing an additional output corresponding to the measurement of the elapse of a specific interval of time, and the second image is captured after a specific interval of time.

[0306] In another example, the first operation corresponds to a transaction enabling operation for enabling the device to authenticate a secure transaction, and the second operation corresponds to enabling the device to authenticate a secure transaction and authenticating a secure transaction according to some embodiments. For example, in an online payment context where credit card information is required, the enabling operation is to prepare the NFC subsystem to provide a payment certificate at the payment terminal or other points of sale, or at the time of a user login request that requires the user's certificate of credit and password. Specifically, according to this example, the method starts by detecting a first input corresponding to a request to perform a transaction enabling operation to enable the device to authenticate a secure transaction, and in response to the detection of the first input, provides a first output including a tactile component. Also, in response to the detection of the first input, the device is enabled to authenticate a secure transaction. After performing the transaction enabling operation, a second input corresponding to a request to authenticate a secure transaction is detected, and in response to the detection of the second input, a second output including a tactile component is provided. The second output includes the first output related to the provision of an additional output corresponding to the authentication of a secure transaction and an additional operation. Also, in response to the detection of the second input, a secure transaction is authenticated.

[0307] According to some embodiments, the first operation corresponds to a save operation for saving the content within an existing file, and the second operation corresponds to a named save operation for saving the content within the existing file to a new file. In this example, a first input corresponding to a request to perform the save operation is detected, and in response to the detection of the first input, a first output including a tactile component is provided. Also, in response to the detection of the first input, the save operation is performed to save the content within the existing file. After the save operation is performed, a second input corresponding to a request to perform a named save operation to save the content within the existing file to a new file is detected, and in response to the detection of the second input, a second output including a tactile component is provided. The second output includes the first output combined with the provision of an additional output corresponding to the creation of a new file according to some embodiments. Also, in response to the detection of the second input, the named save operation is performed.

[0308] According to some embodiments, the first operation corresponds to a send operation for sending a reply to a message in a message inbox, and the second operation corresponds to a send operation and an archive operation for sending a reply to a message in the message inbox and removing the reply from the message inbox.

[0309] It should be understood that the specific order described for the operations in FIG. 12 is merely exemplary and is not intended to indicate that the described order is the only order in which the operations can be performed. One of ordinary skill in the art will recognize various ways to reorder the operations described herein. Additionally, it should be noted that the details of the other processes described herein with respect to the other methods described herein (e.g., methods 700, 900, 1000, 1100, 1300, 2500, 2600, 3000, and 3200) are also applicable in a manner similar to method 1200 described above in connection with FIG. 12. For example, the input, operations, applications, and tactile outputs described above with reference to method 1200 optionally have one or more of the characteristics of the input, operations, applications, and tactile outputs described herein with reference to the other methods described herein (e.g., methods 700, 900, 1000, 1100, 1300, 2500, 2600, 3000, and 3200). For the sake of brevity, these details are not repeated here. Tactile output associated with an alert condition class

[0310] According to some embodiments, whether two detected alert conditions are in the same class or different classes of alert conditions serves as a basis for providing corresponding tactile output. Providing tactile output that correlates with whether the alert conditions are within the same class or different classes of alert conditions creates a more efficient human-machine interface, thereby reducing the time it takes for the user to perform an operation, and as a result, reducing energy consumption and increasing the battery life of the battery that powers the device. FIG. 13 is an exemplary method flowchart of method 1300 for detecting first and second alert conditions and providing corresponding output according to some embodiments. Note that in some embodiments, different steps than those shown in FIG. 13 may be performed.

[0311] According to some embodiments, the method begins by detecting, at 1305, the occurrence of a first alert condition. The first alert condition can be any of the various alert conditions described elsewhere in this specification.

[0312] According to some embodiments, a first output is provided, at 1310, in response to detecting the occurrence of the first alert condition, and includes a first tactile component and a first non-tactile component. According to various embodiments, the non-tactile component can be either audio or visual (e.g., graphic, text). According to some embodiments, when an audio component is included, the audio component and the tactile component are aligned. The tactile waveform and the audio waveform need not be perfectly aligned, and the device 100 may consider the fact that, for a particular class of tactile and audio outputs, the tactile and audio outputs will be perceived by the user as occurring simultaneously or in synchrony even if they are slightly out of time with each other (e.g., because the audio output is processed more quickly than the tactile output, providing the tactile output before the audio output may, depending on the situation, cause the user to perceive the audio and tactile outputs as occurring simultaneously or in synchrony). According to some embodiments, the non-tactile portion includes a text component.

[0313] According to some embodiments, after providing the first output, device 100 detects the occurrence of a second alert condition at 1315 and, in response to detecting the occurrence of the second alert condition, provides a second output at 1320 that includes a second tactile component and a second non-tactile component. According to some embodiments, when the first alert condition and the second alert condition are different alert conditions within the same class of alert conditions, the first output and the second output share one or more same components and have one or more different components. Alternatively, when the first alert condition and the second alert condition are within the same class of alert conditions, the first and second outputs have one or more common components. According to some embodiments, when the first alert condition and the second alert condition are different alert conditions within different classes of alert conditions, the first tactile component is different from the second tactile component and the first non-tactile component is different from the second non-tactile component. Alternatively, when the first alert condition and the second alert condition are within different classes of alert conditions, the first and second outputs do not have common components.

[0314] According to some embodiments, classes of alert conditions include messages, activity alerts, media playback alerts, virtual assistant alerts, system alerts, schedule reminders, and internet browser updates.

[0315] According to some embodiments, when the first alert condition and the second alert condition correspond to separate instances of the same alert condition, the first output is identical to the second output. If computing device 100 has a touch-sensitive display, the first tactile component of the first output and the second tactile component of the second output are provided via the touch-sensitive display on the computing device.

[0316] In some embodiments, the first and second non-tactile components are audio outputs. For non-tactile outputs that include an audio output, the tactile output is accompanied by a corresponding audio output according to some embodiments, and the audio portion and the tactile portion are aligned when at least a portion of the audio output is simultaneous with at least a portion of the tactile output, or the audio output occurs substantially simultaneously with the tactile output according to some embodiments. The tactile waveform and the audio waveform need not be perfectly aligned, and the device 100 may consider the fact that for a particular class of tactile and audio outputs, even if the tactile and audio outputs are slightly out of time, they will be perceived by the user as occurring simultaneously or in sync (e.g., since the audio output is processed more quickly than the tactile output, providing the tactile output before providing the audio output may, depending on the situation, cause the user to perceive the audio output and the tactile output as occurring simultaneously or in sync). Changes in the accompanying audio portion may also vary between embodiments that include an audio output. For example, each of the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more audio qualities (e.g., pitch, timbre, etc.) associated with the audio component associated with the first output may vary in a particular situation.

[0317] In some embodiments, the same tactile waveform produces different perceptions when different audio waveforms are associated with it. For example, if an audio (incoming call) tone is played at a high pitch instead of a low pitch and the same tactile component (or a slightly similar tactile component that the average user would not be able to distinguish between them) is associated with it, the perception of the tactile component will be different even though the tactile is the same. The higher or shriller version of the incoming call tone is thought to be used with the second output to draw the user's attention more. In some embodiments, the first tactile output is similar, identical, or substantially identical to the second tactile output.

[0318] In some embodiments, the first and second non-tactile components are visual outputs such as text, graphics, or metadata output.

[0319] According to some embodiments, the first alert condition and the second alert condition are in the same class of alert conditions, and this class is associated with the alert conditions for the application. In this example, all alerts from each application are classified into a common class, and the class is distinguished by the application. For example, all virtual assistant events optionally have the same tactile component with different audio components. In another example, incoming messages and outgoing messages have the same tactile output with different audio, and the same tactile output is optionally based on being associated with a common instant messaging application. The common component (the tactile component in this example) identifies the application. In yet another example, events for a personal automated assistant including confirmation events, cancellation events, and activation events optionally all have the same audio, but different tactile outputs. [Table 6]

[0320] Table 6 shows message receive alerts and message send alerts for a messaging application according to some embodiments. In this example, various modes are listed for an active application state. In the case of a new message receive, as shown in rows 5, 6, and 7, a tap type haptic is output and an audio of the initial setting for message receive is output. In the case of a new message send, as shown in the last row, a tap type haptic is output and an audio of the initial setting for message send is output. Message receive (the first alert condition) and message send (the second alert condition) are within the same class of alert conditions where the class is associated with the application (messaging), so the haptic output for both alert conditions is the same, but the audio output is different. Table 6 provides conditions and corresponding outputs according to some embodiments, which are merely illustrative. Other combinations of alerts, states, modes, conditions, and outputs are used according to various embodiments.

[0321] According to some embodiments, the first alert condition and the second alert condition are in the same class of alert conditions, and this class is associated with the category of the application. For example, according to some embodiments, the classes of applications include messaging applications, game applications, health / fitness applications, productivity applications, or social networking applications. In some embodiments, applications having common or similar functions or performing the same are optionally members of a common class of applications. Examples of classes of applications include text-based communication applications (instant messaging, email, social media), voice-based communication applications (phone, voice over internet), multimedia applications (photo viewer application, camera application, music player, video player), text editing applications (word (registered trademark), notepad), scheduling applications (calendar, alarm), and internet browser applications (chrome (registered trademark), safari (registered trademark)). Thus, according to some embodiments, the class is distinguished by the function across the applications. As one example, according to some embodiments, email messages and text messages belong to different applications (email and instant messaging), but are associated with applications in the same class of applications (both text-based communication applications), and thus have the same tactile output and different audio outputs.

[0322] According to some embodiments, the first alert condition and the second alert condition are in the same class of alert conditions, and this class corresponds to a function within a defined class of functions. For example, alerts that report on the achievement of activity goals within an activity monitoring application (e.g., approaching the goal, falling below the goal, achieving the goal, achieving 10% of the goal, achieving 50% of the goal, etc.) are optionally classified within the same class of alerts. According to some embodiments, various alarms or reminders within the same scheduling application are set for various defined time instances or are set to report on meeting various different defined criteria.

Table 7

[0323] Table 7 shows various activity-related goals according to some embodiments. In this example, it shows three different alert conditions belonging to the same alert condition class, namely, a fixed end timer, 50% of the activity of the goal, and the activity that has reached the goal. The class of alert conditions shown corresponds to a function within a defined class of functions of alerts that convey information about the achievement of activity goals within an activity monitoring application. For each event in the table, the tactile output is of the tap type, while the audio output varies for different types (e.g., initial activity chime, no audio, strong activity chime). Table 7 provides conditions and corresponding outputs according to some embodiments, but this is merely illustrative. Other combinations of alerts, states, modes, conditions, and outputs are used in various embodiments.

[0324] According to some embodiments, the first alert condition and the second alert condition are associated with two different applications, the first alert condition and the second alert condition are in the same class of alert conditions, and the class corresponds to functions within a defined class of functions in the two different applications. According to some embodiments, examples of defined classes of functions include alerts for incoming text communications (incoming emails, incoming instant messages, incoming social media messages) from another user across applications classified in the same class, alerts for incoming voice communications (phone calls, voice over internet, voice calls within an email application) from another user across all applications classified in a common class, system alerts for application software upgrades for all applications classified in a common class, and system alerts for various states of the device (connected / non - available for Wi - Fi®, GPS on / off, battery life indicator, device on / off).

Table 8

[0325] Table 8 includes various alert conditions for which software upgrades are available for four different application types: calendar, messaging, social networking, and games. These alert conditions are each associated with a different application but are in the same class of alert conditions corresponding to functions within a defined class of functions. In this case, it is an upgrade of the application software for all applications. For each alert condition, the haptic of the buzz tap is the output. Some of the alert conditions also have non-haptic outputs. In the case of a social networking or game application, the haptic output is greeted by a pop-up type notification. Table 8 provides conditions and corresponding outputs according to some embodiments, but this is merely illustrative. Other combinations of alerts, states, modes, conditions, and outputs are used according to various embodiments.

[0326] According to some embodiments, when the first alert condition and the second alert condition are different alert conditions within the same class of alert conditions, the first tactile component of the first output is the same (or substantially the same) as the second tactile component of the second output, and the first non-tactile component (e.g., audio or visual) of the first output is different from the second non-tactile component of the second output. Each of the examples in Tables 4, 5, and 6 above is classified into this category because they share a common tactile component and different non-tactile components. In some embodiments, the "same" tactile component is a tactile component generated based on the same tactile output command delivered to the tactile output generator, even if the tactile output generator generates somewhat different tactile outputs (even if they are perceptually substantially the same to an average user). In some embodiments, audio components can share attributes but are still different. For example, the same incoming call tone or musical score played using different pitches or different instruments. Overlaying different audio on the same tactile output (e.g., the same audio played at a high pitch vs. a low pitch, a metallic sound vs. a glass sound vs. a ceramic sound) produces different perceptions. In some embodiments, different tactile components are tactile components generated based on different tactile output commands delivered to the tactile output generator so as to generate tactile outputs that can be perceptually distinguished by an average user).

[0327] According to some embodiments, the opposite relationship applies to these respective outputs with respect to tactile-non-tactile similarity and difference. When the first alert condition and the second alert condition are different alert conditions within the same class of alert conditions, the first non-tactile component of the first output is the same (or substantially the same) as the second non-tactile component of the second output, and the first tactile component of the first output is different from the second tactile component of the second output.

[0328] According to some embodiments, when the first alert condition and the second alert condition correspond to the same alert condition, the first output is accompanied by a first additional component that is different from the second additional component associated with the second output. For example, the first and second incoming instant messages are both incoming instant messages and thus have the same alert condition. In this example, the first additional component is specific to the first event that triggered the first alert condition, and the second additional component is specific to the second event that triggered the second alert condition. For example, according to some embodiments, the additional component of the first incoming instant message includes visual components (text, metadata, graphics) specific to the first instant message (e.g., the name of the sender, the contact information of the sender, the body / content of the message). Similarly, the additional component of the second incoming instant message includes visual components (text, metadata, graphics) specific to the second instant message (e.g., the name of the sender, the contact information of the sender, the body / content of the message). According to some embodiments, the first additional component associated with the first output optionally provides information (e.g., text information) describing the first event that triggered the first alert (e.g., a fragment of the message or a text version of the auto-assisted response), and the second additional component associated with the second output provides information (e.g., text information) describing the second event that triggered the second alert.

[0329] In accordance with some embodiments, after providing the second output at 1320, device 100 detects the occurrence of a third alert condition at 1325 and, in response to detecting the occurrence of the third alert condition, provides a third output at 1330 that includes a third haptic component and a third non-haptic component. When the first alert condition, the second alert condition, and the third alert condition are different alert conditions within the same class of alert conditions, the first output, the second output, and the third output optionally share one or more same components and have one or more different components. When the third alert condition is within a different class of alert conditions than the first alert condition and the second alert condition, the third haptic component is optionally different from both the first haptic component and the second haptic component, and the third non-haptic component is optionally different from both the first non-haptic component and the second non-haptic component. Alternatively, in accordance with some embodiments, when the first alert condition, the second alert condition, and the third alert condition are different alert conditions within the same class of alert conditions, the first output, the second output, and the third output have one or more common components (e.g., haptic, non-haptic), and when the third alert condition is within a different class of alert conditions than the first alert condition and the second alert condition, the third haptic output has no components in common with the first output and the second output.

[0330] It should be understood that the specific order described for the operations in FIG. 13 is merely exemplary and is not intended to indicate that the described order is the only order in which the operations can be performed. One of ordinary skill in the art will recognize various ways to reorder the operations described herein. Additionally, it should be noted that the details of the other processes described herein with respect to other methods (e.g., methods 700, 900, 1000, 1100, 1200, 2500, 2600, 3000, and 3200) described herein are also applicable in a manner similar to method 1300 described above in connection with FIG. 13. For example, the input, alert condition, application, and tactile output described above with reference to method 1300 optionally have one or more of the characteristics of the input, alert condition, application, and tactile output described herein with reference to the other methods (e.g., methods 700, 900, 1000, 1100, 1200, 2500, 2600, 3000, and 3200) described herein. For the sake of brevity, these details are not repeated here.

[0331] The operations described above with reference to FIGS. 7, 9-13 are optionally implemented by the components shown in FIGS. 1A, 1B, and 14-19. For example, the reception of an input, the detection of an alert condition, the determination of a state, the determination of an alert condition trigger, the provision of a tactile output and an audio output, and the implementation of the operations are optionally implemented by an event sorter 170, an event recognizer 180, and an event handler 190. An event monitor 171 within the event sorter 170 detects a contact on the touch-sensitive display 112, and an event dispatcher module 174 distributes event information to the application 136-1. Each event recognizer 180 of the application 136-1 compares the event information with a respective event definition 186 and determines whether a first contact at a first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When each predefined event or sub-event is detected, the event recognizer 180 activates an event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or invokes data update 176 or object update 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses respective GUI update 178 to update what is displayed by the application. Similarly, it will be apparent to those skilled in the art how other processes are implemented based on the components shown in FIGS. 1A, 1B, and 14-19.

[0332] As described above, the operations described with reference to FIGS. 7 and 9 to 13 are optionally implemented by the components shown in FIGS. 14 to 19. FIG. 14 shows an exemplary functional block diagram of a device 100 configured according to the principles of the various embodiments to be described. It will be understood by those skilled in the art that the functional blocks described in FIG. 14 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments to be described. Thus, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0333] As shown in FIG. 14, the device 100 includes a display unit 1401 configured to display an application, a touch sensing surface unit 1403 configured to receive user contact, and a processing unit 1405 coupled to the display unit 1401 and the touch sensing surface unit 1403. In some embodiments, the processing unit 1405 includes a detection unit 1410, a state determination unit 1415, a tactile providing unit 1420, a participation determination unit 1425, and an output determination unit 1430.

[0334] The processing unit 1405 is configured to detect alert conditions associated with an application running on a computing device (e.g., using the detection unit 1410). The processing unit 1405 is also configured to determine the state associated with the application at a time point associated with the detected alert condition (e.g., using the state determination unit 1415). In accordance with the determination that the application was active at the time point associated with the alert condition, the processing unit 1405 provides a first haptic output representing the occurrence of the alert condition (e.g., using the haptic providing unit 1420), the first haptic output having a first set of output characteristics, and in accordance with the determination that the application was inactive at the time point associated with the alert condition, provides a second haptic output representing the occurrence of the alert condition (e.g., using the haptic providing unit 1420), the second haptic output having a second set of output characteristics, and the second haptic output is different from the first haptic output.

[0335] The following paragraphs

[0335] to

[0347] describe different embodiments that may be implemented individually or in any combination by the device 100 illustrated in FIG. 14.

[0336] The processing unit 1405 determines the state associated with the application at a time point associated with the alert condition by determining whether a user interface for the application was displayed on the touch screen of the device at the time point associated with the alert condition (e.g., using the state determination unit 1415).

[0337] The processing unit 1405 determines the state associated with the application at a time point associated with the alert condition by determining whether user input for the application triggered the alert condition by means of a user interaction different from the alert condition (e.g., using the state determination unit 1415).

[0338] When the application is in an active state based on user interaction detected at the time of the alert condition, the processing unit 1405 determines the level of user engagement associated with the user interaction at the time of the alert condition (e.g., using the engagement determination unit 1425), and determines one or more of the output characteristics of the first set of haptic outputs based on the determined level of engagement (e.g., using the output determination unit).

[0339] The processing unit 1405 that determines that the application is in an active state includes determining that the application was running in the foreground on the computing device at the time associated with the alert condition. The processing unit 1405 that determines that the application is in an inactive state includes determining that the application was not running or was running in the background on the computing device at the time associated with the alert condition.

[0340] The display unit 1401 displays each user interface window corresponding to the application simultaneously with one or more other user interface windows on the user interface of the multi-application window on the device. The processing unit 1405 determines (e.g., using the state determination unit 1415) whether each user interface window corresponding to the application was displayed in the foreground of the user interface of the multi-application window at the time associated with the alert condition.

[0341] The display unit 1401 displays one or more other user interface windows that are simultaneously on the user interface of the multi-application window on the device at the time associated with the alert condition, and the processing unit 1405 determines the state associated with the application (e.g., using the determination unit 1415) at the time associated with the alert condition, including determining whether each user interface window corresponding to the application is displayed on the user interface of the multi-application window at the time associated with the alert condition.

[0342] In some embodiments, the alert condition corresponds to an event automatically triggered by an application, a notification of an automatically initiated event received by the application from a source external to the device, or a notification of a manually initiated event received by the application from a human user other than the user operating the device.

[0343] The processing unit 1405 provides an output including a first haptic output associated with a first audio output and a second haptic output associated with a second audio output (e.g., using the output determination unit 1430), the first haptic output being the same as the second haptic output, and the first audio output being different from the second audio output. Alternatively, the processing unit 1405 provides an output including a first haptic output associated with an audio output and a second haptic output associated with the same audio output (e.g., using the output determination unit 1430), the first haptic output being different from the second haptic output.

[0344] Processing unit 1405 is associated with a second haptic output output by an audio output, and the waveform of the second haptic output is generated based on, reproduced from, and synchronized with the waveform of the accompanying audio output. Processing unit 1405 provides an output that includes a first haptic output not accompanied by an audio output and a second haptic output accompanied by an audio output (e.g., using output determination unit 1430). In some embodiments, the first haptic output is similar to the second haptic output. In some embodiments, the second haptic output has a greater intensity than the first haptic output.

[0345] (Provided, for example, by output determination unit 1430) The characteristics of the first set of haptic outputs include one or more of amplitude, duration, regularity, repetition frequency, or selection of haptic features of the first haptic output. In some embodiments, the second haptic output is different from and has a greater intensity than the first haptic output.

[0346] Processing unit 1405 determines, by determining that the state of the computing device at the time associated with the alert condition (e.g., using a state determination unit) was active at the time the computing device was associated with the alert condition, and the determination that the application was in an inactive state includes the determination that the computing device was inactive at the time it was associated with the alert condition. According to some embodiments, the characteristics of the first and second sets correspond to the device type of the computing device.

[0347] In some embodiments, at a first time point, while the application is in an active state, the processing unit 1405 detects a first alert condition associated with the application (e.g., using the detection unit 1410), and in response to detecting the first alert condition while the first application is in an active state, the processing unit 1404 provides a first output representing the first alert condition (e.g., using the output determination unit 1430). At a second time point, while the first application is in an inactive state, the processing unit 1405 detects a second alert condition, and in response to detecting the second alert condition while the application is in an inactive state, the processing unit 1405 provides a second tactile output representing the second alert condition (e.g., using the output determination unit 1440). The second tactile output is different from the first tactile output.

[0348] In some embodiments, the device has a touch-sensitive display (e.g., provided by the touch-sensitive surface unit 1403), and the first tactile output and the second tactile output are provided via the touch-sensitive display on the computing device (e.g., using the output determination unit 1430). As described above, the operations described with reference to FIGS. 7 and 9-13 are optionally implemented by the components shown in FIGS. 14-19. FIG. 14 shows an exemplary functional block diagram of a device 100 configured according to the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 14 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0349] As described above, the operations described with reference to FIGS. 7 and 9 to 13 are optionally implemented by the components shown in FIGS. 14 to 19. FIG. 15 shows an exemplary functional block diagram of a device 100 configured in accordance with the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 15 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Accordingly, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0350] As shown in FIG. 15, the device 100 includes a display unit 1501 configured to display an application, a touch sensing surface unit 1503 configured to receive user contact, and a processing unit 1505 coupled to the display unit 1501 and the touch sensing surface unit 1503. In some embodiments, the processing unit 1505 includes a detection unit 1510, a trigger determination unit 1515, a tactile providing unit 1520, an external event determination unit 1525, an output deformation providing unit 1530, an urgency determination unit 1535, a context determination unit 1540, a modulation unit 1545, and a personal communication determination unit 1550.

[0351] The processing unit 1505 is configured to detect the occurrence of an alert condition (e.g., using the detection unit 1510) associated with an application, and in response to detecting the occurrence of the alert condition, determine (e.g., using the trigger determination unit 1515) whether the alert condition was triggered by an event manually initiated. In accordance with the determination that the alert condition was triggered by an event manually initiated, the processing unit 1505 provides a first tactile output corresponding to the manually initiated event notification (e.g., using the tactile providing unit 1520), and in accordance with the determination that the alert condition was triggered by an event automatically initiated, the processing unit 1505 provides a second tactile output corresponding to the automatically initiated event notification (e.g., using the tactile providing unit 1520), and the second tactile output is different from the first tactile output.

[0352] The following paragraphs

[0353] ~

[0364] describe various embodiments that may be implemented individually or in any combination by the device 100 illustrated in FIG. 15.

[0353] According to some embodiments, the processing unit 1505 determines whether the alert condition was triggered by an event manually initiated by determining (e.g., using the trigger determination unit 1515) whether the alert condition corresponds to an event initiated by a human user. In some embodiments, the processing unit 1505 determines whether the alert condition was triggered by an event manually initiated by determining (e.g., using the trigger determination unit 1515) whether the alert condition corresponds to an event initiated by a human user other than the user of the computing device.

[0354] According to some embodiments, the processing unit 1505 determines that the alert condition has been triggered by an event manually initiated, (e.g., using the trigger determination unit 1515,) by determining whether the alert condition corresponds to an input received from a human user. The input identifies the user of the computing device, and the input includes an instruction from the human user to send an alert to the user of the computing device regarding the input.

[0355] According to some embodiments, the processing unit 1505 determines, (e.g., using the trigger determination unit 1515,) whether the alert condition has been automatically triggered by an application, or is a notification of an automatically initiated event received by the application from a source external to the device. Alternatively, the processing unit 1505 determines that the alert condition has been triggered by an automatically initiated event, (e.g., using the trigger determination unit 1515,) by determining that the alert condition occurs at a predetermined time or satisfies a predetermined trigger condition.

[0356] According to some embodiments, in accordance with the determination that the alert condition has been triggered by an automatically initiated event, the processing unit 1505 determines, (e.g., using the external event determination unit 1525,) whether the automatically initiated event corresponds to an automatically initiated event occurring outside the device. In accordance with the determination that the automatically initiated event corresponds to an event occurring outside the device, the processing unit 1505 provides, (e.g., using the output transformation providing unit 1530,) a first transformation of a second haptic output corresponding to the notification of the automatically initiated event occurring outside. In accordance with the determination that the automatically initiated event corresponds to an event initiated within the device, the processing unit 1505 provides, (e.g., using the output transformation providing unit 1530,) a second example of a transformation of a second haptic output generated internally and corresponding to the notification of the automatically initiated event.

[0357] According to some embodiments, in accordance with a determination that an alert condition has been triggered by an event that was manually initiated, processing unit 1505 determines the degree of urgency associated with the manually initiated event (e.g., using urgency determination unit 1535), and modulates the intensity of a first haptic output based on the degree of urgency (e.g., using modulation unit 1545). According to some embodiments, in accordance with a determination that an alert condition has been triggered by an event that was manually initiated, processing unit 1505, according to some embodiments, determines one or more context attributes associated with the manually initiated event (e.g., using context determination unit 1540), and modulates the intensity of the first haptic output based on the one or more context attributes (e.g., using modulation unit 1545).

[0358] According to some embodiments, in accordance with a determination that an alert condition has been triggered by an event that was automatically initiated, processing unit 1505 analyzes the characteristics associated with the alert condition, determines one or more context attributes associated with the automatically initiated event (e.g., using context determination unit 1540), and modulates the intensity of a second haptic output based on the one or more context attributes (e.g., using modulation unit 1545).

[0359] According to some embodiments, in accordance with a determination that an alert condition has been triggered by an event that was automatically initiated, processing unit 1505 determines the degree of time-varying urgency associated with the automatically initiated event (e.g., using urgency determination unit 1535), and modulates the intensity of the second haptic output over a specific time window to indicate the degree of time-varying urgency (e.g., using modulation unit 154).

[0360] In some embodiments, a first haptic output is accompanied by a first audio output, a second haptic output is accompanied by a second audio output, the first haptic output is similar to the second haptic output, and the first audio output is different from the second audio output. In some embodiments, a first haptic output is accompanied by a first audio output, a second haptic output is accompanied by a second audio output, the first audio output is similar to the second audio output, and the first haptic output is different from the second haptic output. In some embodiments, a first haptic output is accompanied by a first audio output, a second haptic output is not accompanied by an audio output. The first haptic output is optionally similar to the second haptic output.

[0361] According to some embodiments, a first haptic output corresponding to a notification of a manually initiated event is stronger than a second haptic output corresponding to a notification of an automatically initiated event. Depending on the situation, the first haptic output is characterized by a first set of properties including one or more of the amplitude of the first haptic output, the duration of the first haptic output, the regularity associated with the first haptic output, the repetition frequency of the haptic features in the first haptic output, the selection of the haptic features constituting the first haptic output, and the second haptic output is characterized by a second set of properties including one or more of the amplitude of the second haptic output, the duration of the second haptic output, the regularity associated with the second haptic output, the repetition frequency of the haptic features in the second haptic output, the selection of the haptic features constituting the second haptic output.

[0362] According to some embodiments, the processing unit 1505 detects a first alert condition corresponding to an incoming email message from a first human sender (e.g., using the detection unit 1510). In response to a determination that the incoming email message corresponds to an event that was manually initiated, the processing unit 1505 provides a first tactile output (e.g., using the tactile providing unit 1520). The processing unit 1505 detects a second alert condition corresponding to an incoming text message from a second human sender (e.g., using the detection unit 1510), and in response to a determination that the incoming text message corresponds to an event that was manually initiated, provides a first tactile output (e.g., using the tactile providing unit 1520).

[0363] In some embodiments, the device has a touch-sensitive display (e.g., provided by the touch-sensitive surface unit 1503), and the first tactile output and the second tactile output are provided via the touch-sensitive display on the computing device (e.g., using the tactile providing unit 1520).

[0364] According to some embodiments, the processing unit 1505 detects the occurrence of an alert condition associated with an application (e.g., using the detection unit 1510), and in response to detecting the occurrence of the alert condition, determines (e.g., using the personal communication determination unit 1550) whether the alert condition corresponds to a personal communication from a sender within a contact list associated with the user. In accordance with a determination that the alert condition corresponds to a personal communication from a sender within the contact list associated with the user, the processing unit 1505 provides a first tactile output corresponding to a personal alert notification (e.g., using the tactile providing unit 1520), and in accordance with a determination that the alert condition does not correspond to a personal communication from a sender within the contact list associated with the user, the processing unit 1505 provides a second tactile output corresponding to an automatic alert notification (e.g., using the tactile providing unit 1520). Here, the first tactile output is larger in scale than the second tactile output.

[0365] As described above, the operations described with reference to FIGS. 7 and 9 to 13 are optionally implemented by the components shown in FIGS. 14 to 19. FIG. 16 shows an exemplary functional block diagram of a device 100 configured according to the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 16 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Accordingly, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0366] As shown in FIG. 16, the device 100 includes a display unit 1601 configured to display an application, a touch sensing surface unit 1603 configured to receive user contact, and a processing unit 1605 coupled to the display unit 1601 and the touch sensing surface unit 1603. In some embodiments, the processing unit 1605 includes a detection unit 1610, an output providing unit 1615, a modulation condition determination unit 1620, an output correction unit 1625, an engagement measurement unit 1630, a haptic input determination unit 1635, an output termination unit 1640, a surrounding condition determination unit 1645, an output delay unit 1650, and a priming providing unit 1655.

[0367] The processing unit 1605 is configured to detect a first alert condition on a computing device associated with the reception of user input for an application (e.g., using the detection unit 1610), and in response to the detection of the first alert condition, provide a first haptic output having a first intensity corresponding to the user input for the application (e.g., using the output providing unit 1615). After providing the first haptic output, the processing unit 1605 detects a second alert condition associated with the reception of a predetermined system event in the application (e.g., using the detection unit 1610), and in response to the detection of the second alert condition, provides a second haptic output having a second intensity corresponding to the predetermined system event (e.g., using the output providing unit 1615). The second intensity is greater than the first intensity.

[0368] The following paragraphs

[0036] to

[0386] describe various embodiments that can be implemented by the device 100 illustrated in FIG. 16, individually or in any combination.

[0369] In some embodiments, the first haptic output is feedback provided directly in response to the user input and as a result of the user input. According to some embodiments, the second alert condition corresponds to an event automatically triggered by the application, a notification of an event automatically triggered and received by the application, or a notification of an event manually initiated and received by the application.

[0370] According to some embodiments, the processing unit 1605 determines whether there is an input-based modulation condition (e.g., using the modulation condition determination unit 1620), and in response to the determination that there is an input-based modulation condition, before providing the first haptic output, the processing unit 1605 modifies the first haptic output (e.g., using the output modification unit 1625).

[0371] In some embodiments, after detecting each alert condition, the processing unit 1605 determines that each alert condition originated from a respective user input received on the computing device, and identifies the degree of user engagement associated with the detected user input (e.g., using the engagement measurement unit 1630). The processing unit 1605 modulates the first signal strength of the first haptic output based on the degree of user engagement (e.g., using the output modification unit 1625).

[0372] In some embodiments, after detecting each alert condition, the processing unit 1605 determines that each alert condition originated from the reception of a tactile user input on the device (e.g., using the tactile input determination unit 1635), and identifies the location on the device where the tactile user input was received. The processing unit 1605 modulates the spatial haptic energy profile of the first haptic output by providing more than a particular ratio of the haptic energy of the first haptic output within a particular radius threshold of the identified location where the tactile user input on the device was received (e.g., using the output modification unit 1625).

[0373] In some embodiments, after detecting each alert condition, the processing unit 1605 determines that each alert condition originated from the reception of a tactile user input on the device (e.g., using the tactile input determination unit 1635), and modulates the spatial haptic energy profile of the first haptic output by decreasing the amplitude of the haptic energy of the first haptic output such that the first haptic output is perceivable at the location on the device where the tactile user input was received (e.g., using the output modification unit 1625). The location on the device where the tactile user input is received may change over time, and the spatial haptic energy may change over time corresponding to the temporally changing location where the tactile user input was received.

[0374] In some embodiments, after detecting each alert condition, the processing unit 1605 determines that each alert condition has resulted from the reception of a tactile user input (e.g., using the tactile input determination unit 1635), and identifies the temporally varying morphological attributes of the tactile user input. The processing unit 1605 gradually modifies the morphological characteristics of the first tactile output to mimic the temporally varying morphological attributes of the tactile user input (e.g., using the output modification unit 1625). According to some embodiments, the temporally varying morphological attributes include a time-dependent contact pressure profile between the tactile user input and the touch-sensitive surface of the computing device, and modifying includes modulating over time the energy profile of the first tactile output to reproduce the time-dependent contact pressure profile between the tactile user input and the touch-sensitive surface of the computing device (e.g., using the output modification unit 1625).

[0375] In some embodiments, after detecting each alert condition, the processing unit 1605 determines that each alert condition has resulted from a continuous user input (e.g., using the tactile input determination unit 1635), and in response to this determination, starts the first tactile output at the start time of the continuous user input (e.g., using the output providing unit 1615). The processing unit 1605 detects the end of the continuous user input and ends the first tactile output at the end time of the continuous user input (e.g., using the output ending unit 1640).

[0376] The processing unit 1605 determines whether there are modulation conditions based on ambient conditions (e.g., using the ambient condition determination unit 1645), and in response to the determination that there are modulation conditions based on ambient conditions, modifies the second tactile output (e.g., using the output modification unit 1625) before providing the second tactile output.

[0377] In some embodiments, after detecting each alert condition associated with each application, processing unit 1605 determines that each alert condition did not result from the reception of user input, and in response to the determination that each alert condition did not result from the reception of user input, processing unit 1605 determines whether one or more ambient conditions (e.g., using ambient condition determination unit 1645) were present in the device at the time of occurrence of each alert condition as a potential interference with the haptic output. In accordance with the determination that no interfering ambient conditions are present in the device, processing unit 1605 provides each second haptic output based on the user's receptivity state (e.g., using output providing unit 1615). In accordance with the determination that one or more ambient conditions are present in the device as a potential interference with the haptic output, processing unit 1605 delays the provision of each second output to the user (e.g., using output delay unit 1650).

[0378] In some embodiments, after detecting each alert condition associated with each application, processing unit 1605 determines that each alert condition did not result from the reception of user input, and in response to the determination that each alert condition did not result from the reception of user input, processing unit 1605 determines whether one or more ambient conditions (e.g., using ambient condition determination unit 1645) were present in the device at the time of occurrence of each alert condition as a potential interference with the haptic output. In accordance with the determination that no interfering ambient conditions are present for the device, processing unit 1605 provides (e.g., using output modification unit 1625) a first variation of a second haptic output to the user, the first variation of the second haptic output having a first set of output characteristics. In accordance with the determination that one or more ambient conditions are present for the device as a potential interference with the haptic output, processing unit 1605 provides (e.g., using output modification unit 1625) a second variation of a second haptic output to the user, the second variation of the second haptic output having a second set of output characteristics, the second set of characteristics being different from the first set of characteristics, and the second variation of the second haptic output having a greater intensity than the first variation of the second haptic output.

[0379] In some embodiments, after detecting each alert condition associated with each application, processing unit 1605 determines that each alert condition did not result from the receipt of user input, and in response to the determination that each alert condition did not result from the receipt of user input, processing unit 1605 provides a priming haptic output as a precursor to a second haptic output (e.g., using priming provision unit 1655), the priming haptic output increasing the level of engagement of the user operating the device with respect to the second haptic output, and at least a portion of the priming haptic output being emitted louder than the second haptic output. Additionally, processing unit 1605 provides a second haptic output following the provision of the priming haptic output (e.g., using output provision unit 1615) and within a particular time interval of the provision of the priming haptic output. In some embodiments, the priming haptic output includes a priming haptic component with a temporally varying waveform pattern characterized by a gradually increasing intensity of the haptic component over time. In some embodiments, the priming haptic output includes a priming haptic component with a temporally varying waveform pattern characterized by a gradually decreasing intensity of the haptic component after an emphasized haptic feature.

[0380] In some embodiments, there is a priming haptic output prior to the second haptic output, and the first haptic output is provided directly in response to the corresponding alert condition resulting from the received user input without a corresponding preceding priming haptic output. In some embodiments, the characteristics of the priming haptic output are selected based on the urgency or context associated with the alert condition.

[0381] In some embodiments, the processing unit 1605 detects a first alert condition corresponding to a user's selection of a user interface element displayed on an application user interface associated with a first application (e.g., using the detection unit 1610). In response to the first alert condition, the processing unit 1605 provides a respective first haptic output representing the user's selection of the user interface element (e.g., using the output providing unit 1615), and each first haptic output includes a first haptic component including first haptic features of a first intensity and a first duration. After providing each first haptic output, the processing unit 1605 detects a second alert condition corresponding to an alert notification received by the first application (e.g., using the detection unit 1610), and in response to the second alert condition, the processing unit 1605 provides a respective second haptic output representing the receipt of the alert notification (e.g., using the output providing unit 1615), and each second haptic output includes a second haptic component including second haptic features of a second intensity and a second duration, and each second haptic output is more pronounced than each first haptic output, different from each first haptic output, based on the second haptic features being more pronounced than the first haptic features, the second intensity being greater than the first intensity, or the second duration being longer than the first duration.

[0382] In some embodiments, the processing unit 1605 detects a first alert condition associated with a first application (e.g., using the detection unit 1610), i.e., a first alert condition resulting from each user input, and provides each first haptic output representing the first alert condition (e.g., using the output providing unit 1615). After providing each first haptic output, the processing unit 1605 detects a second alert condition associated with the first application (e.g., using the detection unit 1610), i.e., a second alert condition not resulting from each user input, and provides each second haptic output representing the second alert condition (e.g., using the output providing unit 1615), and each second haptic output has a greater intensity than each first output and is different from each first output.

[0383] In some embodiments, the first haptic output has a first set of characteristics including one or more of the amplitude of the first haptic output, the duration of the first haptic output, the regularity associated with the first haptic output, the repetition frequency of the tactile features in the first haptic output, the selection of the tactile features constituting the first haptic output, and the second haptic output has a second set of characteristics including one or more of the amplitude of the second haptic output, the duration of the tactile components associated with the second haptic output, the regularity associated with the second haptic output, the repetition frequency of the tactile features in the second haptic output, the selection of the tactile features constituting the second haptic output.

[0384] In some embodiments, the first haptic output is accompanied by a first audio output, the second haptic output is accompanied by a second audio output, the first haptic output is similar to the second haptic output, and the first audio output is different from the second audio output. In some embodiments, the first haptic output is accompanied by a first audio output, the second haptic output is accompanied by a second audio output, the first audio output is similar to the second audio output, and the first haptic output is different from the second haptic output.

[0385] In some embodiments, the first intensity of the first haptic output and the second intensity of the second haptic output are further adjusted in inverse proportion to the volume setting on the device. According to some embodiments, the second haptic output is accompanied by an audio output, and the first haptic output is not accompanied by an audio output.

[0386] In some embodiments, the computing device includes a touch-sensitive display for receiving user input (e.g., by the touch-sensitive surface unit 1603), and the first haptic output and the second haptic output are provided via the touch-sensitive display on the computing device (e.g., using the output providing unit 1615).

[0387] As described above, the operations described with reference to FIGS. 7 and 9-13 are optionally implemented by the components shown in FIGS. 14-19. FIG. 17 shows an exemplary functional block diagram of a device 100 configured in accordance with the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 17 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Accordingly, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0388] As shown in FIG. 17, the device 100 includes a display unit 1701 configured to display an application, a touch-sensitive surface unit 1703 configured to receive user contact, and a processing unit 1705 coupled to the display unit 1701 and the touch-sensitive surface unit 1703. In some embodiments, the processing unit 1705 includes an input receiving unit 1710, a continuous output start unit 1715, a continuous output end unit 1720, a feedback providing unit 1725, and an additional input receiving unit 1730.

[0389] The processing unit 1705 is configured to receive an input corresponding to a first part of a multi-part operation performed by an application running on a computing device (e.g., using the input receiving unit 1710), and in response to receiving the input corresponding to the first part of the multi-part operation, start a continuous haptic output sequence (e.g., using the continuous output start unit 1715). After starting the continuous haptic output sequence, the processing unit 1705 receives an input corresponding to a second part of the multi-part operation (e.g., using the input receiving unit 1710), and in response to receiving the input corresponding to the second part of the multi-part operation, ends the continuous haptic output sequence (e.g., using the continuous end unit 1720).

[0390] The following paragraphs

[0391] ~

[0406] describe various embodiments that may be implemented individually or in any combination by the device 100 illustrated in FIG. 17.

[0391] In some embodiments, the detection of the input corresponding to the first part of the multi-part operation is detected on the touch sensing surface of the device (e.g., using the touch sensing surface unit 1703), and the input corresponding to the second part of the multi-part operation is detected on the touch sensing surface of the device (e.g., using the input receiving unit 1710). According to some embodiments, the haptic output sequence is provided via the touch sensing surface (e.g., using the continuous output start unit 1715).

[0392] In some embodiments, the input corresponding to the first part of the multi-part operation is detected by the device (e.g., using the input receiving unit 1710), and the input corresponding to the second part of the multi-part operation is detected by a second device. In some embodiments, the input corresponding to the first part of the multi-part operation and the input corresponding to the second part of the multi-part operation are detected by the device (e.g., using the input receiving unit 1710).

[0393] According to some embodiments, the multi-part operation is a secure transaction, and the first part of the multi-part operation includes protecting the device to authenticate the secure transaction, and the second part of the multi-part operation includes authenticating the secure transaction. According to some embodiments, the multi-part operation is a secure transaction, and the first part of the multi-part operation includes determining that user authentication is required to perform the secure transaction, and the second part of the multi-part operation includes receiving user authentication for the secure transaction.

[0394] In some embodiments, the first part of the multi-part operation includes closing a draft of a document, and the second part of the multi-part operation includes returning to the draft of the document. In some embodiments, the input corresponding to the first part of the multi-part operation is a first user input that interacts with an application running on the device, and the input corresponding to the second part of the multi-part operation is a second user input that interacts with the application, and the second user input is different from the first user input.

[0395] According to some embodiments, detecting an input corresponding to the first part of the multi-part operation includes detecting a contact on a touch-sensitive surface (e.g., using input receiving unit 1710), and detecting an input corresponding to the second part of the multi-part operation includes detecting a movement of the contact on the touch-sensitive surface (e.g., using input receiving unit 1710). In some embodiments, detecting an input corresponding to the first part of the multi-part operation includes detecting a movement of the contact on the touch-sensitive surface (e.g., using input receiving unit 1710), and detecting an input corresponding to the second part of the multi-part operation includes detecting a lift-off of the contact on the touch-sensitive surface (e.g., using input receiving unit 1710).

[0396] In some embodiments, the inputs corresponding to the first and second parts of the multi-part operation include a single gesture, the input corresponding to the first part of the multi-part operation is the initial part of the gesture, and the input corresponding to the second part of the multi-part operation is the subsequent part of the gesture. According to some embodiments, the gesture is initiated on a portion of the touch-sensitive surface corresponding to a first position within a user interface displayed on the device's display (e.g., using the touch-sensitive surface unit 1703) and ends on a second portion of the touch-sensitive surface corresponding to a second position within the user interface that is different from the first position.

[0397] In some embodiments, the input corresponding to the first part of the multi-part operation initiates an event associated with each application, the event is sustained until an input corresponding to the second part of the multi-part operation is received, and the input corresponding to the second part of the multi-part operation ends the event associated with each application (e.g., using the continuous output termination unit 1720).

[0398] According to some embodiments, the application is a text editing application, and the input corresponding to the first part of the multi-part operation is the selection of a first user interface element that enables the text entry mode of the text editing application. According to some embodiments, the multi-part operation includes one or more inputs for text entry into the text editing application, and the event corresponds to a text operation within the text editing document in response to the input for text entry. The input corresponding to the second part of the multi-part operation is the selection of a user interface element that disables the text entry mode of the text editing application.

[0399] In some embodiments, the input corresponding to the first part of the multi-part operation is the selection of a user interface element that initiates a transaction, and the multi-part operation includes one or more inputs for entry of information necessary to process the transaction. The event in some embodiments corresponds to the number of one or more data entry fields that receive information necessary to process the transaction, and the input corresponding to the second part of the multi-part operation is the selection of a user interface element that authenticates the execution of the transaction.

[0400] In some embodiments, the input corresponding to the second part of the multi-part operation occurs after the input corresponding to the first part of the multi-part operation and is eventually separated from the input corresponding to the first part of the multi-part operation by a specific interval of time corresponding to the duration of the event, and the continuous tactile output sequence is provided throughout the duration of the event and indicates the occurrence of the event. The characteristics of the continuous tactile output sequence are optionally selected based on the event initiated by the input corresponding to the first part of the multi-part operation.

[0401] In some embodiments, the processing unit 1705 provides a first tactile feedback in response to receiving an input corresponding to the first part of the multi-part operation (e.g., using the feedback providing unit 1725), the first tactile feedback is selected based on the event and indicates the start of the event, and the first tactile feedback is different from the continuous tactile output sequence. The processing unit 1705 further provides a second tactile feedback in response to receiving an input corresponding to the second part of the multi-part operation (e.g., using the feedback providing unit 1725), and the second tactile feedback is different from the continuous tactile output sequence.

[0402] According to some embodiments, an input corresponding to a first part of a multi-part operation pauses a continuous event associated with an application, and an input corresponding to a second part of the multi-part operation resumes the continuous event associated with the application.

[0403] In some embodiments, the application is a voice communication application, and the event is an ongoing voice communication occurring at the time of receipt of an input corresponding to the first part of the multi-part operation. In some embodiments, the input corresponding to the first part of the multi-part operation is a selection of a first user interface element of the voice communication application, and this selection pauses the ongoing voice communication, and the input corresponding to the second part of the multi-part operation is a selection of a second user interface element that resumes the voice communication.

[0404] In some embodiments, after the start of a continuous tactile output sequence in response to the first part of the multi-part operation and before receipt of an input corresponding to the second part of the multi-part operation, the processing unit 1705 receives one or more additional user inputs different from the input corresponding to the second part of the multi-part operation (e.g., using the additional input receiving unit 1730). The processing unit 1705 continues to provide the continuous tactile output regardless of the additional user inputs until receipt of the input corresponding to the second part of the multi-part operation (e.g., using the continuous output start unit 1715). In some embodiments, the intensity of the continuous tactile output sequence increases over time until a subsequent input corresponding to the multi-part operation is received.

[0405] In accordance with some embodiments, in response to detecting an input corresponding to a first part of a multi-part operation (e.g., using the input receiving unit 1710), the processing unit 1705 starts a continuous audio output sequence associated with a continuous tactile output sequence (e.g., using the continuous output start unit 1715), and in response to detecting an input corresponding to a second part of the multi-part operation (e.g., using the input receiving unit 1710), ends the continuous audio output sequence (e.g., using the continuous output end unit 1720).

[0406] In some embodiments, the temporally varying tactile energy profile of the tactile output sequence mimics the temporally varying acoustic energy profile of the audio output sequence. Depending on the situation, in response to a determination to start an event associated with each application without interrupting an event in which the input corresponding to the first part of the multi-part operation continues, the processing unit 1705 provides a tactile output sequence without an associated audio output sequence (e.g., using the continuous output start unit 1715), and in response to a determination that the input corresponding to the first part of the multi-part operation interrupts a continuous event associated with each application, the processing unit 1705 provides a continuous audio output sequence associated with a continuous tactile output sequence (e.g., using the continuous output start unit 1715).

[0407] As described above, the operations described with reference to FIGS. 7 and 9-13 are optionally implemented by the components shown in FIGS. 14-19. FIG. 18 shows an exemplary functional block diagram of a device 100 configured in accordance with the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 18 may be optionally combined or separated into sub-blocks to implement the principles of the various embodiments described. Accordingly, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0408] As shown in FIG. 18, the device 100 includes a display unit 1801 configured to display an application, a touch sensing surface unit 1803 configured to receive user contact, and a processing unit 1805 connected to the display unit 1801 and the touch sensing surface unit 1803. In some embodiments, the processing unit 1805 includes an input detection unit 1810, an output providing unit 1815, an operation performing unit 1820, an image capture unit 1825, a secure transaction authentication unit 1830, and a storage operation performing unit 1835.

[0409] The processing unit 1805 is configured to detect a first input corresponding to a request to perform a first operation (e.g., using the input detection unit 1810), and in response to the detection of the first input, is configured to provide a first haptic output including a haptic component (e.g., using the output providing unit 1815). Also, in response to the detection of the first input, the processing unit 1805 is configured to perform a first operation (e.g., using the operation performing unit 1820). After performing the first operation, the processing unit 1805 is configured to detect a second input corresponding to a request to perform a second operation including the first operation and an additional operation (e.g., using the input detection unit 1810), and in response to the detection of the second input, provides a second output including a haptic component (e.g., using the output providing unit 1815), and the second output includes the first output combined with the provision of an additional output corresponding to the additional operation. Also, in response to the detection of the second input, the processing unit 1805 is configured to perform a second operation (e.g., using the operation performing unit 1820).

[0410] The following paragraphs

[0411] ~

[0420] describe various embodiments that can be implemented by the device 100 illustrated in FIG. 18, individually or in any combination.

[0411] In some embodiments, in response to detection of a second input, the processing unit 1805 performs a first operation (e.g., using the operation performing unit 1820 and the output providing unit 1815), simultaneously provides a first output, performs an additional operation, and simultaneously provides an additional output.

[0412] In some embodiments, in response to detection of a second input, the processing unit 1805 performs the first operation (e.g., using the operation performing unit 1820) before performing the additional operation and provides the first output (e.g., using the output providing unit 1815) before providing the additional output. In some embodiments, in response to detection of a second input, the processing unit 1805 performs the first operation after performing the additional operation (e.g., using the operation performing unit 1820) and provides the first output after providing the additional output (e.g., using the output providing unit 1815).

[0413] In some embodiments, in response to detection of a second input, the processing unit 1805 performs at least a part of the first operation simultaneously with a part of the additional operation (e.g., using the operation performing unit 1820) and provides at least a part of the first output simultaneously with a part of the additional output (e.g., using the output providing unit 1815).

[0414] In some embodiments, the first operation corresponds to capturing an image using a camera, and the second operation corresponds to capturing an image after a specific interval of time. In some embodiments, the first operation corresponds to a transaction enabling operation for enabling the device to authenticate a secure transaction, and the second operation corresponds to enabling the device to authenticate a secure transaction and authenticating the secure transaction.

[0415] According to some embodiments, the first operation corresponds to a save operation for saving the content within an existing file, and the second operation corresponds to a save-as operation for saving the content within the existing file to a new file. According to some embodiments, the first operation corresponds to a send operation for sending a reply to a message in a message inbox, and the second operation corresp...

Claims

1. A method for a processor to execute on a computing device having a tactile output generator, the method comprising: detecting, by the computing device, a first input corresponding to a request from a user of the computing device to perform a first action that does not include an additional action; in response to detecting the first input, providing a first output including a first tactile component indicative of execution of the first action in response to detecting the first input; performing the first action without performing the additional action; after performing the first action, detecting, by the computing device, a second input corresponding to a request from the user to perform a second action including the first action and the additional action; in response to detecting the second input and without requesting additional input from the user after the second input, providing a second output, where the second output includes: the first tactile component indicative of execution of the first action in response to detecting the second input; and additional output including a second tactile component indicative of execution of the additional action in response to detecting the second input; performing the second action including the first action and the additional action; and wherein the computing device is a wearable computing device for remotely controlling a remote computing device, and the first action is executed on the remote computing device.

2. In response to detecting the second input, performing the first action and simultaneously providing the first output; and performing the additional action and simultaneously providing the additional output; The method according to claim 1, further comprising.

3. In response to detecting the second input, performing the first action before performing the additional action; and providing the first output before providing the additional output; The method according to claim 1 or 2, further comprising.

4. In response to detecting the second input, performing the first action after performing the additional action; and providing the first output after providing the additional output; The method according to claim 1 or 2, further comprising.

5. In response to detecting the second input, Executing at least a portion of the first operation simultaneously with a portion of the additional operation; Providing at least a portion of the first output simultaneously with a portion of the additional output; The method according to claim 1 or 2, further comprising. **Claim 6** The remote computing device includes a camera, The first operation corresponds to capturing an image with the camera, The method according to claim 1 or 2, wherein the second operation corresponds to capturing an image after a specified time interval. **Claim 7** The first operation corresponds to a transaction enabling operation for enabling the computing device to permit a secure transaction, The method according to any one of claims 1 to 5, wherein the second operation corresponds to enabling the computing device to permit a secure transaction and permitting the secure transaction. **Claim 8** The first operation corresponds to a saving operation for saving content in an existing file, The method according to any one of claims 1 to 5, wherein the second operation corresponds to a named saving operation for saving the content in the existing file to a new file. **Claim 9** The first operation corresponds to a sending operation for sending a reply to a message in the message inbox, The method according to any one of claims 1 to 5, wherein the second operation corresponds to a sending and archiving operation for sending a reply to a message in the message inbox and deleting it from the message inbox. **Claim 10** The additional operation has a variable attribute, The method according to any one of claims 1 to 5 and 7 to 9, wherein the additional output is based on the value of the variable attribute. **Claim 11** The method according to any one of claims 1 to 5 and 7 to 9, wherein the second tactile component is separate from the first tactile component. **Claim 12** The method according to any one of claims 1 to 5 and 7 to 11, wherein the additional output includes a non-tactile component. **Claim 13** The method according to any one of claims 1 to 5 and 7 to 11, wherein the second output includes a text component that identifies the additional operation as different from the first operation. **Claim 14** The additional output includes an audio component, The method according to any one of claims 1 to 5, and 7 to 11, wherein the first output does not include an audio component.

15. The computing device includes a touch-sensitive display, the first input and the second input are received through the touch-sensitive display, The method according to any one of claims 1 to 14, wherein the first tactile component of the first output and the second tactile component of the second output are provided via the touch-sensitive display.

16. A device having a tactile output generator, a touch-sensitive surface, and a memory storing one or more programs, wherein the one or more programs detect, on the device, a first input corresponding to a request from a user of the device to perform a first action that does not include an additional action; in response to detecting the first input, provide a first output including a first tactile component indicating execution of the first action corresponding to detecting the first input; perform the first action without performing the additional action; after performing the first action, detect, on the device, a second input corresponding to a request from the user of the device to perform a second action including the first action and the additional action; in response to detecting the second input and without requiring an additional input from the user after the second input, provide a second output, where the second output includes both the first tactile component indicating execution of the first action corresponding to detecting the second input and an additional output including a second tactile component indicating execution of the additional action corresponding to detecting the second input; perform the second action including the first action and the additional action; and includes instructions for the device is a wearable computing device for remotely controlling a remote computing device, and the first action is executed on the remote computing device.

17. The device according to claim 16, wherein the one or more programs include instructions for performing the method according to any one of claims 2 to 15.

18. A computer program having executable code that, when executed, causes a processor of a computing device having a tactile output generator to On the computing device, detecting a first input corresponding to a request from a user of the computing device to perform a first operation that does not include an additional operation; In response to detecting the first input; Providing a first output including a first tactile component indicating execution of the first operation in response to detecting the first input; Performing the first operation without performing the additional operation; After performing the first operation, on the computing device, detecting a second input corresponding to a request from the user to perform a second operation including the first operation and the additional operation; In response to detecting the second input and without requesting additional input from the user after the second input; Providing a second output, where the second output; Includes both the first tactile component indicating execution of the first operation in response to detecting the second input and; An additional output including a second tactile component indicating execution of the additional operation in response to detecting the second input; Performing the second operation including the first operation and the additional operation; Causing to be executed; The computing device is a wearable computing device that remotely controls a remote computing device; The first operation is a computer program executed on the remote computing device. **Claim 19** The computer program according to claim 18, wherein when the executable code is executed, it causes the processor to execute the method according to any one of claims 2 to 15.

Images