Semantic framework for variable haptic output

Abstract

Embodiments of the present disclosure relate to a semantic framework for variable haptic output. The invention provides methods and apparatuses that organize multiple haptic output variants into a tightly integrated semantic framework that uses various information about alert conditions and triggers, application context and other conditions to provide a haptic output system that shares characteristics between related events. In some embodiments, an event class or application class provides a basis for corresponding haptic output. In some embodiments, whether an alert prominence setting is on or off is provided as a basis for adding an increased prominence haptic output to standard haptic output for an alert. In some embodiments, consistent haptics provide branding of associated application classes, applications and / or contexts.

Description

[0001] This application is a divisional application of the invention patent application with an international filing date of July 23, 2015, international application number PCT / US2015 / 041858, which entered the Chinese national phase on February 16, 2017, Chinese national application number 201580044092.6, and entitled "Semantic Framework for Variable Haptic Output". Technical Field

[0002] This disclosure generally relates to methods and apparatus for providing haptic feedback, and more specifically to methods and apparatus for a semantic framework of haptic feedback corresponding to conditions associated with a device or application. Background Technology

[0003] Haptic feedback systems provide users with the ability to interact with subsystems through touch or contact. Haptic systems facilitate these tactile interactions by using actuators, sensors, or both. Summary of the Invention

[0004] This document discloses haptic feedback methods and apparatuses that organize multiple haptic output variants into a tightly integrated semantic framework. This semantic framework provides a haptic output system that shares features among related events using various information about alert conditions and triggers, application context, and other conditions. This disclosure relates to systems and methods for providing haptic responses under various conditions. The disclosed haptic response methods can be implemented using any suitable software, hardware, or both.

[0005] In some implementations, the state associated with the application provides a basis for providing corresponding haptic output at the time associated with a detected alert condition. For example, in response to detecting an alert condition, the state associated with the application at the time associated with the alert condition is determined. If the application is active at the time associated with the alert condition, a first haptic output representing the occurrence of the alert condition is provided; and if the application is inactive at the time associated with the alert condition, a second haptic output representing the occurrence of the alert condition and different from the first haptic output is provided.

[0006] In some implementations, whether a detected alert condition is triggered manually or automatically provides the basis for providing the corresponding tactile output. For example, in response to detecting an alert condition, it is determined whether the alert condition is triggered by a manually initiated event. If the alert condition is triggered by a manually initiated event, a first tactile output corresponding to the manually initiated event notification is provided. If the alert condition is triggered by an automatically initiated event, a second tactile output corresponding to the automatically initiated event notification is provided, and the second tactile output is different from the first tactile output.

[0007] In some implementations, the detected alert condition, associated with user input or a predetermined system event, provides a basis for providing corresponding haptic output. For example, in response to detecting a first alert associated with receiving user input to the application, a first haptic output corresponding to the user input is provided. Following the first haptic output, in response to detecting a second alert condition associated with receiving a predetermined system event in the application, a second haptic output of greater intensity than the first haptic output is provided, corresponding to the predetermined system event.

[0008] In some implementations, the detected alert condition provides a basis for providing a corresponding tactile output for a part of a multi-part operation. For example, in response to receiving input corresponding to the first part of a multi-part operation, an ongoing tactile output sequence is initiated. After initiation and in response to receiving input corresponding to the second part of a multi-part operation, the ongoing tactile output sequence is terminated.

[0009] In some implementations, the detected request to perform an operation provides a basis for providing a corresponding tactile output to a subgroup of another operation. For example, in response to detecting a first input corresponding to a request to perform a first operation, a first output including a tactile component is provided and the first operation is performed. After performing the first operation, and in response to detecting a second input corresponding to a request to perform a second operation including the first operation and an auxiliary operation, a second output including a tactile component and the second output including the first output and an auxiliary output corresponding to the auxiliary operation is provided, and the second operation is performed.

[0010] In some implementations, whether two detected alert conditions are of the same or different types of alert conditions or the application provides a basis for providing corresponding tactile outputs. For example, in response to the detection of a first alert condition, a first output including a first tactile component and a first non-tactile component is provided. After providing the first output, and in response to the detection of a second alert condition, a second output including a second tactile component and a second non-tactile component is provided. If the first alert condition and the second alert condition are different alert conditions of the same type of alert conditions, the first output and the second output share one or more identical components and have one or more different components; however, if the first alert condition and the second alert condition are different alert conditions of different types of alert conditions, the first tactile component is different from the second tactile component and the first non-tactile component is different from the second non-tactile component. In another example,

[0011] In some implementation schemes,

[0012] Note that various embodiments of the methods described herein can be combined with other embodiments described herein. The features and advantages described in the specification are not exhaustive, and in particular, many additional features and advantages will become apparent to those skilled in the art upon reading the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been chosen in principle for readability and instruction purposes, and such choice may not be necessary to depict or define the subject matter of the invention. Attached Figure Description

[0013] Figure 1A A block diagram is provided to illustrate a portable multi-functional device with a touch-sensitive display according to some embodiments.

[0014] Figure 1B A block diagram illustrating exemplary components for event handling according to some implementation schemes.

[0015] Figure 2 A portable multi-functional device with a touchscreen is shown according to some embodiments.

[0016] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some embodiments.

[0017] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some implementations is shown.

[0018] Figure 4B An exemplary user interface, independent of the display, is shown for a multifunctional device with a touch-sensitive surface according to some embodiments.

[0019] Figure 5 Examples of various tactile waveforms according to some implementation schemes are shown.

[0020] Figure 6 Examples of various audio waveform shapes according to some implementation schemes are shown.

[0021] Figure 7 A flowchart illustrating a method for detecting and associating a first alert condition with an application based on some implementation schemes, and providing a corresponding tactile output.

[0022] Figure 8 An exemplary user interface for an email application according to some implementation schemes is shown.

[0023] Figure 9 A flowchart of a method for detecting and associating a first alert condition with an application according to some implementation schemes, and providing haptic output depending on whether the condition is triggered by a manually or automatically initiated event.

[0024] Figure 10 A flowchart illustrating a method for detecting a first alert condition from user input and a second alert condition from a predetermined event associated with an application, and providing corresponding haptic output, according to some implementation schemes.

[0025] Figure 11 A flowchart illustrating a method for detecting and associating a first warning condition with multi-part operation according to some implementation schemes, and providing a corresponding tactile output.

[0026] Figure 12 A flowchart of a method for performing first and second operations based on detection according to some implementation schemes, including first and second inputs, performing operations, and providing corresponding outputs.

[0027] Figure 13 This is a flowchart illustrating a method for detecting first and second warning conditions and providing corresponding outputs according to some implementation schemes.

[0028] Figure 14 This is a functional block diagram of an electronic device according to some implementation schemes.

[0029] Figure 15 This is a functional block diagram of an electronic device according to some implementation schemes.

[0030] Figure 16 This is a functional block diagram of an electronic device according to some implementation schemes.

[0031] Figure 17 This is a functional block diagram of an electronic device according to some implementation schemes.

[0032] Figure 18 This is a functional block diagram of an electronic device according to some implementation schemes.

[0033] Figure 19 This is a functional block diagram of an electronic device according to some implementation schemes.

[0034] Figure 20 A device housing including a first acoustic port is shown according to some embodiments.

[0035] Figure 21A A touch sensor circuit corresponding to a self-capacitive touch pixel electrode and a sensing circuit according to some embodiments is shown.

[0036] Figure 21B A self-capacitance touch sensor according to some embodiments is shown.

[0037] Figure 22A A tactile actuator according to some implementation schemes is shown.

[0038] Figure 22B A tactile actuator according to some implementation schemes is shown.

[0039] Figure 23A For some implementation schemes of wearable devices.

[0040] Figure 23B The illustration shows a user wearing a wearable device and a second electronic device in their pocket, according to some implementation schemes.

[0041] Figure 24 A schematic diagram of a wearable electronic device according to some implementation schemes is shown.

[0042] Figure 25 A flowchart illustrating methods for providing various tactile and other feedback alerts based on type, according to some implementation schemes.

[0043] Figure 26 A flowchart of a method for selectively increasing the salience of various tactile sensations according to some implementation schemes.

[0044] Figure 27 An exemplary user interface for alert-highlighting settings on a device is shown according to some implementation schemes.

[0045] Figure 28 This is a functional block diagram of an electronic device according to some implementation schemes.

[0046] Figure 29 This is a functional block diagram of an electronic device according to some implementation schemes.

[0047] Figure 30 A flowchart is provided for a method of detecting first and second conditions and generating corresponding alerts, including tactile and audio outputs, according to some implementation schemes.

[0048] Figure 31 Various examples of waveforms according to some implementation schemes are shown.

[0049] Figure 32 A flowchart illustrating a method for detecting first conditions in first and second scenarios according to some implementation schemes and generating corresponding alerts including tactile and audio outputs.

[0050] Figure 33a Various tactile and audio waveforms, occurring separately and simultaneously in time, are shown according to some implementation schemes.

[0051] Figure 33b Various tactile and audio waveforms that overlap in time are shown according to some implementation schemes.

[0052] Figure 33cVarious tactile and audio waveforms that overlap in time according to some implementation schemes are shown.

[0053] Figure 34 This is a functional block diagram of an electronic device according to some implementation schemes.

[0054] Figure 35 This is a functional block diagram of an electronic device according to some implementation schemes. Detailed Implementation

[0055] The embodiments described herein disclose a haptic feedback method and apparatus that organizes multiple haptic output variants into a tightly integrated semantic framework, which provides a haptic output system that shares features among related events using various information about alert conditions and triggers, application context, and other conditions corresponding to the haptic output.

[0056] In some implementations, the state associated with the application at the time of the detected alert condition associated with the application provides a basis for providing the corresponding haptic output. In some implementations, whether the detected alert condition is triggered manually or automatically provides a basis for providing the corresponding haptic output. In some implementations, the association between the detected alert condition and user input or a predetermined system event provides a basis for providing the corresponding haptic output.

[0057] In some implementations, the detected alert condition may provide a basis for providing a corresponding tactile output for one part of a multi-part operation. In some implementations, the detected request to perform an operation may provide a basis for providing a corresponding tactile output for a subgroup of another operation. In some implementations, the two detected alert conditions may provide a basis for providing a corresponding tactile output for whether they are the same or different types of alert conditions.

[0058] Exemplary device

[0059] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0060] It will also be understood that while the terms “first,” “second,” etc., are used in some instances herein to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first contact may be named a second contact, and similarly, a second contact may be named a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact.

[0061] The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and in the appended claims, the singular forms “a” and “an” are intended to include the plural forms as well, unless the context expressly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the items listed in connection with the description. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0062] As used herein, depending on the context, the term "if" may optionally be interpreted as meaning "when," "at," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrases "if it is determined..." or "if [the stated condition or event] is detected" may optionally be interpreted as meaning "when it is determined..." or "in response to determination..." or "when [the stated condition or event] is detected," or "in response to the detection of [the stated condition or event]."

[0063] This document describes implementations of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some implementations, the device is a portable communication device such as a mobile phone that also includes other functions such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). iPod and Device. Optionally, other portable electronic devices may be used, such as laptop or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0064] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0065] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0066] Various applications running on the device optionally use at least one shared physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or changed from one application to another and / or within the respective application. In this way, the shared physical architecture of the device (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0067] The focus is now on implementation schemes for portable devices with touch-sensitive displays. Figure 1AThis is a block diagram illustrating a portable multi-functional device 100 with a touch-sensitive display 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and may also be called a “touch-sensitive display system.” Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 (e.g., touch-sensitive surfaces, such as the touch-sensitive display system 112 of device 100) for detecting the intensity of contact on device 100. Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (e.g., generating haptic output on a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0068] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute (alternative) for the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface adjacent to the contact, and / or the resistance and / or variation of the touch-sensitive surface adjacent to the contact may optionally be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement) has been exceeded. In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure) has been exceeded. Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited physical space, which is used (e.g., on a touch-sensitive display) to display an indication and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0069] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the tactile surface remains unchanged, movement of the tactile surface can optionally be interpreted or sensed by the user as “roughness” of the tactile surface. While such interpretations of touch by the user will be limited by the user’s individualized sensory perceptions, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a user’s specific sensory perception (e.g., “release click,” “press click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception described by a typical (or average) user.

[0070] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits.

[0071] Memory 102 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100 (such as CPU 120 and peripheral interface 118) is optionally controlled by memory controller 122.

[0072] Peripheral interface 118 can be used to couple the input and output peripherals of the device to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0073] In some implementations, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some implementations, they are optionally implemented on separate chips.

[0074] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). Wireless communication optionally employs any of a number of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n). Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a mono or binaural headphone) and input (e.g., a microphone).

[0075] I / O subsystem 106 couples input / output peripherals on device 100, such as touchscreen 112 and other input control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / send electrical signals to other input control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click wheels, etc. In some alternative embodiments, input controller 160 is optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing devices such as a mouse. One or more buttons (e.g., ... Figure 2 Optionally, 208) includes an up / down button for volume control of speaker 111 and / or microphone 113. One or more buttons optionally include a push-down button (e.g., Figure 2 (206 in the middle).

[0076] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. Visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects.

[0077] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or tactile contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In one exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.

[0078] Touchscreen 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. Touchscreen 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption therein. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology, such as that from Apple Inc. (Cupertino, California), is used. iPod and The technology discovered in [the text].

[0079] Touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. The user optionally interacts with touchscreen 112 using any suitable object or accessory such as a stylus, finger, etc. In some embodiments, the user interface is designed to operate primarily with finger-based contact and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0080] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that differs from the touchscreen and does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0081] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0082] The device 100 may optionally also include one or more optical sensors 164. Figure 1AAn optical sensor 164 is shown coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device, allowing the user to optionally acquire an image of that user for use in the video conference while viewing other video conference participants on the touchscreen display.

[0083] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor 165 is shown coupled to a strength sensor controller 159 in the I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrostatic sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of the device 100, opposite to the touchscreen display 112 located on the front of the device 100.

[0084] The device 100 optionally also includes one or more proximity sensors 166. Figure 1A A proximity sensor 166 is shown coupled to the peripheral device interface 118. Alternatively, the proximity sensor 166 is coupled to an input controller 160 in the I / O subsystem 106. In some embodiments, the proximity sensor turns off and disables the touchscreen 112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0085] The device 100 may optionally also include one or more tactile output generators 167. Figure 1AA haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., inside / outside the surface of device 100) or laterally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.

[0086] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may be optionally coupled to an input controller 160 in an I / O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from the one or more accelerometers. Device 100 may optionally include, in addition to the accelerometer 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location and orientation (e.g., portrait or landscape) of device 100, as described in U.S. Patent Application 11 / 969,800, filed January 4, 2008, the entire contents of which are incorporated herein by reference.

[0087] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 stores device / global internal state 157, such as... Figure 1A and Figure 3As shown in the diagram. The device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views or other information occupy various areas of the touchscreen display 112; sensor state, which includes information obtained from the device's various sensors and input control devices 116; and position information about the device's position and / or orientation.

[0088] Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0089] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, Wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used on iPod (Apple Inc. trademark) devices.

[0090] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0091] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least one subset of intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen display can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0092] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact lift is detected, before or after contact begins to move is detected, before contact ends is detected, before or after an increase in contact intensity is detected and / or before or after a decrease in contact intensity is detected). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some implementations, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding a third threshold results in a third action. In some implementations, a comparison between the feature intensity and one or more thresholds is used to determine whether to perform one or more actions (e.g., whether to select an appropriate option or abandon the corresponding action), rather than to determine whether to perform the first or second action.

[0093] In some implementations, a portion of the gesture is identified to determine the characteristic intensity. For example, a touch-sensitive surface may receive a series of swipes that transition from a starting position to an ending position, where the intensity of the contact increases. In this example, the characteristic intensity of the contact at the ending position may be based only on a portion of the series of swipes, rather than the entire swipe (e.g., only a portion of the swipe at the ending position). In some implementations, a smoothing algorithm may be applied to the intensity of the swipe contact before determining its characteristic intensity. For example, the smoothing algorithm may optionally include one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic intensity.

[0094] The touch / motion module 130 optionally detects user gesture input. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the icon location). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0095] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term "graphics" includes any object that can be displayed to a user, and non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0096] In some implementations, the graphics module 132 stores data to be used to represent graphics. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from applications, etc., specifying the graphics to be displayed, and also receives coordinate data and other graphic attribute data if necessary, and then generates screen image data for output to the display controller 156.

[0097] The haptic feedback module 133 includes various software components for generating instructions used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0098] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0099] GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to a phone 138 for location-based dialing; to a camera 143 as image / video metadata; and to applications that provide location-based services such as weather desktop apps, local yellow pages desktop apps, and map / navigation desktop apps).

[0100] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0101] • Contacts module 137 (sometimes called address book or contact list);

[0102] • Telephone module 138;

[0103] • Video conferencing module 139;

[0104] • Email client module 140;

[0105] • Instant Messaging (IM) module 141;

[0106] Fitness support module 142;

[0107] • Camera module 143 for still images and / or video images;

[0108] • Image management module 144;

[0109] • Browser module 147;

[0110] • Calendar module 148;

[0111] • Desktop mini-program module 149, which optionally includes one or more of the following: weather desktop mini-program 149-1, stock desktop mini-program 149-2, calculator desktop mini-program 149-3, alarm clock desktop mini-program 149-4, dictionary desktop mini-program 149-5 and other desktop mini-programs obtained by the user, and user-created desktop mini-program 149-6;

[0112] • Desktop app creator module 150 for creating user-created desktop apps 149-6;

[0113] • Search module 151;

[0114] • Video and music player module 152, which optionally consists of a video player module and a music player module;

[0115] • Notepad module 153;

[0116] • Map module 154; and / or

[0117] • Online video module 155.

[0118] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0119] In conjunction with the touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communication via telephone 138, video conferencing 139, email 140, or IM 141, etc.

[0120] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in address book 137, modify already entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0121] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0122] Incorporating RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0123] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS protocols for internet-based instant messaging), receiving an instant message, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" means both telephone-based messages (e.g., messages transmitted using SMS or MMS) and internet-based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

[0124] Incorporating RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for: creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (exercise equipment); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing fitness music; and displaying, storing, and transmitting fitness data.

[0125] Incorporating the touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0126] Incorporating the touchscreen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or photo album), and storing still images and / or video images.

[0127] Combining RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) according to user instructions.

[0128] In conjunction with touchscreen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions that allow a user to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141, instead of email client module 140, is used to send links to specific online videos.

[0129] Each of the modules and applications identified above corresponds to an executable instruction set used to perform one or more of the functions and methods described above.

[0130] Combining RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and calendar-related data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0131] In conjunction with RF circuitry 108, touchscreen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, desktop applet module 149 is optionally a micro-application downloaded and used by a user (e.g., weather desktop applet 149-1, stock desktop applet 149-2, calculator desktop applet 149-3, alarm clock desktop applet 149-4, and dictionary desktop applet 149-5) or a user-created micro-application (e.g., user-created desktop applet 149-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop applet includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop applet).

[0132] Combining RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134 and browser module 147, the desktop applet creator module 150 can optionally be used by the user to create desktop applets (e.g., to transfer user-specified portions of a webpage to a desktop applet).

[0133] In conjunction with the touchscreen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions to search for text, music, sound, images, video, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0134] Incorporating touchscreen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

[0135] Incorporating the touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0136] In conjunction with RF circuitry 108, touchscreen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions; data related to shops and other points of interest at or near a specific location; and other location-based data) according to user instructions.

[0137] In conjunction with touchscreen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions that allow a user to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141, instead of email client module 140, is used to send links to specific online videos.

[0138] Each module and application identified above corresponds to a set of executable instructions for performing one or more of the functions described above, as well as the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as standalone software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures described above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0139] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for the operation of device 100, the number of physical input control devices (such as push-buttons, dial pads, etc.) on device 100 is optionally reduced.

[0140] The predefined set of functions, performed exclusively via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, when a user touches the touchpad, the device 100 is navigated from any user interface displayed on the device 100 to the main menu, home menu, or root menu. In such embodiments, a touchpad is used to implement a "menu button." In some embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0141] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some embodiments. In some embodiments, memory 102 ( Figure 1A (middle) or memory 370 ( Figure 3This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137-13, 155, 380-390).

[0142] Event classifier 170 receives event information and determines the application 136-1 to which the event information should be delivered and the application view 191 of application 136-1. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates the current application view displayed on touch-sensitive display 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0143] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by the application 136-1 or information ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0144] Event monitor 171 receives event information from peripheral interface 118. The event information includes information about sub-events (e.g., user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral interface 118 transmits information it receives from I / O subsystem 106 or sensors (such as proximity sensor 166), accelerometer 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.

[0145] In some implementations, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In some implementations, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0146] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0147] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0148] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events considered as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0149] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure that handles the sub-events. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that forms an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source to which it is identified as the hit view.

[0150] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In some embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher-level views in the hierarchy will still remain actively participating views.

[0151] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver module 182.

[0152] In some implementations, the operating system 126 includes an event classifier 170. Alternatively, the application 136-1 includes an event classifier 170. However, in some implementations, the event classifier 170 is a separate module or part of another module (such as the contact / motion module 130) stored in the memory 102.

[0153] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In some embodiments, one or more application views among the event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0154] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies events from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0155] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a longitudinal orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device pose).

[0156] Event comparator 184 compares event information with predefined event or sub-event definitions and determines, or determines or updates, the state of an event or sub-event based on the comparison. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch movement, touch cancellation, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as dragging on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 112, and lifting the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0157] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0158] In some implementations, the definition of the corresponding event 187 also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0159] When the corresponding event recognizer 180 determines that the sub-event string does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process ongoing sub-events based on touch gestures.

[0160] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, tags, and / or lists indicating how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, tags, and / or lists indicating how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, tags, and / or lists indicating whether sub-events are delivered to a different level of change in a view or programmatic hierarchy.

[0161] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and delaying) the sub-event to the corresponding hit view. In some implementations, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag retrieves the tag and executes a predefined procedure.

[0162] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event string or to an actively participating view. The event handler associated with the sub-event string or the actively participating view receives the event information and executes a predetermined procedure.

[0163] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module 145. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 176 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends it to graphics module 132 for display on a touch-sensitive display.

[0164] In some implementations, event handler 190 includes, or has access to, data updater 176, object updater 177, and GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In some implementations, they are included in two or more software modules.

[0165] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using input devices, and not all user input is initiated on the touchscreen. For example, optional mouse movement and mouse button presses, optionally combined with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may be used as inputs corresponding to sub-events to define the events to be identified.

[0166] Figure 2A portable multifunction device 100 with a touchscreen 112 is shown according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figures) or one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some specific embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0167] Device 100 optionally also includes one or more physical buttons, such as a "home" button or a menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. In some embodiments, menu button 204 includes a fingerprint sensor for recognizing a fingerprint on menu button 204. The fingerprint sensor is optionally used to determine whether the fingerprint on menu button 204 has a fingerprint that matches the fingerprint used to unlock device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.

[0168] In one embodiment, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, a volume control button 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to: power on / off the device by pressing the button and holding it in the pressed state for a predetermined time interval; lock the device by pressing the button and releasing it before a predetermined time interval has elapsed; and / or unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic outputs for a user of device 100.

[0169] Figure 3 This is a block diagram of an exemplary multifunctional device with a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the reference above). Figure 1A The tactile output generator 167 and sensor 359 (e.g., optical sensor, accelerometer, proximity sensor, touch sensor, and / or similar to those mentioned above) are described. Figure 1A The contact strength sensor 165 is a contact strength sensor. The memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100. Figure 1A The memory 370 stores programs, modules, and data structures similar to, or subsets thereof, those stored in the memory 102 of the portable multifunction device 100. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 ( Figure 1A The memory 102 may optionally not store these modules.

[0170] Figure 3Each of the elements identified above is optionally stored in one or more of the previously mentioned memory devices. Each of the modules identified above corresponds to a set of instructions for performing the functions described above. The modules or programs identified above (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures described above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0171] Now turn our attention to the implementation of the user interface (“UI”) optionally implemented on the portable multifunction device 100.

[0172] Figure 4A An exemplary user interface for an application menu on a portable multifunction device 100 according to some embodiments is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof:

[0173] Signal strength indicator 402 for wireless communications such as cellular and Wi-Fi signals; time 404; Bluetooth indicator 405; battery status indicator 406; tray 408 with icons of frequently used applications, such as: an icon 416 labeled "Phone" for telephone module 138, optionally including an indicator 414 of the number of missed calls or voicemail messages; an icon 418 labeled "Mail" for email client module 140, optionally including an indicator 410 of the number of unread emails; an icon 420 labeled "Browser" for browser module 147; and an icon 420 for video and music player module 152, also known as iPod (Apple). (Trademark of Inc.) Module 152, labeled with the icon 422 as “iPod”; and icons for other applications, such as: for IM module 141, labeled with the icon 424 as “text”; for calendar module 148, labeled with the icon 426 as “calendar”; for image management module 144, labeled with the icon 428 as “photo”; for camera module 143, labeled with the icon 430 as “camera”; for online video module 155, labeled with the icon 432 as “online video”; for stock desktop app 149-2, labeled with the icon 422 as “…”. The icons are as follows: 434 for "Stocks"; 436 for "Map" in the map module 154; 438 for "Weather" in the weather desktop app 149-1; 440 for "Clock" in the alarm clock desktop app 149-4; 442 for "Fitness Support" in the fitness support module 142; 444 for "Notepad" in the notepad module 153; and 446 for providing access to the settings of the device 100 and its various applications 136 for setting applications or modules.

[0174] It should be pointed out that, Figure 4A The icon labels shown are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0175] Figure 4B A touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)) is shown. Figure 3 Devices (e.g., tablets or touchpads 355) Figure 3An exemplary user interface on device 300. Device 300 may also optionally include one or more contact intensity sensors (e.g., one or more of sensors 357) for detecting the intensity of contact on the tactile surface 451, and / or one or more tactile output generators 359 for generating tactile outputs for the user of device 300.

[0176] While some examples of input on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given in the following examples, in some embodiments, the device detects input on a touch-sensitive surface separate from the display, such as... Figure 4B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., on the display (e.g., 450) that is aligned with the main axis on the display (e.g., 451). Figure 4B The principal axis corresponding to 453 in the middle (e.g., Figure 4B (452 in the middle). According to some embodiments, the device detects contact with the touch-sensitive surface 451 at a position corresponding to a corresponding position on the display (e.g., Figure 4B (460 and 462 in the example) Figure 4B In the diagram, 460 corresponds to 468 and 462 corresponds to 470. Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and the display of a multi-functional device (e.g. Figure 4B When 450 is separated from the touch-sensitive surface, user input detected by the device on the touch-sensitive surface (e.g., touches on 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may be optionally used for other user interfaces described herein.

[0177] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger touch, finger tap, finger swipe), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of the touch, followed by termination of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or one mouse and multiple finger touches may optionally be used simultaneously.

[0178] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other positional marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3 The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of enabling direct interaction with user interface elements on the touchscreen display) Figure 1A The touch-sensitive display system 112 or Figure 4A In some embodiments of the touchscreen 112, the detected touch on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to focus movement between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that transmits the user's expected interaction with the user interface (e.g., by indicating to the device the user interface element with which the user expects to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0179] Tactile output and associated processes

[0180] Attention now turns to implementations of tactile output and associated processes that can be implemented on electronic devices, such as device 300 or portable multifunction device 100. Although the following description is relative to device 100, the implementations herein can be incorporated into device 300.

[0181] Figure 5Examples of various tactile waveform morphologies according to some implementation schemes are shown. Various tactile types act as atomic tactile components, which are used in various combinations to produce various tactile waveforms with different meanings. The following examples are described to create a vocabulary distinguishing between the tactile output types described herein. The tactile output types described herein are not intended to be exhaustive, and other tactile waveforms may also be used.

[0182] Waveform 505, according to some embodiments, represents an exemplary tactile waveform or "tapping" type tactile output with a relatively large amplitude, which the user can easily perceive (e.g., significantly above the absolute threshold of tactile perceptibility for a given tactile device) even with a single tap. As seen with various tap tactile outputs 505a-e, the waveform varies in amplitude, duration, and frequency. Typically, human hearing can perceive frequencies from 5 Hz to 29 Hz and from 0N to 1.0N (Son, equivalent to ~0-40 dBA) depending on the kHz range. The amplitude that humans can hear is most sensitive in the 1-4 kHz range.

[0183] Waveform 510, according to some embodiments, represents an example of a tactile waveform with a relatively small amplitude (e.g., alternatively, a shorter duration) compared to waveform 505, or a "micro-tap" type tactile output that is not easily perceived by the user of the device outputting the tactile feedback (e.g., close to a low threshold of touch perceptibility for a given tactile device). A "micro-tap" type tactile output is detected as a single micro-tap, but is easily perceptible if multiple micro-tap outputs are consecutive. Micro-tap waveforms 510a-c differ in amplitude, duration, and frequency.

[0184] Waveform 515 represents an exemplary combination of tactile sensations of two micro-taps following a single tap. Since micro-taps are not easily perceived individually, a waveform with a tap preceding the micro-tap can "activate" the user by drawing their attention to the tap, making the micro-tap more noticeable.

[0185] Waveform 520 represents an example of a "fade in / fade out" type waveform, which has a relatively long duration and a decreasing amplitude over time. Waveform 520a is a "tap-fade in / fade out" that begins with a larger tap amplitude and ends with a micro-tap amplitude, while waveform 520b is a micro-tap-fade in / fade out that begins with a smaller micro-tap amplitude and ends with a lower amplitude.

[0186] Waveform 525 represents an exemplary "humming" waveform of continuous high frequency, typically lasting for a relatively long duration (e.g., 0.5 to 1 second). The humming sound may have a high or low amplitude depending on the implementation and is perceived by the user as a consistent vibration.

[0187] Various audio waveforms are optionally synchronized with various tactile waveforms, thereby corresponding to or differing from the frequency, amplitude, and duration characteristics of the tactile output in the acoustic domain. Various atomic tactile components, such as those described above, are used in various combinations to generate various tactile waveforms corresponding to the audio waveforms. The atomic tactile components are modulated by different parameters (e.g., amplitude, count / repetition, timing offset) to generate different audio patterns directly from the tactile device itself, such that the audio waveform corresponds to the tactile waveform. Additionally, according to some embodiments, some waveforms generated by the tactile actuator (e.g., tactile output generator 167) produce audible outputs similar to machine sounds, such as "ticks." Alternatively, in some embodiments, separate audio signal generators and output devices (e.g., audio generators and speakers) can be used to synchronize output audio waveforms having waveforms different from the tactile waveforms.

[0188] Figure 6 Examples of various audio waveform shapes according to some implementation schemes are shown. Waveform 605 represents an example of an audio waveform with various amplitudes, durations, and frequencies. Sometimes, specific waveforms are created to mimic the common sounds associated with the activity to which the audio is paired. Waveform 610 is an example of a specific waveform. Waveform 610a is a waveform that mimics the sound of a walkie-talkie, while waveform 610b is a waveform that mimics the sound of a heartbeat.

[0189] In some implementations, when the tactile and audio waveforms have corresponding shapes—for example, if the tactile and audio waveforms have peaks at similar times and similar durations, making the two waveforms appear similar—they are output synchronously. Waveforms 615 (tactile; dashed line) and 620 (audio; solid line) are shown independently and are then aligned and synchronized in time. In this example, the time alignment causes a tactile waveform with a series of taps to be synchronized with a heartbeat-detecting audio waveform, thereby producing a combined sound and heartbeat sensation, for example, when used in conjunction with a heartbeat monitor.

[0190] In some implementations, audio components share properties but are still distinct; for example, using the same bell or tone (same amplitude, frequency, and duration) played on different scales or instruments. Overlaying different audio frequencies on the same tactile output (e.g., high-pitched versus low-pitched, the same audio played on metal versus glass versus ceramic, etc.) produces different semantic components. These examples are not intended to be exhaustive, and other audio waveforms may also be used.

[0191] For example, a corresponding tactile output may be accompanied by a corresponding audio output, wherein at least a portion of the corresponding audio output occurs simultaneously with at least a portion of the corresponding tactile output, or the corresponding audio output occurs close in time to the corresponding tactile output, such that the corresponding tactile output and the corresponding audio output are perceived as concurrent or simultaneous. The tactile and audio waveforms do not need to be perfectly aligned; the device may take into account the fact that for certain types of tactile and audio outputs, even a slight temporal offset may be perceived by the user as concurrent or synchronous (e.g., because the audio output is processed faster than the tactile output, providing the tactile output before providing the audio output will, in some cases, make the user perceive that the audio and tactile outputs are occurring concurrently or synchronously). Variations in the accompanying audio portion may differ between embodiments including the audio output. For example, the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more musical qualities (e.g., scale, pitch, etc.) associated with the audio component associated with the first output may differ in certain situations.

[0192] Haptic output based on application state

[0193] According to some implementation schemes, at a time associated with a warning condition detected by the application, one or more states associated with the application provide a basis for providing corresponding haptic output. Providing haptic output associated with the application's state results in a more efficient human-machine interface, thereby reducing the amount of time the user spends performing operations, thus reducing energy consumption and increasing battery life of battery-powered devices. Figure 7 This is a flowchart of a method 700 for detecting warning conditions associated with an application and providing corresponding haptic output according to some implementation schemes. Note that in some implementation schemes, the following can be performed: Figure 7 Different steps beyond those shown.

[0194] According to some implementations, the method begins with device 100 detecting 705 an alert condition associated with an application running on computing device 100. According to various implementations, alert conditions for applications running on computing device 100 take various forms. An alert condition is any event, notification, or other alert concerning the device user. One type of alert condition corresponds to an event automatically triggered by or from within the application. For example, application-triggered alert conditions include pre-determined or scheduled alerts, such as reminders, scheduled notifications, or application-generated alarms. A second type of alert condition is an automatically initiated event notification received by the application from an external source. For example, system-generated emails or messages, such as bulk emails, spam, system-generated emails involving mailing lists, or any other system-generated communication. A third type of alert condition is a manually initiated event notification received by the application from a human user other than the user operating the device. For example, an incoming message or call from a known email address or a phone number on the user's contact list. Various alert conditions can be detected, for example, detecting the receipt of an email or text message when the user interface of an email or instant messaging application is displayed on the device's display.

[0195] According to some implementations, in response to an alert condition, device 100 determines the state of 710 associated with the application at the time associated with the alert condition. In one example, the time associated with the alert condition is the time when the alert is received. In a second example, the time associated with the alert condition is the time when the alert is detected.

[0196] In some implementations, determining the state of 710 associated with the application at the time associated with the alert condition includes determining whether the application's user interface is displayed on the device at or around the time of the alert condition. In some cases, the time of the alert condition is used; in others, the time period discussed includes specific time intervals around the time associated with the alert condition, such as during a predefined time window spanning the interval before and / or after the time associated with the alert condition.

[0197] According to some implementations, state 710 can be determined based on whether the application is running on the computing device at the time associated with the alert condition, wherein the application is determined to be active based on whether it is running in the foreground. For example, user interface elements visible on the device display when the application is running. In this case, the application is determined to be inactive based on whether it is not running on the computing device or is running in the background. For example, the device is not in active use, or is not performing calculations under the guidance of the application, or no user interface elements indicating the application's activity state are displayed. Optionally, a combination of the state of the user interface display and the application's activity state is used to determine state 710.

[0198] refer to Figure 8 According to some implementations, an example of an email application 800 actively running in the foreground and visible to the user interface on device 100 is shown. In this example, the email application 800 is active. Conversely, in Figure 4A The image shows an example of an inactive email application on device 100, where the email application 800 is either not running or running in the background because it is not visible on device 100.

[0199] Refer again Figure 7 In this example, the state of 710 associated with the application at the time associated with the alert condition is determined based on whether the corresponding user interface window of the application is displayed in the foreground of the multi-application window user interface at the time associated with the alert condition. For example, if multiple application windows are displayed in a stack, the application in the foreground of the z-layer is considered the active application. As is known in the art, the corresponding user interface window is an application window that can receive and / or is configured to receive user input or user interaction, sometimes referred to as a focused window.

[0200] According to some implementations, at a time associated with an alert condition, one or more other user interface windows are simultaneously displayed on a multi-application window user interface on the device. In this case, determining the state of 710 associated with an application at the time associated with the alert condition includes determining whether the corresponding user interface window of the application is displayed in the multi-application window user interface at the time associated with the alert condition.

[0201] According to some implementations, another way to determine the state of 710 associated with the application at the time associated with the alert condition includes whether user input to the application was detected at the time associated with the alert condition, where the user interaction differs from the alert condition. For example, user input in various cases includes user touch input, voice input, or visual / eye-based input, or any method of receiving user input to the application. In some cases, the time associated with the alert is the time of the alert condition, and in others, it includes a specific time surrounding the time associated with the alert condition, such as during a predefined time window spanning the interval before and / or after the time associated with the alert condition.

[0202] In some implementations, where device 100 determines the level of user engagement associated with the user interaction at the time associated with the alert condition based on the detected user interaction application being active, device 100 also determines the level of user engagement associated with the user interaction at the time associated with the alert condition. In some cases, user engagement is a range of possible user interactions with the application, from the lowest level of engagement such as a single eye gaze, to the intermediate level of engagement such as a single voice or touch, up to the highest level of engagement such as a combination of touch, voice, and eye gaze focused on the application (e.g., current engagement level, historical engagement level, or a combination of both over a pre-determined time period).

[0203] In some implementations, determining the state of 710 associated with the application at the time associated with the alert condition includes determining the state of the computing device at the time associated with the alert condition. In this example, determining whether the application is active includes determining that the computing device is active at the time associated with the alert condition. For example, according to some implementations, an active device includes a device that is powered on (e.g., the display is on) and / or actively used. Similarly, determining whether the application is inactive includes determining that the computing device is inactive in some way at the time associated with the alert condition. For example, in various cases, a powered-off device, a device in sleep or hibernation mode, a device not in active use, and a device with the display off each correspond to an inactive device.

[0204] According to some implementation schemes, based on determining that the application is active at a time associated with the warning condition, device 100 provides 715 a first tactile output having a first set of output characteristics.

[0205] The first set of characteristics of the first tactile output includes one or more of the following: amplitude, duration, regularity, repetition frequency, or variation in the atomic tactile characteristics of the first tactile output. In some cases, the first tactile output is accompanied by a first audio output. For example, a corresponding tactile output may be accompanied by a corresponding audio output, wherein at least a portion of the corresponding audio output occurs simultaneously with at least a portion of the corresponding tactile output, or the corresponding audio output occurs close in time to the corresponding tactile output, such that the corresponding tactile output and the corresponding audio output are perceived as concurrent or simultaneous. The tactile and audio waveforms do not need to be perfectly aligned, and device 100 may take into account the fact that for certain types of tactile and audio outputs, even a slight temporal offset will be perceived by the user as concurrent or synchronous (e.g., because the audio output is processed faster than the tactile output, providing the tactile output before providing the audio output will, in some cases, make the user perceive the audio and tactile outputs as concurrent or synchronous). Variations in the accompanying audio portion may differ between embodiments that include the audio output. For example, the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more musical qualities (e.g., scale, pitch, etc.) associated with the audio component associated with the first output may differ in certain situations.

[0206] According to some embodiments, the first and second sets of features correspond to the device type of the computing device. For example, for the same event, device 100 may optionally provide different tactile outputs based on the device type (e.g., camera vs. watch vs. laptop computer or other handheld device). However, in some embodiments, the difference between the tactile components of the first output and the tactile components of the second output is maintained regardless of the device on which the output is performed. In this case, the same pattern of tactile output is provided across all device types, but the difference in the magnitude of the tactile output is based on the device on which the output is performed.

[0207] Depending on the implementation, the variations within the device itself and the resulting details of the tactile output differ. For example, in some cases, the device has a touch-sensitive display. In some implementations, the first and second tactile outputs are provided via a touch-sensitive display on a computing device. In this example, device 100 may receive both user touch input via a touch-sensitive display and provide tactile output via the same touch-sensitive display. According to some implementations, this bidirectional touch sensitivity allows device 100 to provide feedback based on the received user input.

[0208] As mentioned above, device 100 optionally determines the level of user engagement associated with user interaction at the time associated with the alert condition. In these cases, device 100 also determines one or more of a first set of output features for the first tactile output based on the determined level of engagement. In some embodiments, the amplitude, duration, repetition frequency, and / or tactile components of the audio may vary inversely with the measurement of user interaction / engagement with the application. Therefore, for a higher level of user interaction, the first set of output features is selected such that the first output has a lower intensity, making it more subtle and less noticeable. Thus, an overly strong signal is not exposed to the user when the user already has a high level of engagement or interaction with the application. According to some embodiments, if the user is less engaged, a higher intensity signal is provided to the user to draw additional attention to the device or application. For example, if the detected user interaction includes touch input or touch gestures (and / or user gaze) reflecting a high level of user interaction / engagement, a weaker or less noticeable output is provided to the user because the user may become aware of the alert condition through increased engagement with the application (interacting with the application through gaze or touch). If the detected user interaction includes voice input but does not reflect accompanying touch or eye contact at a lower level or measurement with the application, a stronger or more noticeable output may be provided to draw additional attention to the alert condition.

[0209] Based on the determination that the application is inactive at the time associated with the warning condition, the device 100 optionally provides a second tactile output 720 representing the occurrence of the warning condition, the second tactile output having a second set of output characteristics, wherein the second tactile output is different from the first tactile output.

[0210] According to some embodiments, the second haptic output differs from the first haptic output and has a greater intensity than the first haptic output. For example, the signal amplitude is optionally greater, the frequency is higher, and / or the duration is longer than the first haptic output. In some cases, the second haptic output has a greater intensity than the first haptic output. In some embodiments, the haptic component of the first output has a first set of parameters (e.g., lower amplitude, less periodicity / regularity, lower repetition frequency or event count, shorter output duration, less intrusive / noticeable / obvious) and consists of atomic haptic components that are less noticeable (e.g., micro-taps) during the active state of the device than during the second state. Similarly, if accompanied by a corresponding audio component, the audio component of the first output has a first set of parameters (e.g., lower amplitude, shorter output duration, less intrusive / noticeable / strong audio tone) during the active state and a second set of parameters during the inactive state. According to some embodiments, the purpose of the higher intensity output is to increase the user's attention to the alert condition when the application is inactive.

[0211] As described above, the state of device 100 is optionally used to determine the modulation or output characteristics of the haptic output. For example, according to some embodiments, when the device is in a power-saving state (e.g., because the device battery life has decreased below a predefined threshold, such as 5%, 10%, 15%, or 20%), power-saving haptic outputs (e.g., haptic outputs with lower amplitude, shorter duration, and / or lower frequency) are used, even if those haptic outputs are unlikely to be noticed by the user. When the device is not in a power-saving state, in some cases a default haptic output is used, even if those haptic outputs are not power-saving (e.g., haptic outputs with relatively higher amplitude, longer duration, and / or higher frequency compared to power-saving haptic outputs). In some embodiments, because the device is in a power-saving state, the difference between the haptic components of the first output and the haptic components of the second output is maintained even when using power-saving haptic outputs.

[0212] For the first tactile output, the second tactile output is optionally accompanied by an audio output, the waveform of the second tactile output being generated based on a mirror image and synchronized with the waveform of the accompanying audio output. In some embodiments, the waveform of the tactile component is generated from the waveform of the corresponding audio component and matched, simulated, mirrored, or synchronized with the waveform of the corresponding audio component, such as in combination with... Figure 6 As stated above.

[0213] Various combinations of haptic and audio inputs can be output by the device to provide different semantic information to the user. As a first example, a first haptic output is not accompanied by an audio output, but a second haptic output is accompanied by an audio output. As a second example, a first haptic output is accompanied by a first audio output, and a second haptic output is accompanied by a second audio output different from the first audio output.

[0214] When accompanied by different audio waveforms, a given tactile waveform can generate different perceptions and thus semantic information. For example, a tactile component output with a high-frequency audio tone produces different perceptible units compared to the same tactile component output with a low-frequency audio tone, and therefore generates semantic information. A high-frequency tone can be used to elicit a noticeable increase in the user's attention to the type of event represented by the tactile output, while a low-frequency tone can be used to indicate a change in the state of the event type. For example, a given tactile output can be used to indicate the receipt of a text message, accompanied by a high-frequency tone for urgent messages (or messages from a designated sender) or a low-frequency tone for non-urgent messages or messages received from a non-designated sender. Furthermore, two related alerts (e.g., an alert corresponding to the receipt of a text message and an alert corresponding to the receipt of an email message) have different corresponding audio components with the same rhythm and therefore use the same tactile component (e.g., sharing the tactile component provides information that the two alerts correspond to a message receipt alert, while different audio components provide information about the type of message received).

[0215] In some implementations, the first tactile output is similar to, the same as, or substantially the same as the second tactile output. As an example, the first tactile output is accompanied by an audio output, and the second tactile output is accompanied by the same audio output; in this example, the first tactile output differs from the second tactile output. Recall... Figure 8 Activity email application 800 and Figure 4A The above example illustrates an inactive email application. The following is a sample alert condition message for email application 800 and the output corresponding to various conditions, states, modes, and statuses.

[0216] Table 1: Emails

[0217]

[0218] Table 1 shows the various possible outputs for a single alert condition based on received emails. The first seven rows show the application in an active state with no new messages, depending on the various modes the application might be in when receiving emails: where the application displays the mailbox screen, inbox, selected messages, outgoing messages, ongoing drafts, email search field, and the application running in the background. No output (haptic or other means) is presented because no new messages are received in the first eight rows.

[0219] However, the last eight lines show the conditions for receiving new messages and the corresponding outputs. As shown, for most active modes, the haptic output is a micro-tap and the audio output is the default received email audio. However, for the last line where the application is inactive and the penultimate line where the application is active but running in the background, the haptic output is a fade-in / fade-out tap and the audio output remains the default received email. Therefore, audio output is maintained regardless of whether the application is active, but the haptic output has a higher intensity (fade-in / fade-out tap versus micro-tap) when the application is inactive or running in the background. Table 1 provides conditions and corresponding outputs according to some implementations, but is merely exemplary; other combinations of alerts, states, modes, conditions, and outputs are used according to different implementations.

[0220] As an example, at a first time point, when the application is active, device 100 detects 705 a first alert condition associated with the application and, in response to detecting the first alert condition when the first application is active, provides 715 a first output representing the first alert condition. At a second time point, when the application is inactive, device 100 detects a second alert condition and, in response to detecting the second alert condition when the application is inactive, provides 720 a second tactile output representing the second alert condition, the second tactile output being different from the first tactile output.

[0221] It should be understood that, Figure 7 The specific order of operations described herein is merely exemplary and is not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with reference to other methods (e.g., methods 900, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) also apply in a similar manner to the above references. Figure 7 Method 700 is described herein. For example, the inputs, alert conditions, applications, states, and haptic outputs described above with reference to method 700 optionally have one or more of the features of other methods described herein (e.g., methods 900, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) in terms of inputs, alert conditions, applications, states, and haptic outputs. For the sake of brevity, these details will not be repeated here.

[0222] Tactile output triggered by alert conditions

[0223] Warning conditions can be triggered manually or automatically. According to some implementations, whether a detected warning condition is triggered manually or automatically provides the basis for providing the corresponding tactile output. Providing tactile output associated with whether warning conditions are triggered manually or automatically results in a more efficient human-machine interface, thereby reducing the amount of time the user spends performing operations, thus reducing energy consumption and increasing battery life of battery-powered devices. Figure 9 A flowchart of a method 900 for detecting a first alert condition associated with an application according to some embodiments, and providing haptic output depending on whether the condition is triggered by a manually initiated or automatically initiated event. Note that in some embodiments, the following can be performed: Figure 9 Different steps beyond those shown.

[0224] The method begins with device 100 detecting 905 alert conditions associated with an application. Depending on the implementation, alert conditions take various forms for applications running on computing device 100. Alert conditions can be any event, notification, or other alert concerning the device user. One type of alert condition corresponds to manually initiated event-triggered events, while a second type corresponds to automatically initiated event-triggered events. In some cases, device 100 has a touch-sensitive display.

[0225] In response to the detection of an alert condition, device 100 determines whether the alert condition was triggered by a manually initiated event. This step includes determining whether the alert condition corresponds to an event initiated by a human user. Various factors can contribute to device 100's determination of whether an event was initiated by a human user, as further described in conjunction with Table 2.

[0226] In some implementations, determining that a 910 alert condition characteristic indicates that the alert condition was triggered by a manually initiated event includes determining whether the alert condition corresponds to an event initiated by a human user other than the user of the computing device. Some exemplary conditions used for this determination are shown in Table 2 below.

[0227] In some implementations, determining that an alert condition is triggered by a manually initiated event includes determining whether the alert condition corresponds to input received from a human user, identifying the user of the computing device, and including instructions from the human user to alert the user of the computing device regarding the input. In some implementations, the input received from the human user comes from the user of device 100; however, in some implementations, the input is received from another user on another device (e.g., a personal message sent from another user to the device user).

[0228] Table 2 illustrates several exemplary scenarios in which device 100 has information for determining whether the 910 alert message is triggered by a manually initiated event, whether it is initiated by a human user, and whether that user is a different user from the user of device 100. The applications shown in Table 2 include messaging, email, phone, calendar, timer, activity monitor, and social networks, but any other application that provides similar information to device 100 in making that determination may also be included.

[0229] Table 2: Manual / Automatic

[0230]

[0231] For the example shown, the Human / Automatic list indicates whether a particular alert, condition, and application corresponds to an alert that can be initiated manually (humanly), automatically, or by either a human or automated means. For example, rows two, four, six, and seven each indicate a manual action by a human user. In these examples, the action has been specifically adopted by the user of the device (user column), such as user input into applications on the device (e.g., messaging, email, or phone applications), making it easy for device 100 to identify the user as a human user, and more specifically, a device user. However, in other cases, some alert conditions are of a type known to device 100 and can be generated manually or automatically by a human user. For example, rows one, three, and five have alert conditions of this type. Specifically, new messages received in a messaging application are typically manual (human-initiated) events, such as text or other messages sent by different device users. However, messages can also be automatically generated by the system, such as spam or advertising text messages. Similarly, new email messages can be initiated by a human user or automatically by the system, such as mass emails generated by a machine, spam emails, or emails from subscription mailing list services. In the last example, incoming calls in a phone application are typically human-initiated events, but can also be automatically initiated calls from a system-generated caller list, such as for fundraising or as part of a campaign. In each of these cases, additional information about the source of the alert condition is needed to make a more conclusive determination of whether the event was initiated manually or automatically.

[0232] For example, for an email or message, device 100 can determine whether the alert condition corresponds to personal communication from a sender in a contact list that is managed, created, controlled, or otherwise associated with the user. In some cases, the simplest example is a physical contact file, such as a contact card with an email address or phone number, but other contact lists also include email addresses or phone numbers from previous communications with the user. These examples are only in the fact that there are many other ways in which device 100 can use this method to determine whether the communication could have been manually initiated by a human user.

[0233] Another example is shown in the second-to-last row of Table 2. In this example, the social networking application sends a notification that a user has been tagged in a post. While the communication itself originates from the social networking server, the notification content indicates that the post was manually initiated by a human user connected to a device user connected to the social networking application.

[0234] According to some implementation schemes, based on the determined warning conditions triggered by a manually initiated event, device 100 provides 915 a first tactile output corresponding to the manually initiated event notification.

[0235] Once device 100 determines that the alert condition was initiated by a manual event, it provides a tactile output corresponding to the alert condition. According to some embodiments, the first tactile output is provided via a touch-sensitive display of a computing device. In some embodiments, the determination also includes determining a measurement of the urgency associated with the manually initiated event and modulating the intensity of the first tactile output 912 based on the measurement of the urgency. For example, device 100 analyzes the content of an incoming message from another end user to determine the urgency level of the message. Therefore, in some embodiments, device 100 examines date information in an email or the subject / content of the message, keywords used in the subject or message (e.g., “urgent”), or examines metadata included in the message, such as a “important” tag. According to some embodiments, device 100 determines whether the sender or recipient of the message includes an individual marked as important by the user or with some other special indication. Device 100 then provides 915 a tactile and / or audio output with an intensity (e.g., amplitude, duration, frequency) proportional to the urgency procedure associated with the content of the incoming message. According to some implementation schemes, typically, expected or less urgent notifications / outputs have one set of output characteristics, while unexpected notifications have a stronger second set of output characteristics, thereby attempting to attract the user's increased attention.

[0236] In some cases, device 100 determines one or more contextual attributes associated with a manually initiated event and, based on these, modulates the intensity of the 912 haptic output. For example, various factors are considered to evaluate the contextual attributes associated with an incoming manual notification and the intensity of the haptic or audio notification to be provided. Contextual attributes include the identity of the recipient sending the message, other recipients containing the message and their identifiers, the timing of the message reception, the user's location when the notification is received, the activity the user is performing when the notification is received, the medium used to transmit the message, etc. This table is not exhaustive.

[0237] According to some implementations, the first tactile output corresponding to a manually initiated event notification is stronger than the second tactile output corresponding to an automatically initiated event notification. For example, a stronger output is more likely to draw the user's attention to a manually initiated event. In some cases, the tactile feedback of a manually initiated alert, by being more personalized, targeted, and more likely to be directly relevant to the user of the device, helps to draw the user's immediate attention to events that the system has determined require more immediate user attention.

[0238] According to some implementation schemes, for example, the first tactile output is characterized by a first set of features, including one or more of the following: the amplitude of the first tactile output, the duration of the first tactile output, the regularity associated with the first tactile output, the repetition frequency of the tactile features in the first tactile output, and the selection of tactile features constituting the first tactile output. The second tactile output is characterized by a second set of features, including one or more of the following: the amplitude of the second tactile output, the duration of the second tactile output, the regularity associated with the second tactile output, the repetition frequency of the tactile features in the second tactile output, and the selection of tactile features constituting the second tactile output.

[0239] In some cases, two manually initiated events can be received in a single line. A first alert condition can be detected corresponding to an incoming email message from a first human sender, and in response to determining that the incoming email message corresponds to a manually initiated event, device 100 provides a first haptic output. A second alert condition is detected corresponding to an incoming text message from a second human sender, and in response to determining that the incoming text message corresponds to a manually initiated event, device 100 provides a first haptic output. Many other variations are possible according to some implementations.

[0240] In response to detecting an alert condition associated with the application, and based on determining that the alert condition is triggered by an auto-initiated event, device 100 provides 920 a second tactile output corresponding to the auto-initiated event notification, wherein the second tactile output is different from the first tactile output.

[0241] Many examples have automatically triggered alert conditions. Referring again to Table 2, the alert conditions for calendar, timer, and activity monitor applications each change automatically and are generated by the application itself. In a slightly different example, the last row shows alert conditions as an activity summary for a social network, presented as an email. In this example, the alert condition is a scheduled, periodic occurrence. Therefore, even though the summary may include some event information about user-initiated actions (e.g., an email from a social network summary containing a specific post, tagged by another user in a post above), the email is not considered a manually initiated event because it is not triggered by a request or instruction from a human user (it only includes that information). According to some implementations, a second haptic output is provided via a touch-sensitive display of the computing device.

[0242] According to some implementations, device 100 determines that alert conditions are automatically triggered by an application (e.g., pre-determined or scheduled alerts, such as application-generated reminders or automatic alarms), or automatically initiated event notifications received by the application from external sources (e.g., automatically / system-generated emails or messages). For example, a sender email address like noreply@domain.com implies that an email was sent automatically because no real human sender responded to the email.

[0243] According to some implementations, based on the determination that the automatically initiated event corresponds to an event occurring outside of device 100, device 100 provides a first variation of the second haptic output corresponding to a notification of the externally initiated event. Similarly, based on the determination that the automatically initiated event corresponds to an event initiated internally, device 100 provides a second variation of the second haptic output corresponding to a notification of an internally generated automatically initiated event. Therefore, according to some implementations, different notifications are provided for the same application depending on whether the event is generated by an incoming message.

[0244] According to some implementation schemes, device 100 determines that an alert condition occurs at a predetermined time or that a report indicating the fulfillment of predetermined trigger conditions has an automatic triggering function. For example, the fulfillment of some predetermined trigger conditions (e.g., your activity goals for the day have been met, you have left a geofenced area, etc.) is an automatic type of triggering.

[0245] For manually initiated alert triggers, device 100 analyzes characteristics associated with the alert conditions to determine one or more contextual attributes associated with the automatically initiated event and modulates the intensity of the second output 912 based on one or more contextual attributes. Depending on the circumstances, various factors are considered when evaluating the contextual attributes associated with the automatic notification and the intensity of the haptic or audio notification to be provided. These include, for example, the notification type (e.g., single event versus recurring event, such as a periodic / repeating alarm), the user's location at the time the event occurs, whether the notification is externally or internally triggered, the activity the user is performing at the time the notification is received, and, for external notifications, the medium used to deliver the message.

[0246] For manually triggered events, and for automatically initiated events, device 100 determines a time-varying measurement of the urgency associated with the automatically initiated event and modulates the intensity of a second tactile output 912 indicating the time-varying measurement of the urgency within a specific time window. For example, for automatically initiated (e.g., internal, pre-scheduled) triggers, this set of tactile parameters is modulated 912 based on the urgency of the event associated with the notification or timing. According to some embodiments, for example, for a camera application, device 100 provides a series of micro-tap outputs constituting a captured photo, with the intervals between micro-tap outputs gradually decreasing as the time of photo capture approaches, providing a sense of urgency or a time-varying output that aids user understanding of timing parameters.

[0247] In some implementations, the first set of features of the first tactile output differs from the second set of features of the second tactile output. Similarly, if accompanied by a corresponding audio component, the audio component of the first output has a first set of parameters (higher amplitude, longer output duration, more disturbing / perceptible / obvious audio notes) than the automatically initiated event notification.

[0248] Specific examples include detecting alert conditions associated with an application and accordingly determining whether the alert condition corresponds to personal communication from a sender in a contact list associated with the user, wherein a first tactile output corresponding to a personal alert notification is provided, or whether the alert condition does not correspond to personal communication from a sender in a contact list associated with the user, wherein a second tactile output corresponding to an automatic alert notification is provided, wherein the first tactile output has a greater intensity than the second tactile output.

[0249] The tactile output may optionally be accompanied by a corresponding audio output, wherein at least a portion of the corresponding audio output occurs simultaneously with at least a portion of the corresponding tactile output, or the corresponding audio output occurs close in time to the corresponding tactile output, such that the tactile output and the corresponding audio output are perceived as concurrent or simultaneous. The tactile and audio waveforms do not need to be perfectly aligned, and the device 100 may take into account the fact that, for certain types of tactile and audio outputs, even a slight temporal offset may be perceived by the user as concurrent or synchronous (e.g., because the audio output is processed faster than the tactile output, providing the tactile output before providing the audio output will, in some cases, make the user perceive that the audio and tactile outputs are occurring concurrently or synchronously). Variations in the accompanying audio portion may differ between embodiments including the audio output. For example, the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more musical qualities (e.g., scale, pitch, etc.) associated with the audio component associated with the first output may differ in certain situations.

[0250] According to some implementations, for the first tactile output, the second tactile output is accompanied by an audio output, the waveform of the second tactile output being generated based on a mirror image and synchronized with the waveform of the accompanying audio output. In some implementations, the waveform of the tactile component is generated from the waveform of the corresponding audio component and matched, simulated, mirrored, or synchronized with the waveform of the corresponding audio component, such as in combination with... Figure 6 As stated above.

[0251] According to some implementations, various combinations of tactile and audio waveforms are output. According to some implementations, a first tactile output is not accompanied by an audio output, but a second tactile output is accompanied by an audio output. In one example, a first tactile output is accompanied by a first audio output and a second tactile output is accompanied by a second audio output; in this case, the first tactile output is the same as the second tactile output, and the first audio output is different from the second audio output. In some implementations, the same tactile waveform produces different sensations when accompanied by different audio waveforms. For example, if an audio (ringtone) tone is played in a high-pitched to low-pitched manner and accompanied by the same tactile component (or substantially similar tactile components that a typical user would not be able to distinguish), the sensation of the tactile component is different despite being the same tactile. A higher-pitched or sharper version of the ringtone can be used for the second output to increase the user's attention. In some implementations, the first tactile output is similar to, the same as, or substantially the same as the second tactile output.

[0252] In another example, the first tactile output is accompanied by an audio output and the second tactile output is accompanied by the same audio output, but in this example, the first tactile output is different from the second tactile output.

[0253] It should be understood that, Figure 9 The specific order of operations described herein is merely exemplary and is not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with reference to other methods (e.g., methods 700, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) also apply in a similar manner to the above references. Figure 9 Method 900 is described herein. For example, the inputs, alert conditions, events, applications, states, and haptic outputs described above with reference to method 900 may optionally have one or more of the features of other methods described herein (e.g., methods 700, 1000, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) in terms of inputs, alert conditions, events, applications, states, and haptic outputs. For the sake of brevity, these details will not be repeated here.

[0254] Feedback on notification-based haptic output

[0255] Alert conditions can be triggered by user input or pre-defined system events (e.g., reminders, notifications). According to some implementations, whether a detected alert condition is associated with user input or a pre-defined system event provides a basis for providing corresponding tactile output. Providing tactile output associated with the type of alert condition results in a more efficient human-machine interface, thereby reducing the amount of time the user spends performing operations, thus reducing energy consumption and increasing battery life of battery-powered devices. Figure 10 A method 1000 is provided to detect a first alert condition from user input and a second alert condition from a predetermined event associated with an application, and to provide corresponding haptic output, according to some embodiments. Note that in some embodiments, it is possible to perform... Figure 10 Different steps beyond those shown.

[0256] The method begins by detecting a first alert condition on computing device 100 associated with receiving user input to an application. According to some embodiments, the first alert condition may be any of a variety of alert conditions described elsewhere herein, and the application may be any application executing on device 100. According to some embodiments, computing device 100 includes a touch-sensitive display for receiving user input.

[0257] According to some embodiments, in response to the detection of a first alert condition, device 100 provides 1010 a first tactile output having a first intensity and corresponding to user input to an application. The first tactile output is provided in direct response to user input and optionally as feedback as a result of user input. According to some embodiments, a stronger tactile feedback (e.g., higher amplitude or longer amplitude) is used for undesirable alerts that do not respond to user input than for feedback that directly responds to user input. In some embodiments, undesirable alerts are alerts that occur outside the context in which the user's attention is directed towards the device (e.g., alerts that occur when the user is not actively using the device or when the user is not looking at the device). For example, an electronic message alert generated by the device when the user is wearing the device but not looking at the device and not interacting with an instant messaging application is an undesirable alert, while the same alert received when the user looks at the device and / or opens an instant messaging application on the device would be a desired alert. Similarly, a user expects feedback to a specific user input (such as a "click" when the user places a pin on a map) because the user is actively engaging with the device and is therefore more likely to notice a less intense tactile alert. According to some embodiments, a first tactile output is provided via a touch-sensitive display on the computing device 100. The first intensity may be a predetermined or adjustable combination of the amplitude, frequency, and / or duration of the tactile waveform, as described elsewhere herein.

[0258] Device 100 also detects 1015 a second alert condition associated with receiving a predetermined system event in the application. For example, the second alert condition corresponds to an event that is automatically triggered by the application or triggered from within the application. According to various embodiments, this event includes alerts that occur at a predetermined time or reports that a predetermined criterion is met, application-generated reminder notifications or alarms, automatically initiated event notifications received by the application (e.g., automatically / system-generated emails or messages, such as system-generated bulk emails, spam, system-generated emails pointing to mailing lists, etc.) or manually initiated event notifications received by the application (e.g., events initiated by a human user—such as incoming messages or calls from known email addresses or phone numbers on the user's contact list).

[0259] In response to the detection of a second alert condition, device 100 provides 1020 a second tactile output having a second intensity and corresponding to a predetermined system event, wherein the second intensity is greater than the first intensity. The second intensity is greater than the first intensity in one or more aspects of amplitude, frequency, and / or duration. When computing device 100 includes a touch-sensitive display, the second tactile output is provided via the touch-sensitive display on the computing device.

[0260] This design aims to increase the user's attention to alert conditions not originating from user input by providing a larger second intensity of the haptic output. For example, compared to a user who has not touched the device with their finger, user feedback when using an application—where the finger may have already touched the device 100—requires a smaller haptic output to be noticed by the user. Feedback upon touching the device is expected by the user and is therefore optionally weaker and more subtle than unexpected output. According to some implementations, direct manipulation will result in brief, immediate haptic feedback (<100 milliseconds). Haptic feedback is less intrusive in response to user input (e.g., with lower amplitude, less haptic characteristics, longer periodicity / repetition, shorter duration, weaker haptic characteristics, etc.). On the other hand, notifications are typically unexpected outputs and therefore require greater strength, thereby increasing user attention in some cases. Thus, notifications are optionally longer (~1.5 seconds) and somewhat more intrusive.

[0261] For example, device 100 detects 1005 a first alert condition corresponding to a user selection of a user interface element displayed on an application user interface associated with a first application, and in response to the first alert condition, provides 1010 a corresponding first tactile output representing the user selection of the user interface element. According to some embodiments, the corresponding first tactile output includes a first tactile component having a first tactile feature with a first intensity and a first duration. According to some embodiments, after providing the corresponding first tactile output, device 100 detects 1015 a second alert condition corresponding to an alert condition received by the first application, and in response to the second alert condition, provides 1020 a corresponding second tactile output representing the receipt of an alert notification. The corresponding second tactile output includes a second tactile component: (1) the second tactile component includes a second tactile feature with a second intensity and a second duration, and (2) the second tactile component is different from and more pronounced than the corresponding first output based on the second tactile feature being more pronounced than the first tactile feature, the second intensity being greater than the first intensity, or the second duration being longer than the first duration. For example, an incoming telephone call may have a stronger tactile output than answering or hanging up a call on a telephone application.

[0262] Table 3: Telephone

[0263]

[0264]

[0265] Table 3 shows the telephone application and output corresponding to three different alert conditions: incoming call, answering a call, and hanging up a call. In this example, the user answering or hanging up a call is an action expected by the user taking the action, and the corresponding tactile output accompanying this action is considered feedback to the user. For these two alert conditions, the tactile output is less intense (e.g., a micro-tap). In contrast, an incoming call is a less desirable type of alert condition, and since an incoming call actually exists for a limited period of time, a more intense tactile feedback is intended to gain the user's attention (e.g., a buzzing sound). According to some implementations, a stronger tactile feedback (e.g., a higher amplitude or longer tactile feedback) is used for undesirable alerts that do not respond to user input than for those tactile feedbacks that directly respond to user input.

[0266] Table 3 provides conditions and corresponding outputs according to some implementation schemes, but is merely exemplary. Other combinations of alerts, statuses, modes, conditions, and outputs may be used depending on the implementation scheme.

[0267] In some cases, tactile outputs may vary based on whether they originate from specific user input. In some embodiments, device 100 first detects a first alert condition associated with a first application, the first alert condition originating from corresponding user input. For example, a user provides touch input to type an email message in an email application. According to some embodiments, device 100 provides a corresponding first tactile output representing the first alert condition. Continuing the example above, the device provides a low-intensity, short-interval buzzing sound for each user touch input. After providing the corresponding first tactile output, device 100 detects a second alert condition associated with the first application, the second alert condition not originating from receiving user input. In the context of an email application, this alert condition is optionally an incoming email in the email application. According to some embodiments, device 100 then provides a corresponding second tactile output representing the second alert condition, the corresponding second tactile output differing from the corresponding first output and having a greater intensity. For example, the second tactile output may optionally have a greater intensity and / or a prolonged tap to notify the user of an incoming message.

[0268] According to some embodiments, the first and second tactile outputs may have various features that affect their intensity and ability to attract the user's attention. For example, the first tactile output has a first set of features including one or more of the following: the amplitude of the first tactile output, the duration of the first tactile output, the regularity associated with the first tactile output, the repetition frequency of tactile features in the first tactile output, and the selection of tactile features constituting the first tactile output. The second tactile output has a second set of features including one or more of the following: the amplitude of the second tactile output, the duration of the tactile components associated with the second tactile output, the regularity associated with the second tactile output, the repetition frequency of tactile features in the second tactile output, and the selection of tactile features constituting the second tactile output. According to various embodiments, any feature may differ between the first and second tactile outputs, or all of them may differ.

[0269] According to some implementations, various combinations of tactile and audio outputs are provided. In some implementations, a first tactile output is not accompanied by an audio output, but a second tactile output is accompanied by an audio output. In one example, a first tactile output is accompanied by a first audio output and a second tactile output is accompanied by a second audio output; in this case, the first tactile output is the same as the second tactile output, and the first audio output is different from the second audio output. In some implementations, the same tactile waveform produces different sensations when accompanied by different audio waveforms. For example, if an audio (ringtone) tone is played in a high-pitched tone against a low-pitched tone, and accompanied by the same tactile component (or substantially similar tactile components that a typical user would not be able to distinguish), the sensation of the tactile component is different despite being the same tactile. A higher-pitched or sharper version of the ringtone can be used in the second output to increase the user's attention. In some implementations, the first tactile output is similar to, the same as, or substantially the same as the second tactile output.

[0270] According to some implementations, any tactile output may be accompanied by an audio output, wherein the waveform of the second tactile output is generated based on a mirror image and synchronized with the waveform of the accompanying audio output. In some implementations, the waveform of the tactile component is generated from the waveform of the corresponding audio component and matched, simulated, mirrored, or synchronized with the waveform of the corresponding audio component, such as in combination with... Figure 6 The tactile and audio waveforms do not need to be perfectly aligned. Device 100 may take into account the fact that, for certain types of tactile and audio outputs, even a slight temporal offset will be perceived by the user as concurrent or synchronized (e.g., because the audio output is processed faster than the tactile output, providing the tactile output before the audio output will, in some cases, make the user perceive that the audio and tactile outputs are occurring concurrently or synchronously). Variations in the accompanying audio components may differ between implementations that include the audio output. For example, the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more musical qualities (e.g., scale, pitch, etc.) associated with the audio component associated with the first output may differ in certain situations.

[0271] In some implementations, there is a coordination between haptic intensity and device volume. For example, the first intensity of the first haptic output and the second intensity of the second haptic output are adjusted in a relationship opposite to the volume setting on the device, such that if the audio channel is turned off or the volume is reduced, the haptic output is optionally amplified, either independently or exceeding and above the default / predetermined intensity setting.

[0272] In some implementations, before providing the first tactile output, device 100 determines 1025 whether an input-based modulation condition exists, and in response to determining the existence of the input-based modulation condition, device 100 modifies 1030 the first tactile output before providing the first tactile output.

[0273] For example, an input-based modulation condition is user engagement. In this case, device 100 detects a corresponding alert condition and determines that the alert condition originates from a corresponding user input received on a computing device. According to some embodiments, device 100 then identifies a measurement of user engagement associated with the detected user input and modulates the first signal strength of a first tactile output based on the measurement of user engagement. For example, in some embodiments, the intensity of the audio (e.g., amplitude, duration, repetition frequency) and / or the tactile component varies inversely with the measurement of user interaction / engagement with the application / device. For higher levels / measurements of user interaction / engagement (such as touch or direct manipulation), output characteristics are selected such that the first output is subtle or less intrusive / perceptible. Thus, when the user already has a high level of engagement or interaction with the application / device, the user is not bombarded with overly strong feedback signals. For example, the weakest output is provided for touch / direct manipulation, such as if the user touches the device while providing input, the user is not bombarded with overly strong tactile feedback. On the other hand, if the user is less engaged, a stronger signal is provided to attract additional attention.

[0274] Table 4: Participation / Haptic Feedback Level

[0275]

[0276]

[0277] Table 4 illustrates user engagement at various levels of alert conditions, showing the changes in user focus and the resulting tactile output level. The tactile output levels shown can be viewed as a sliding scale of the tactile intensity levels required to gain user attention given primary and secondary user focus. For example, when the alert condition is in feedback form, the highest level of user engagement is direct manipulation of the application that generated the alert condition. For instance, if the user inputs information into the application, which is already the user's primary focus, a smaller tactile input is needed to attract the user's attention, and therefore the lowest level of tactile input, "Level 1," can be output. This might be the minimum possible tactile input, such as a single micro-tap.

[0278] Skip to the third line. If the user is touching the device but inputting into a different application, the user is focused on the device and therefore has a certain level of engagement, but because the user is using a different application than the one providing the alert, the user has a lower level of engagement than when inputting into the application providing the alert. In this case, the appropriate output is a "Level 2" haptic output. For example, a Level 2 output might be two micro-tapes in a row.

[0279] Skipping to the second-to-last line, the user might have the alerting application active but running in the background, and might not have touched the device at the time of the alert. As a result, in this case, a stronger haptic might be needed to draw the user's attention. Therefore, a "level 3" haptic output might be appropriate, for example, an initial tap followed by a micro-tap or a series of micro-tapes.

[0280] Finally, the last line shows the user where the application issuing the alert is inactive. In some cases, the application isn't even running in the background, or the device is asleep. In this instance, the maximum haptic, "level 4," is appropriate to attract the user's attention. In some cases, for example, a buzzing haptic output might be suitable. According to some implementations, a stronger haptic (e.g., higher amplitude or longer haptic) is used for unwanted alerts that don't respond to user input than for haptic feedback that directly responds to user input.

[0281] This is merely an example, using a limited number of scenarios and examples of haptic output, and assumes that all applications and alerts are treated the same; this may not be the case in other situations. Furthermore, a particular level of haptic output may be the "default" output, for example, level 2, and subsequently, depending on the user's focus, the device may adjust the intensity of the haptic output upwards or downwards. Table 4 provides conditions and corresponding outputs according to some implementations, but is merely exemplary; other combinations of alerts, states, focus, conditions, and outputs may be used depending on the specific implementation.

[0282] In some implementations, the tactile output is modified based on the location of the user input. In this example, device 100 detects a corresponding alert condition and determines that the alert condition originates from receiving tactile user input on the device. By reducing the amplitude of the tactile energy of the first tactile output to make the location of receiving the tactile user input on the device perceptible, device 100 modulates the spatial tactile energy distribution of the first tactile output. According to some implementations, the reduced amplitude of the tactile energy may make it imperceptible at locations far from the user's finger / tactile input location. For example, a low amplitude may be perceptible by the finger at the point of contact rather than other parts of the hand and / or wrist. Similarly, in some implementations, device 100 detects a corresponding alert condition and determines that the alert condition originates from receiving tactile user input on the device, but then specifically identifies the location on the device where the tactile user input is received and modulates the spatial tactile energy distribution of the first tactile output by providing tactile energy greater than a specific proportion of the tactile energy of the first tactile output within a specific radius threshold of the location where the tactile user input is received on the device. According to some implementations, the location on the device where the tactile user input is received changes over time, and therefore the spatial tactile energy corresponding to the time-varying location of the received tactile user input also changes over time.

[0283] According to some implementation schemes, the morphological characteristics of the tactile output simulate the tactile input. For example, device 100 detects a corresponding warning condition, determines that the warning condition originates from receiving tactile user input, and subsequently identifies the temporal morphological attributes of the tactile user input. According to various implementation schemes, the temporal morphological attributes of the tactile input include detecting contact on a touch-sensitive surface, monitoring whether the contact remains on the surface and how far and in what direction it moves, and determining the amount of pressure applied to the contact point. Subsequently, device 100 modifies the morphological characteristics of the first tactile output over time to simulate the temporal morphological attributes of the tactile user input. For example, the morphological characteristics simulated by the first tactile output include changing the amplitude, duration, frequency, etc., of the tactile output reflecting the tactile input.

[0284] In some implementations, the waveform of the tactile component does not need to be perfectly aligned with the user's tactile input, and the device 100 may take into account the fact that the tactile output will be perceived by the user concurrently or simultaneously with the tactile input, although they may be slightly off-time (e.g., because the tactile output is processed faster than the user can perceive it as a delayed response, thus making the user perceive the tactile output concurrently or simultaneously). In some implementations, specific tactile features of a predetermined set of available tactile features are selected to constitute the tactile component of the first output, the selection being based on which features best simulate the user input. Some examples include micro-taps for simple touches, pen-like taps for longer / deeper touches, humming sounds for swipes, and humming sounds with a final, enhanced sound for swipes with a final finger lift. According to some implementations, for example, time-varying morphological properties include, for example, the time-based contact pressure distribution between the tactile user input and the touch-sensitive surface of the computing device, and modifications including modulating the energy distribution of the first tactile output over time to reflect the time-based contact pressure distribution between the tactile user input and the touch-sensitive surface of the computing device.

[0285] According to some implementations, haptic output can simulate a period of continuous user input, gestures, mechanical adjustments, or other device manipulation by the user. For example, device 100 first detects a corresponding alert condition and determines that the alert condition originates from continuous user input. According to some implementations, continuous user input includes haptic user input on a touch-sensitive display on a computing device, rotating a knob, pressing a button, or other user input involving device manipulation with detectable start / end times. In some implementations, the device may determine that the alert condition originates from gestures, mechanical adjustments, etc. In response to the determination, device 100 initiates a first haptic output at the initiation time of the continuous user input, detects the termination of the continuous user input, and subsequently terminates the first haptic output (e.g., mimicking user input) at the termination time of the continuous user input. In some implementations, the haptic output initiates at the start time of the user input and terminates at the end time of the user input. Therefore, the haptic feedback / output generally extends throughout the duration of the user input. The haptic output can be provided at the location of the user input and is not limited to a touch-sensitive surface. For example, if a rotating knob on the side of a watch is turned, a haptic sensation similar to a click can provide continuous feedback as the user turns the knob.

[0286] In some implementations, before providing the second tactile output, device 100 determines 1035 whether there are environmental condition-based modulation conditions, and in response to determining the existence of environmental condition-based modulation conditions and before providing the second tactile output, device 1040 modifies the second tactile output.

[0287] For example, device 100 determines whether there are environmental conditions that could potentially impede the user's perception of tactile output. In this example, device 100 determines whether one or more environmental conditions affecting the device could potentially interfere with the tactile output when a corresponding warning condition occurs. According to some embodiments, environmental conditions include: noise (which may include measured ambient noise and identification of the user in a library, museum, cinema, hospital, or other places generally known to be quiet), movement / vibration that affects the user's physical receptivity to tactile sensation, level of engagement with the device (e.g., as described above: browsing, touching, talking), and proximity of the user to the device (e.g., the device on the user's person being located at a distance greater than a specified proximity threshold from the device, etc.).

[0288] Table 5: Environmental Conditions / Haptic Feedback

[0289]

[0290]

[0291] Table 5 illustrates various environmental conditions that may exist during alert conditions and for various application states, according to some implementation schemes, and possible modifications to the haptic output provided for those conditions. The environmental conditions shown include low and high noise levels, low and high vibration levels, short and long distances from the device, and high and low user engagement levels, as related to the sections above and Table 4. This table is set up with examples of various environmental conditions and how the haptic output can be modified to adjust for a given application state (active or inactive). Table 5 only shows modifications to haptic output, but similar modifications can be made to non-haptic outputs, such as audio. The haptic modification column lists possible no modification (keeping the default haptic for alerts), increases of 1, 2, or 3, and decreases of 1; many variations are possible, these are merely examples. For some conditions, such as high noise, low vibration (inactive), high vibration (active), and long distance from the device (lines 3, 4, 6, 7, and 11), the device can modify the default haptic by adding one or two steps to these environmental conditions that might cause a certain level of distraction for the user, such as a micro-tap to a tap or two micro-tapes. For other conditions, such as high vibration (inactive) or long distance from the device (lines 8 and 12), the device can modify the default haptic by adding a larger number to these environmental conditions that might cause a larger level of distraction for the user, such as a micro-tap to a buzzing sound. In some cases, the haptic may be accompanied by audio output as part of the modification. Alternatively, or in addition to modifying the intensity of the haptic, the device may delay the haptic output until the environmental conditions become more favorable for the user to receive the haptic output, for example, for an environmental condition marked as increase 3. In this example, the device may periodically determine the environmental conditions to ensure that the interfering environmental conditions have decreased and therefore the haptic can be output. In other cases, no delay is applied. These examples are very limited, using a limited number of cases and examples of haptic output, and assume that all applications and alerts are treated the same, which may not be the case in other situations.

[0292] In some implementations, no modification is required because the environmental conditions do not cause much distraction for the user (e.g., rows marked "None" in the modification column). Finally, under some environmental conditions, the device may reduce the default haptic feedback associated with the alert condition; for example, for high engagement levels, less output is needed from the moment the user engages. Table 5 provides conditions and corresponding outputs according to some implementations, but is merely exemplary; other combinations of environmental conditions, alerts, statuses, and outputs may be used depending on the implementation.

[0293] According to some implementations, device 100 delays tactile output. First, device 100 determines that there are no interfering environmental conditions for the device. In this example, device 100 provides a corresponding second tactile output based on the user's state of receptivity, and delays providing the corresponding second output to the user based on the determination that one or more environmental conditions exist as potential interferences to the tactile output. For example, the delay may continue until a subsequent instance where no interfering environmental conditions exist (e.g., device 100 determines again at a later point in time whether interfering environmental conditions exist). For example, if the user is very active when receiving the notification (e.g., running or exercising) and therefore cannot perceive the tactile sensation, the device may delay the tactile output until the activity is completed, such as waiting until the user finishes the activity, and then provide the tactile output corresponding to the notification. In other cases, the alert is urgent enough that the device will not delay it regardless of environmental conditions. In these examples, the determination of environmental conditions acts as a proxy for estimating the user's physical receptivity to the tactile sensation, for example, based on the properties of the user's current surroundings.

[0294] According to some implementations, device 100 provides variations of haptic output, such as modifications described in conjunction with Table 5. In this example, device 100 detects a corresponding alert condition associated with a corresponding application and determines that the corresponding alert condition does not originate from received user input. In response to determining that the corresponding alert condition does not originate from received user input, device 100 determines whether one or more environmental conditions exist for the device as potential interference with the haptic output when the corresponding alert condition occurs, and based on determining that no interfering environmental conditions exist for the device, device 100 provides the user with a first variation of a second haptic output, the first variation of the second haptic output having a first set of output features. According to some implementations, for example, the variation may be a softer, shorter duration haptic output including more subtle haptic features (e.g., micro-taps). Based on determining that one or more environmental conditions exist for the device as potential interference with the haptic output, device 100 provides the user with a second variation of the second haptic output, the second variation of the second haptic output having a second set of output features different from the first set of features, and the second variation of the second haptic output having a greater intensity than the first variation of the second haptic output.

[0295] In some implementations, device 100 may modify haptic output to "activate" the user. For example, device 100 first detects a corresponding alert condition associated with a corresponding application and then determines that the alert condition does not originate from received user input. According to some implementations, in response to determining that the alert condition does not originate from received user input, the device provides an activation haptic output as a precursor to a second haptic output, the activation haptic output being designed to increase the level of user engagement operating the device to that of the second haptic output, wherein at least a portion of the activation haptic output is more pronounced than the second haptic output. Device 100 optionally provides the second haptic output during and after a specified time interval in which the activation haptic output is provided.

[0296] According to some embodiments, device 100 may provide a variation of activation. In one example, the activation haptic output includes an activation haptic component having a time-varying waveform morphology characterized by a progressively increasing intensity of the haptic component over time. In some embodiments, the activation haptic output includes a haptic waveform with gradually increasing intensity, subtly preparing the user for a notification. According to some embodiments, a significant increase in audio volume accompanies the appearance of the haptic component after activation. In another example, the activation haptic output optionally includes an activation haptic component having a time-varying waveform morphology characterized by an enhanced haptic feature followed by a gradual decrease in the intensity of the haptic component. According to some embodiments, alternatively, device 100 may specifically activate the user's sensitivity by first providing a stronger activation haptic (and optionally audio) output and then providing a milder output once the user has felt the activation. According to some embodiments, in response to a corresponding alert condition derived from received user input, a second haptic output precedes the activation haptic output and directly provides the first haptic output, without a corresponding prior activation haptic output. In various implementation schemes, the characteristics of tactile output are selected based on the urgency or context associated with the warning condition, and the intensity of the output reflects the urgency of the warning.

[0297] It should be understood that, Figure 10 The specific order of operations described herein is merely exemplary and is not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with reference to other methods (e.g., methods 700, 900, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) also apply in a similar manner to the above references. Figure 10Method 1000 is described. For example, the inputs, alert conditions, applications, and haptic outputs described above with reference to method 1000 optionally have one or more of the features of other methods described herein (e.g., methods 700, 900, 1100, 1200, 1300, 2500, 2600, 3000, and 3200) in the inputs, alert conditions, applications, and haptic outputs described herein. For the sake of brevity, these details will not be repeated here.

[0298] Tactile output for multi-part operation

[0299] According to some implementation schemes, whether the detected warning condition is part of a multi-part operation provides a basis for providing the corresponding tactile output. Providing tactile output associated with whether the warning condition is part of a multi-part operation results in a more efficient human-machine interface, thereby reducing the amount of time the user spends performing operations, thus reducing energy consumption and improving the battery life of battery-powered devices. Figure 11 A flowchart of a method 1100 for detecting and associating a first alert condition with multi-part operation according to some implementation schemes, and providing a corresponding tactile output. Note that in some implementation schemes, it is possible to perform... Figure 11 Different steps beyond those shown.

[0300] The method begins by receiving 1105 input corresponding to the first part of a multipart operation performed by an application running on computing device 100. In some embodiments, the input corresponding to the first part of the multipart operation is detected on a touch-sensitive surface of device 100.

[0301] In response to receiving an input corresponding to the first part of a multi-part operation, device 100 initiates an ongoing tactile output sequence 1110. According to various embodiments, the output tactile sequence may be ongoing, and the sensation may be continuous, constant, variable but continuous, or intermittent over a period of time, as long as it continues until a second input is received. In some embodiments, in response to receiving an input corresponding to the first part of a multi-part operation, the device also initiates an ongoing audio output sequence accompanying the ongoing tactile output.

[0302] According to some embodiments, device 100 has a touch-sensitive surface and provides a sequence of tactile outputs via the touch-sensitive surface. For example, the tactile outputs include tactile vibrations of the touch-sensitive surface in a predetermined pattern perceptible to a user of the device that is directly touching the touch-sensitive surface.

[0303] In some embodiments, in addition to the ongoing tactile output sequence, device 100 also provides 1115 first tactile feedback in response to receiving input corresponding to a first portion of a multi-part operation. In some cases, the first tactile feedback is based on an event and an indication of the initiation of the event, and differs from the ongoing tactile output sequence. According to some embodiments, for example, the first tactile feedback has a stronger tactile output than the ongoing tactile output sequence.

[0304] According to some implementations, after initiating an ongoing tactile output sequence, device 100 receives 1120 an input corresponding to the second part of a multi-part operation. For device 100 with a touch-sensitive surface, the input corresponding to the second part of the multi-part operation is detected on the touch-sensitive surface.

[0305] In some embodiments, two devices are included. Therefore, in some embodiments, device 100 detects inputs corresponding to both a first part and a second part of the multipart operation. However, in some embodiments, device 100 detects the input corresponding to the first part of the multipart operation, and the second device detects the input corresponding to the second part of the multipart operation.

[0306] According to some embodiments, device 100 may receive additional user input. According to some embodiments, for example, after an ongoing haptic output sequence is initiated 1110 in response to the first part of a multipart operation, and before receiving input 1120 corresponding to the second part of the multipart operation, device 100 receives 1125 one or more additional user inputs different from the input corresponding to the second part of the multipart operation. According to some embodiments, in these cases, the device continues to provide ongoing haptic output regardless of the additional user input until the input corresponding to the second part of the multipart operation is received.

[0307] According to some embodiments, in response to receiving input corresponding to the second part of a multi-part operation, device 100 terminates the ongoing haptic output sequence 1130. In some embodiments, in response to detecting input corresponding to the second part of a multi-part operation, device 100 also terminates any ongoing audio output sequence.

[0308] According to some implementations, the input corresponding to the second part of the multipart operation follows the input corresponding to the first part of the multipart operation and is time-spaced from the input corresponding to the first part of the multipart operation by a specified time interval corresponding to the duration of the event. Furthermore, in some cases, the ongoing tactile output sequence provides and indicates the occurrence of the event throughout its duration.

[0309] According to some embodiments, the device also provides 1135 second tactile feedback in response to receiving input corresponding to a second part of a multi-part operation, the second tactile feedback being different from the ongoing tactile output sequence. In some cases, the second tactile feedback indicates the termination of an event and is different from the ongoing tactile output sequence. According to some embodiments, for example, the second tactile feedback has a stronger tactile output than the ongoing tactile output sequence.

[0310] Depending on the implementation, the multi-part operation can take various forms. According to some implementations, the multi-part operation is a secure transaction processing, where the first part of the multi-part operation includes equipping the device to authorize the secure transaction processing. In this example, in response to the user providing credentials, such as a password or biometric authentication such as a fingerprint, the device enters a payment mode, where payment information can be provided to the payment terminal via a Near Field Communication (NFC) subsystem. Therefore, equipping the device is the first part, and will initiate a corresponding ongoing tactile output. Furthermore, according to this example, the second part of the multi-part operation includes authorizing the secure transaction processing. According to some implementations, authorization is thus a second step and includes, for example, providing payment information to the payment terminal when the NFC subsystem enters the NFC field of the payment system from the first step of equipping.

[0311] In some implementations of NFC multipart secure transaction processing, the first part of the multipart operation includes determining that user authentication is required to complete the secure transaction. For example, in response to the NFC subsystem entering the NFC field of the payment terminal, the user is prompted to provide credentials such as a password or biometric authentication such as a fingerprint to authorize the secure transaction. In this example, the second part of the multipart operation includes receiving user authentication for the secure transaction. For example, haptic output is provided to remind the user that user authentication information is required before the payment process can proceed; the user authentication information optionally includes password-based or biometric authentication. In some implementations, the device provides a first haptic output mode to remind the user to provide user authentication information between when user authentication is determined to be required and between when user authentication is received (e.g., between requesting and receiving user authentication), and a second haptic output mode is provided by the device between when user authentication information is received and between when the secure transaction is authorized (e.g., between receiving user authentication and providing payment information to the payment terminal). According to some implementations, in this example, the second haptic output mode differs from the first haptic output mode, which reminds the user to place the NFC subsystem near the payment terminal.

[0312] According to other payment implementations, the input corresponds to secure transaction processing without the use of NFC. In this example, the first part of the multipart operation is the selection of a first user interface element to initiate the transaction. For financial transactions such as online purchases, the selection of an element corresponding to activating online payment allows the user to access the user interface to enter security or confidential information. As another example, a user requests to log in to a secure location, such as a secure email account or an online retailer. Continuing with the above example, the multipart operation includes one or more inputs for entering information required to process the transaction. For example, throughout the duration of information input, inputs provide credit card information and other verification information necessary to authorize the financial transaction, along with a corresponding ongoing haptic output sequence. Similarly, inputs could be a username and password for logging into a secure online account. In some implementations, the event corresponds to filling one or more data entry fields with the information required to process the transaction, and the second part of the multipart operation corresponds to the selection of a user interface element to authorize the completion of the transaction, such as an authorized payment user interface element.

[0313] According to some implementations, the first part of the multipart operation includes closing a draft of a document, while the second part of the multipart operation includes returning to the draft of the document. Examples of closing a document, according to various implementations, include closing a draft of an email or text message to switch to another task or open another application on the device, and returning to the document includes switching back from another task or other application to the draft of the email or text message.

[0314] In some implementations, the inputs are separate inputs that occur sequentially. According to some implementations, for example, the input corresponding to the first part of a multipart operation is a first user input interacting with an application running on the device, while the input corresponding to the second part of the multipart operation is a second user input interacting with the application, wherein the second user input differs from the first user input. For example, the first user input to the application could be a first tap and lift, while the second user input could be a second tap and lift within the same application.

[0315] In some cases, multi-part operations are multi-part gestures. According to some embodiments, for example, the operation includes first detecting input corresponding to a first part of the multi-part operation as a contact on a touch-sensitive surface of device 100, and detecting input corresponding to a second part of the multi-part operation including detecting movement of the contact on the touch-sensitive surface. In an example of a swipe gesture, the first part is the user touching the touch-sensitive surface, and the second part is the user moving the contact on the touch-sensitive surface while maintaining contact with the surface. In another example, the operation includes first detecting input corresponding to a first part of the multi-part operation as movement of the contact on the touch-sensitive surface, and detecting input corresponding to a second part of the multi-part operation when the contact is detected to be lifted off the touch-sensitive surface.

[0316] According to some implementations, the input corresponding to the first and second parts of a multi-part operation includes a single gesture, wherein the input corresponding to the first part of the multi-part operation is the initial portion of the gesture (e.g., touching and holding with a finger) and the input corresponding to the second part of the multi-part operation is the subsequent portion of the gesture (e.g., lifting the finger from the touch). For example, in a text editing or browsing application, pressing a finger on a word first highlights the word, and then continuing to press the finger on the highlighted word invokes the cut-copy-paste menu. In this example, lifting the touch would represent a selection of the display aspect corresponding to that touch and holding.

[0317] In some implementations, the gesture is initiated on a portion of the touch-sensitive surface corresponding to a first location in the user interface displayed on the display of device 100 and terminated on a second portion of the touch-sensitive surface corresponding to a second location in the user interface different from the first location. According to some implementations, for example, a touch-swipe-lift sequence can be used to perform drag-and-drop or to reposition the focus area in a camera viewfinder.

[0318] According to some implementations, input corresponding to the first part of a multi-part operation initiates an event associated with the corresponding application, and this event continues until input corresponding to the second part of the multi-part operation is received, at which point the input corresponding to the second part of the multi-part operation terminates the event associated with the corresponding application. According to some implementations, for example, the first input is a selection of a user interface element (e.g., provided on the application's application user interface) that initiates the event associated with the application, while the second input is a selection of a different user interface element (e.g., on the application user interface of the same application) that terminates the event associated with the application.

[0319] According to some embodiments, the application is a text editing application, and the input corresponding to the first part of the multi-part operation is the selection of a first user interface element that enables a text typing mode for the text editing application. In this example, the multi-part operation includes one or more inputs for text input into the text editing application, the event corresponding to text manipulation in the text editing document in response to the text typing input, and the input corresponding to the second part of the multi-part operation is the selection of a user interface element that disables a text typing mode for the text editing application. According to some embodiments, the text editing application is a communication application such as an email or instant messaging application that allows a device user to communicate with other users. According to some embodiments, the text editing application is a non-communication text editing application, such as a word processing application, notepad or sticky note application, or other applications that allow text typing but do not provide independent communication functionality.

[0320] According to some implementations, the characteristics of the ongoing tactile output sequence are selected based on the event initiated by the input corresponding to the first part of a multi-part operation. For example, placing a call in a tactile output sequence that is kept at a lower intensity (e.g., lower energy, less disruptive, less noticeable) than a financial transaction event or security login provides the user with confidentiality or security information, thereby attracting additional attention to the transaction. In this example, throughout the duration of the multi-part operation (e.g., after entering confidential information but before the transaction is completed), the user is thus reminded that the previous typing allowed him or her to complete the transaction quickly, avoiding a compromise on the confidentiality of the information being entered. In some cases, a timeout function is used to make the confidential information no longer available. Additionally, different tactile output modes optionally provide the user with information about the tasks they are performing.

[0321] According to some implementations, input corresponding to the first part of a multi-part operation pauses an ongoing event associated with the application, while input corresponding to the second part of the multi-part operation resumes the ongoing event associated with the application. For example, in some cases, the first part initiates a pause during an ongoing event. In some implementations, the first input corresponds to a selection of a user interface element to pause an action, such as playing a multimedia file, video clip, song, or an in-progress telephone call, while the second input is a selection of a different user interface element to resume the action.

[0322] According to some implementations, the intensity of an ongoing haptic output sequence increases over time until a subsequent input corresponding to a multipart operation is received. For example, the haptic energy (e.g., amplitude, frequency, haptic characteristics) of the ongoing haptic output sequence is modulated over time (e.g., by increasing intensity or haptic energy) to alert the user to increased urgency, thus providing additional input corresponding to the multipart operation. According to various implementations, such increased output may correspond to an alert that terminates an event initiated by the first part of the multipart input, or resumes an event suspended by the first part of the multipart input.

[0323] In some implementations, the time-varying tactile energy distribution of the tactile output sequence simulates the time-varying acoustic energy distribution of the audio output sequence. According to some implementations, such as combining... Figure 6 The temporal waveforms of the tactile and audio sequences are simulated, mirrored, aligned, and / or synchronized with each other.

[0324] According to some embodiments, the output includes audio output depending on whether the first part of the multipart operation initiates a new event or pauses an ongoing event. According to some embodiments, in response to determining that input corresponding to the first part of the multipart operation does not pause an ongoing event but initiates an event associated with the corresponding application, device 100 provides a haptic output sequence without an accompanying audio output sequence, and in response to determining that input corresponding to the first part of the multipart operation pauses an ongoing event associated with the corresponding application, device 100 provides a continuing audio output sequence accompanied by the continuing haptic output sequence. In some embodiments, the audio output sequence accompanies the haptic output sequence only when the first part of the multipart operation pauses or interrupts an ongoing event or action. During the pause, the user is likely not engaged with the device and / or application. For example, the user may passively wait for a call to be transmitted and therefore tolerate additional stimulation associated with the audio output. However, in some embodiments, if the actual event is initiated by the first part of the multipart operation, such as the user actively typing an email message or entering secure financial or login information, the audio sequence is not accompanied by the haptic sequence, according to some embodiments.

[0325] According to some embodiments, the application is a voice communication application. For example, according to some embodiments, the application may be a regular telephone application, a voice internet application such as Skype, or a voice communication application such as Gchat embedded in an email application. According to some embodiments, the event is an ongoing voice communication occurring upon receiving input corresponding to the first part of a multi-part operation, and the input corresponding to the first part of the multi-part operation is a selection of a first user interface element of the voice communication application that suspends the ongoing voice communication. For example, a voice call may be suspended due to call forwarding or a call hold request. According to some embodiments, the input corresponding to the second part of the multi-part operation is a selection of a second user interface element to resume voice communication.

[0326] It should be understood that, Figure 11 The specific order of operations described herein is merely exemplary and is not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with reference to other methods (e.g., methods 700, 900, 1000, 1200, 1300, 2500, 2600, 3000, and 3200) also apply in a similar manner to the above references. Figure 11 Method 1100 is described. For example, the inputs, operations, applications, and haptic outputs described above with reference to method 1100 may optionally have one or more of the features of other methods described herein (e.g., methods 700, 900, 1000, 1200, 1300, 2500, 2600, 3000, and 3200) in the inputs, operations, applications, and haptic outputs described herein. For the sake of brevity, these details will not be repeated here.

[0327] Tactile output for subgroup operations

[0328] According to some implementation schemes, whether the detected request for an operation is related to an operation that is a subgroup of another operation provides a basis for providing the corresponding tactile output. Providing tactile output related to whether the request for an operation is related to an operation that is a subgroup of another operation results in a more efficient human-machine interface, thereby reducing the amount of time the user spends performing the operation, thus reducing energy consumption and increasing the battery life of battery-powered devices. Figure 12 An exemplary method flowchart of a method 1200 for detecting first and second inputs to perform first and second operations according to some implementation schemes, performing the operations, and providing corresponding outputs. Note that in some implementation schemes, the executable... Figure 12 Different steps beyond those shown.

[0329] The method begins by detecting a first input at 1205 corresponding to a request to perform a first operation. According to some embodiments, the computing device 100 includes a touch-sensitive display, and according to some embodiments, receives the first input via the touch-sensitive display.

[0330] According to some embodiments, in response to detecting a first input, device 100 provides 1210 a first output including a tactile component. According to some embodiments, if the device includes a touch-sensitive display, the tactile component of the first output is provided via the touch-sensitive display.

[0331] Similarly, according to some implementations, in response to the detection of a first input, device 100 performs 1215 a first operation. The first operation can be any operation performed by device 100 in response to the input. According to various implementations, examples described below include capturing an image with a camera, a transaction-supporting operation to enable the device to authorize secure transaction processing, a save operation to save content in an existing file, and a send operation to send a reply to a message in a message inbox; however, this list is not intended to be exhaustive, and other operations of device 100 are contemplated as the steps of the method are followed.

[0332] According to some embodiments, after performing the first operation, device 100 detects 1220 a second input corresponding to a request to perform a second operation, the second operation including the first operation and an auxiliary operation. According to some embodiments, if computing device 100 includes a touch-sensitive display, the second input is received via the touch-sensitive display.

[0333] According to some embodiments, in response to the detection of a second input, device 100 provides 1225 a second output including a tactile component, wherein the second output includes the first output and provides an auxiliary output corresponding to the auxiliary operation. According to some embodiments, if the device includes a touch-sensitive display, the tactile component of the second output is provided via the touch-sensitive display.

[0334] In some embodiments, the tactile component corresponding to the second output includes a first tactile component corresponding to the first output and a second tactile component corresponding to the auxiliary output, and the second tactile component is different from the first tactile component. According to some embodiments, for example, the second tactile component has a higher intensity (e.g., amplitude, duration, frequency) than the first tactile component.

[0335] According to some embodiments, the auxiliary output includes a non-tactile component. According to various embodiments, the non-tactile component can be any audio or visual component (e.g., graphics, text). In some embodiments, the auxiliary output includes an audio component, but the primary output does not include an audio component. For example, audio output is provided during an interval when a camera application measures a specified time interval for a countdown to image capture, but no audio is accompanied by the operation of capturing the image itself.

[0336] For auxiliary outputs including audio output, according to some embodiments, the haptic output is accompanied by a corresponding audio output, and according to some embodiments, the audio and haptic portions are aligned when at least a portion of the audio output is concurrent with at least a portion of the haptic output, or the audio output appears temporally close to the haptic output. The haptic and audio waveforms do not need to be perfectly aligned; device 100 may consider the fact that for certain types of haptic and audio outputs, even a slight temporal offset is perceived by the user as concurrent or synchronized (e.g., because the audio output is processed faster than the haptic output, providing the haptic output before providing the audio output will, in some cases, make the user perceive that the audio and haptic outputs are occurring concurrently or synchronously). Variations in the accompanying audio portion may differ between embodiments including audio output. For example, the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more musical qualities (e.g., scale, pitch, etc.) associated with the audio component associated with the first output may differ in certain situations.

[0337] In some implementations, the same tactile waveform produces different sensations when accompanied by different audio waveforms. For example, if an audio (ring) tone is played in a high-pitched tone against a low-pitched tone, and accompanied by the same tactile component (or substantially similar tactile components that a typical user would not be able to distinguish), the sensation of the tactile component will differ despite being the same tactile sensation. A higher-pitched or sharper version of the ringtone can be used as a second output to increase the user's attention. In some implementations, the first tactile output is similar to, the same as, or substantially the same as the second tactile output.

[0338] According to some implementations, the second output includes a text component that identifies the auxiliary operation as different from the first operation. For example, the text component may describe auxiliary information. For instance, the output corresponding to send and archive operations provides text information indicating that the message has been archived. As another example, the output corresponding to a payment success message indicates that security information has been verified.

[0339] According to some implementations, the auxiliary operation has a variable value, and the auxiliary output has a value based on a variable attribute. For example, according to some implementations, the time interval before the camera takes a picture can be 5 seconds, 10 seconds, 15 seconds, etc. According to some implementations, for a 10-second camera timer, the auxiliary output is therefore held for 10 seconds accordingly.

[0340] According to some implementations, device 100 then performs a second operation 1230. The timing of the execution of the first and second operations may vary accordingly. According to some implementations, the first operation is performed concurrently with providing a first output, and an auxiliary operation is performed concurrently with providing an auxiliary output. For example, for a self-timer, a first output corresponding to the first operation is provided when the timer counts down, and an auxiliary output corresponding to the auxiliary operation is provided when the shutter takes a picture.

[0341] According to some implementation schemes, the first operation may be performed before the auxiliary operation and the first output may be provided before the auxiliary output is provided. For example, for a first operation corresponding to a message sending (e.g., email) function, the second operation is a sending and archiving operation, and the sending operation is performed before the auxiliary operation, which is part of the archiving operation.

[0342] According to some implementation schemes, at least a portion of the first operation is performed concurrently with a portion of the auxiliary operation, and at least a portion of the first output is provided concurrently with a portion of the auxiliary output.

[0343] The following are some examples that help describe the steps of method 1200. In a first example, the first operation corresponds to capturing an image with a camera. According to various embodiments, the camera is integrated into device 100 or is located away from the device but remotely controlled by the device. In this example, the second operation corresponds to capturing an image after a specified time interval. According to some embodiments, specifically, the method of this example begins by detecting a first input corresponding to a request to capture a first image with the camera, and in response to detecting the first input, providing a first output including a tactile component. Also in response to detecting the first input, device 100 captures the first image. According to some embodiments, after capturing the first image, device 100 detects a second input corresponding to a request to capture a second image after a specified time interval, and in response to detecting the second input, provides a second output including a tactile component. In this example, the second output includes the first output and provides an auxiliary output corresponding to the expiration of measuring the specified time interval and capturing the second image after the specified time interval.

[0344] According to some implementations, in another example, the first operation corresponds to a transaction-supporting operation that enables the device to authorize secure transaction processing, while the second operation corresponds to enabling the device to authorize secure transaction processing and authorizing secure transaction processing. For example, in an online payment scenario where credit card information is required, the operation is initiated to prepare the NFC subsystem to provide payment credentials at a payment terminal or other point of sale, or to respond to a user login request requiring user credentials and a password. Specifically, according to this example, the method begins by detecting a first input corresponding to a request to perform a transaction-supporting operation that enables the device to authorize secure transaction processing, and in response to detecting the first input, provides a first output including a haptic component. Also in response to detecting the first input, the device is authorized to authorize secure transaction processing. After performing the transaction-supporting operation, a second input corresponding to a request to authorize secure transaction processing is detected, and in response to detecting the second input, a second output including a haptic component is provided, wherein the second output includes the first output and provides an auxiliary output corresponding to the authorization auxiliary operation. Also in response to detecting the second input, secure transaction processing is authorized.

[0345] According to some implementations, the first operation corresponds to a save operation for saving content in an existing file, while the second operation corresponds to a save-as operation for saving the content in the existing file as a new file. In this example, a first input corresponding to a request to perform the save operation is detected, and in response to detecting the first input, a first output including a haptic component is provided. Also in response to detecting the first input, the save operation for saving content in the existing file is performed. According to some implementations, after performing the save operation, a second input corresponding to a request to perform a save-as operation for saving the content in the existing file as a new file is detected, and in response to detecting the second input, a second output including a haptic component is provided, wherein the second output includes the first output and provides an auxiliary output corresponding to creating a new file. Also in response to detecting the second input, the save-as operation is performed.

[0346] According to some implementation schemes, the first operation corresponds to a sending operation for sending a reply to a message in the message inbox, while the second operation corresponds to a sending and archiving operation for sending a reply to a message in the message inbox and deleting the message from the message inbox.

[0347] It should be understood that, Figure 12The specific order of operations described herein is merely exemplary and is not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with reference to other methods (e.g., methods 700, 900, 1000, 1100, 1300, 2500, 2600, 3000, and 3200) also apply in a similar manner to the above references. Figure 12 Method 1200 is described. For example, the inputs, operations, applications, components, and haptic outputs described above with reference to method 1200 optionally have one or more of the features of other methods described herein (e.g., methods 700, 900, 1000, 1100, 1300, 2500, 2600, 3000, and 3200) in terms of inputs, operations, applications, components, and haptic outputs. For the sake of brevity, these details are not repeated here.

[0348] Tactile output associated with the type of warning condition

[0349] According to some implementation schemes, whether two detected warning conditions are the same or different types of warning conditions provides a basis for providing corresponding tactile outputs. Providing tactile outputs associated with whether the warning conditions are the same or different types of warning conditions results in a more efficient human-machine interface, thereby reducing the amount of time the user spends performing operations, thus reducing energy consumption and increasing the battery life of battery-powered devices. Figure 13 This is an exemplary method flowchart of a method 1300 for detecting first and second alert conditions and providing corresponding outputs according to some implementation schemes. Note that in some implementations, it is possible to perform... Figure 13 Different steps beyond those shown.

[0350] According to some implementation schemes, the method begins by detecting the occurrence of the first warning condition 1305. The first warning condition can be any of the various warning conditions described elsewhere in this document.

[0351] According to some embodiments, in response to the detection of a first alert condition, 1310 provides a first output including a first tactile component and a first non-tactile component. According to various embodiments, the non-tactile component can be any audio or visual component (e.g., graphics, text). According to some embodiments, when an audio component is included, the audio and tactile components are aligned. The tactile and audio waveforms do not need to be perfectly aligned; device 100 may consider the fact that for certain types of tactile and audio outputs, even a slight temporal offset is perceived by the user as concurrent or synchronous (e.g., because audio output is processed faster than tactile output, providing tactile output before providing audio output will in some cases make the user perceive that audio and tactile outputs are occurring concurrently or synchronously). According to some embodiments, the non-tactile component includes a text component.

[0352] According to some embodiments, after providing the first output, device 100 detects the occurrence of a second warning condition 1315, and in response to detecting the occurrence of the second warning condition, provides a second output 1320 including a second tactile component and a second non-tactile component. According to some embodiments, when the first warning condition and the second warning condition are different warning conditions of the same type, the first output and the second output share one or more common components and have one or more different components. Alternatively, when the first warning condition and the second warning condition are warning conditions of the same type, the first and second outputs jointly have one or more components. According to some embodiments, when the first warning condition and the second warning condition are different warning conditions of different types, the first tactile component is different from the second tactile component, and the first non-tactile component is different from the second non-tactile component. Alternatively, when the first warning condition and the second warning condition are different types of warning conditions, the first and second outputs do not have common components.

[0353] According to some implementation plans, the types of alert conditions include messages, event prompts, media replay prompts, visual assistance prompts, system prompts, dispatch reminders, and internet browser updates.

[0354] According to some implementations, when the first warning condition and the second warning condition correspond to independent instances of the same warning condition, the first output and the second output are identical. If the computing device 100 has a touch-sensitive display, the first tactile component of the first output and the second tactile component of the second output are provided via the touch-sensitive display on the computing device.

[0355] In some embodiments, the first and second non-tactile components are audio outputs. For non-tactile outputs including audio outputs, according to some embodiments, the tactile output is accompanied by a corresponding audio output, and according to some embodiments, the audio and tactile portions are aligned when at least a portion of the audio output is concurrent with at least a portion of the tactile output, or the audio output appears temporally close to the tactile output. The tactile and audio waveforms do not need to be perfectly aligned, and device 100 may consider the fact that for certain types of tactile and audio outputs, even a slight temporal offset is perceived by the user as concurrent or synchronized (e.g., because the audio output is processed faster than the tactile output, providing the tactile output before providing the audio output will, in some cases, make the user perceive that the audio and tactile outputs are occurring concurrently or synchronously). Variations in the accompanying audio portions may differ between embodiments including audio outputs. For example, the amplitude of the audio component associated with the first output, the duration of the audio component associated with the first output, and one or more musical qualities (e.g., scale, pitch, etc.) associated with the audio component associated with the first output may differ in some cases.

[0356] In some implementations, the same tactile waveform produces different sensations when accompanied by different audio waveforms. For example, if an audio (ring) tone is played in a high-pitched tone against a low-pitched tone, and accompanied by the same tactile component (or substantially similar tactile components that a typical user would not be able to distinguish), the sensation of the tactile component will differ despite being the same tactile sensation. A higher-pitched or sharper version of the ringtone can be used as a second output to increase the user's attention. In some implementations, the first tactile output is similar to, the same as, or substantially the same as the second tactile output.

[0357] In some implementations, the first and second non-tactile components are visual outputs, such as text, graphics, or metadata outputs.

[0358] According to some implementations, the first and second alert conditions are alert conditions of the same type, where the type is associated with the alert conditions of the application. In this example, all alerts from the corresponding application fall into a common type, and the type is distinguished by the application. For example, all visual assistance events alternatively have the same haptic component containing different audio components. In other examples, incoming and outgoing messages have the same haptic output containing different audio, the same haptic output alternatively based on association with a common instant messaging application. The common component (in this example, the haptic component) identifies the application. In another example, events of a personal automated assistant, including confirm, cancel, and invoke events, alternatively all have the same audio but different haptic outputs.

[0359] Table 6: Messages

[0360] Warning conditions Main application status model condition tactile output Audio output Received message activities information No new messages X X Received message activities information No new messages X X Received message activities draft No new messages X X Received message activities Backend No new messages X X Received message activities information Breaking news tap Default message received Received message activities information Breaking news tap Default message received Received message activities draft Breaking news tap Default message received Message sent activities draft No new messages X X Message sent activities draft Breaking news tap Default message

[0361] Table 6 illustrates message receiving and sending alert conditions for an instant messaging application according to some implementations. In this example, various modes are listed for the active application state. In the case of receiving a new message, as shown in rows five, six, and seven, a tap-type haptic is output along with the message receiving a default audio. In the case of sending a new message, as shown in the last row, a tap-type haptic is output along with the message sending a default audio. Because the received message (first alert condition) and the sent message (second alert condition) are alert conditions of the same kind, where the kind is associated with the application (instant messaging), the haptic output is the same for both alert conditions, but the audio output differs. Table 6 provides conditions and corresponding outputs according to some implementations, but is merely exemplary; other combinations of alerts, states, modes, conditions, and outputs may be used according to different implementations.

[0362] According to some implementations, the first and second warning conditions are warning conditions of the same type, and the type is associated with the category of the application. According to some implementations, for example, the application categories include instant messaging applications, gaming applications, health / fitness applications, productivity applications, or social networking applications. In some implementations, applications having or performing common or similar functions optionally belong to a common category. Examples of application categories include text-based communication applications (instant messaging, email, social media), voice-based communication applications (telephone, voice internet), multimedia applications (photo browsing applications, camera applications, music players, video players), text editing applications (Word, Notepad), scheduling applications (calendar, alarm clock), and internet browser applications (Chrome, Safari). According to some implementations, the category is thus distinguished by functionality across applications. According to some implementations, as an example, by associating applications belonging to the same category (both text-based communication applications), even though email messages and text messages belonging to different applications (email and instant messaging) have the same haptic output and different audio output.

[0363] According to some implementation schemes, the first and second alert conditions are alert conditions of the same type, and the type corresponds to a predefined functional category. For example, alerts reporting the completion of activity goals in an activity monitoring application (e.g., goal approaching, after the goal, goal achieved, 10% of the goal achieved, 50% of the goal achieved, etc.) optionally fall into the same type of alert. According to some implementation schemes, various alarms or reminders within the same predefined application can be set for different predefined time instances, or for reports that meet various different predefined criteria.

[0364] Table 7: Activity Objectives

[0365]

[0366] Figure 7 The table illustrates objectives associated with various activities according to several implementations. In this example, three different alert conditions belonging to the same alert condition category are shown: Fixed End Timer, 50% Activity Target, and Activity Target Achieved. The categories of alert conditions shown correspond to functions in predefined categories of alert features that report on achieving activity targets in an activity monitoring application. For each event in the table, the haptic output is a tap-type output, and the audio output differs for each different type (e.g., default activity bell, no audio, loud activity bell). Table 7 provides conditions and corresponding outputs according to several implementations, but is merely exemplary; other combinations of alerts, statuses, modes, conditions, and outputs may be used depending on the implementation.

[0367] According to some implementations, the first and second alert conditions are associated with two different applications, wherein the first and second alert conditions fall into the same category of alert conditions, and the category corresponds to a function in a predefined category within the two different applications. Examples of functions in the predefined category, according to some implementations, include: alerts reporting incoming text communications (incoming emails, incoming instant messages, incoming social media messages) from another user across applications falling into the same category; alerts reporting incoming voice communications from another user across all applications (voice calls, voice internet, voice calls in email applications) falling into a common category; alerts reporting application software upgrades across all applications falling into a common category; and system alerts reporting various device statuses (Wi-Fi connectivity / unavailable, GPS on / off, battery life indicator, device on / off).

[0368] Table 8: Software Upgrade

[0369]

[0370] Table 8 lists various available alert conditions for software upgrades for four different application types: calendar, messaging, social networking, and games. Each of these alert conditions is associated with a different application, but in the case of alert conditions for the same category of functions corresponding to predefined categories, in which case the application software is upgraded for all applications. For each alert condition, a buzzing sound or a tactile tap is output. Some alert conditions also have non-tactile outputs. In the case of social networking or gaming applications, tactile output is supplemented by pop-up notifications. Table 8 provides conditions and corresponding outputs according to some implementations, but is merely exemplary; other combinations of alerts, statuses, modes, conditions, and outputs are used according to different implementations.

[0371] According to some implementations, when the first warning condition and the second warning condition are different warning conditions of the same kind, the first tactile component of the first output is the same (or substantially the same) as the second tactile component of the second output, and the first non-tactile component of the first output (e.g., audio or visual) is different from the second non-tactile component of the second output. Each of the examples in Tables 4, 5, and 6 above falls into this category because each of them shares a common tactile component and different non-tactile components. In some implementations, the "same" tactile component is the tactile component generated based on the same tactile output instruction sent to the tactile output generator, even if the tactile output generator (sometimes an analog component) generates slightly different tactile outputs (even if they are substantially the same to a typical user). In some implementations, audio components may share attributes but are still different. For example, the same bell or note may be played on different scales or with different instruments. Overlaying different audio on the same tactile output (e.g., high scale versus low scale, the same audio played on metal versus glass versus ceramic tones) produces different sensations. In some implementations, different tactile components are tactile components generated based on different tactile output commands sent to the tactile output generator, thereby producing different tactile outputs perceived by a typical user.

[0372] According to some implementation schemes, inverse correlation is applied to the tactile-to-nontactile similarity and the differences between their corresponding outputs. When the first warning condition and the second warning condition are different warning conditions of the same kind, the first nontactile component of the first output is the same (or substantially the same) as the second nontactile component of the second output, and the first tactile component of the first output is different from the second tactile component of the second output.

[0373] According to some embodiments, when the first alert condition and the second alert condition correspond to the same alert condition, the first output is accompanied by a first additional component that is different from the second additional component accompanying the second output. For example, both the first and second incoming instant messages are incoming instant messages and therefore have the same alert condition. According to some embodiments, in this example, the first additional component is specific to a first event that triggers the first alert condition, and the second additional component is specific to a second event that triggers the second alert condition. For example, the additional component of the first incoming instant message includes visual components (text, metadata, graphics) specific to the first instant message (e.g., sender's name, sender's contact information, message body / content). Similarly, the additional component of the second incoming instant message includes visual components (text, metadata, graphics) specific to the second instant message (e.g., sender's name, sender's contact information, message body / content). According to some embodiments, the first additional component accompanying the first output optionally provides information (e.g., text information) describing the first event that triggers the first alert (e.g., a text version of a message or automated assistance response), and the second additional component accompanying the second output provides information (e.g., text information) describing the second event that triggers the second alert.

[0374] According to some implementation schemes, after providing the second output 1320, device 100 detects the occurrence of a third warning condition 1325, and in response to detecting the occurrence of the third warning condition, provides a third output 1330 including a third tactile component and a third non-tactile component. When the first warning condition, the second warning condition, and the third warning condition are different warning conditions of the same type, the first output, the second output, and the third output optionally share one or more identical components and have one or more different components. When the third warning condition is a warning condition of a different type than the first warning condition and the second warning condition, the third tactile component optionally differs from both the first tactile component and the second tactile component, and the third non-tactile component optionally differs from both the first non-tactile component and the second non-tactile component. According to some implementation schemes, alternatively, when the first warning condition, the second warning condition, and the third warning condition are different warning conditions of the same type and the first, second, and third outputs have one or more common components (e.g., tactile, non-tactile), and when the third warning condition is a warning condition of a different type than the first and second warning conditions, the third tactile output does not have common components with the first and second outputs.

[0375] It should be understood that, Figure 13The specific order of operations described herein is merely exemplary and is not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that details of other processes described herein with reference to other methods (e.g., methods 700, 900, 1000, 1100, 1200, 2500, 2600, 3000, and 3200) also apply in a similar manner to the above references. Figure 13 Method 1300 is described herein. For example, the inputs, alert conditions, applications, and haptic outputs described above with reference to method 1300 optionally have one or more of the features of other methods described herein (e.g., methods 700, 900, 1000, 1100, 1200, 2500, 2600, 3000, and 3200) in their respective inputs, alert conditions, applications, and haptic outputs. For the sake of brevity, these details are not repeated here.

[0376] The above is for reference only. Figure 7 , Figures 9 to 13 The operation is optionally performed by Figures 1A to 1B or Figures 14 to 19 The components depicted herein are used for implementation. For example, receiving input, detecting alert conditions, determining the state, determining alert condition triggering, providing haptic and audio output, and performing operations are optionally implemented by the event classifier 170, the event recognizer 180, and the event handling subroutine 190. The event monitor 171 in the event classifier 170 detects touch on the touch-sensitive display 112, and the event dispatcher module 174 delivers the event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether a first touch at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as selecting an object on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. The event handler 190 optionally utilizes or invokes the data updater 176 or the object updater 177 to update the internal state 192 of the application. In some implementations, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will readily understand that based on... Figures 1A to 1B and Figures 14 to 19 How the components described herein can implement other processes.

[0377] As mentioned above, refer to Figure 7 , Figures 9 to 13 The operation is optionally performed by Figures 14 to 19 The components described herein shall be used for implementation. Figure 14 An exemplary functional block diagram of a device 100 configured according to the principles of various described embodiments is shown. Those skilled in the art will understand that... Figure 14 The functional blocks described herein may be optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0378] like Figure 14 As shown, device 100 includes a display unit 1401 configured to display an application; a touch-sensitive surface unit 1403 configured to receive user touch; and a processing unit 1405 coupled to the display unit 1401 and the touch-sensitive surface unit 1403. In some embodiments, the processing unit 1405 includes a detection unit 1410, a state determination unit 1415, a tactile feedback unit 1420, an engagement determination unit 1425, and an output determination unit 1430.

[0379] Processing unit 1405 is configured to detect warning conditions associated with an application running on a computing device (e.g., using detection unit 1410). Processing unit 1405 is also configured to determine a state associated with the application in response to the detection of a warning condition at a time associated with the warning condition (e.g., using state determination unit 1415). Based on the determination that the application is active at the time associated with the warning condition, processing unit 1405 provides a first tactile output representing the occurrence of the warning condition (e.g., using tactile providing unit 1420), the first tactile output having a first set of output characteristics, and based on the determination that the application is inactive at the time associated with the warning condition, provides a second tactile output representing the occurrence of the warning condition (e.g., using tactile providing unit 1420), the second tactile output having a second set of output characteristics, wherein the second tactile output differs from the first tactile output.

[0380] The following paragraphs

[0335] -

[0347] describe what can be made by Figure 14 Different implementations of the device 100 shown, either alone or in any combination.

[0381] Processing unit 1405 determines the state associated with the application at the time associated with the warning condition by determining whether the user interface of the application is displayed on the touch screen of the device at the time associated with the warning condition (e.g., by means of state determination unit 1415).

[0382] Processing unit 1405 determines the state associated with the application at the time associated with the warning condition by determining whether user input to the application triggers the warning condition (e.g., with the aid of state determination unit 1415), where the user interaction is different from the warning condition.

[0383] If the application is inactive, based on the user interaction detected at the time associated with the alert condition, the processing unit 1405 determines the level of user engagement associated with the user interaction at the time associated with the alert condition (e.g., with the help of the engagement determination unit 1425) and determines one or more of a first set of output features of the first tactile output based on the determined engagement level (e.g., with the help of the output determination unit).

[0384] Processing unit 1405 determines whether an application is active, including determining that the application is running in the foreground on the computing device at the time associated with the alert condition. Processing unit 1405 determines whether an application is inactive, including determining that the application is not running on the computing device or is running in the background on the computing device at the time associated with the alert condition.

[0385] Display unit 1401 concurrently displays the corresponding user interface window for an application on the multi-application window user interface on the device, along with one or more other user interface windows. Processing unit 1405 determines whether the corresponding user interface window for an application is displayed in the foreground of the multi-application window user interface at a time associated with an alert condition (e.g., by means of state determination unit 1415).

[0386] At the time associated with the warning condition, the display unit 1401 concurrently displays one or more other user interface windows on the device with the multi-application window user interface, and the processing unit 1405 determines the state associated with the application at the time associated with the warning condition (e.g., by means of the state determination unit 1415), including determining whether the corresponding user interface window of the application is displayed in the multi-application window user interface at the time associated with the warning condition.

[0387] In some implementations, alert conditions correspond to events that are automatically triggered by the application, automatically initiated event notifications received by the application from sources outside the device, or manually initiated event notifications received by the application from a human user other than the user operating the device.

[0388] Processing unit 1405 provides an output including a first tactile output accompanied by a first audio output and a second tactile output accompanied by a second audio output (e.g., by means of output determining unit 1430), wherein the first tactile output is the same as the second tactile output and the first audio output is different from the second audio output. Alternatively, processing unit 1405 provides an output including a first tactile output accompanied by an audio output and a second tactile output accompanied by the same audio output (e.g., by means of output determining unit 1430), wherein the first tactile output is different from the second tactile output.

[0389] Processing unit 1405 accompanies a second tactile output with an audio output. The waveform of the second tactile output is generated based on a mirror image of the waveform accompanying the audio output, and is mirrored and synchronized with the waveform of the accompanying audio output. Processing unit 1405 provides an output including a first tactile output without an audio output and a second tactile output with an audio output (e.g., using output determination unit 1430). In some embodiments, the first tactile output is similar to the second tactile output. In some embodiments, the second tactile output has a greater intensity than the first tactile output.

[0390] The first set of characteristics of the tactile output includes one or more of the following (e.g., provided by the output determining unit 1430): amplitude, duration, regularity, repetition frequency, or selection of tactile characteristics of the first tactile output. In some embodiments, the second tactile output differs from the first tactile output and has a greater intensity than the first tactile output.

[0391] Processing unit 1405 determines the state of the computing device at the time associated with the warning condition by determining that the computing device is active at the time associated with the warning condition (e.g., by means of a state determination unit), and determines that the application is inactive by determining that the computing device is inactive at the time associated with the warning condition. According to some embodiments, the first and second sets of features correspond to the device type of the computing device.

[0392] In some implementations, at a first time point, when the application is active, processing unit 1405 detects a first warning condition associated with the application (e.g., using detection unit 1410), and in response to detecting the first warning condition when the first application is active, processing unit 1404 provides (e.g., using output determination unit 1430) a first output representing the first warning condition. At a second time point, when the application is inactive, processing unit 1405 detects a second warning condition, and in response to detecting the second warning condition when the application is inactive, processing unit 1405 provides a second tactile output representing the second warning condition (e.g., using output determination unit 1440), the second tactile output being different from the first tactile output.

[0393] In some embodiments, the device has a touch-sensitive display (e.g., provided by touch-sensitive surface unit 1403) and a first tactile output and a second tactile output provided via the touch-sensitive display on the computing device (e.g., by means of output determination unit 1430). Referring to the above, Figure 7 and Figures 9 to 13 The operation is optionally performed by Figures 14 to 19 The components described are used for implementation. Figure 14An exemplary functional block diagram of a device 100 configured according to the principles of various described embodiments is shown. Those skilled in the art will understand that... Figure 14 The functional blocks described herein may be optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0394] As mentioned above, refer to Figure 7 , Figures 9 to 13 The operation is optionally performed by Figures 14 to 19 The components described herein shall be used for implementation. Figure 15 An exemplary functional block diagram of a device 100 configured according to the principles of various described embodiments is shown. Those skilled in the art will understand that... Figure 15 The functional blocks described herein may be optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0395] like Figure 15 As shown, device 100 includes a display unit 1501 configured to display an application; a touch-sensitive surface unit 1503 configured to receive user contact; and a processing unit 1505 coupled to the display unit 1501 and the touch-sensitive surface unit 1503. In some embodiments, the processing unit 1505 includes a detection unit 1510, a trigger determination unit 1515, a tactile feedback unit 1520, an external event determination unit 1525, an output variant provision unit 1530, an urgency determination unit 1535, a context determination unit 1540, a modulation unit 1545, and a personal communication determination unit 1550.

[0396] Processing unit 1505 is configured to detect the occurrence of an alert condition associated with an application (e.g., using detection unit 1510), and in response to detecting the occurrence of an alert condition, to determine whether the alert condition is triggered by a manually initiated event (e.g., using trigger detection unit 1515). If the alert condition is determined to be triggered by a manually initiated event, processing unit 1505 provides a first tactile output corresponding to the manually initiated event notification (e.g., using tactile providing unit 1520); if the alert condition is determined to be triggered by an automatically initiated event, processing unit 1505 provides a second tactile output corresponding to the automatically initiated event notification (e.g., using tactile providing unit 1520), wherein the second tactile output is different from the first tactile output.

[0397] The following paragraphs

[0353] -

[0364] describe what can be obtained from Figure 15 Different implementations of the device 100 shown, either alone or in any combination.

[0398] According to some implementations, processing unit 1505 determines that the alert condition is triggered by a manually initiated event by determining whether the alert condition corresponds to an event initiated by a human user (e.g., using trigger determination unit 1515). In some implementations, processing unit 1505 determines that the alert condition is triggered by a manually initiated event by determining whether the alert condition corresponds to an event initiated by a human user other than the user of the computing device (e.g., using trigger determination unit 1515).

[0399] According to some implementations, the processing unit 1505 determines whether the warning condition is triggered by a manually initiated event (e.g., by means of the trigger determination unit 1515) by determining whether the warning condition corresponds to input received from a human user, the input identifying the user of the computing device, and the input including instructions from the human user to the user of the computing device to warn the user of the input.

[0400] According to some implementations, processing unit 1505 determines that the warning condition is automatically triggered by the application or is an automatically initiated event notification received by the application from an external source of the device (e.g., using trigger determination unit 1515). Alternatively, processing unit 1505 determines that the warning condition is triggered by an automatically initiated event by determining that the warning condition occurs at a predetermined time or by a report that a predetermined trigger condition is met (e.g., using trigger determination unit 1515).

[0401] According to some embodiments, based on the determination that the warning condition is triggered by an automatically initiated event, processing unit 1505 determines whether the automatically initiated event corresponds to an automatically initiated event occurring outside the device (e.g., using external event determination unit 1525). According to some embodiments, based on the determination that the automatically initiated event corresponds to an event occurring outside the device, processing unit 1505 provides a first variant of the second tactile output (e.g., using output variant providing unit 1530), the first variant corresponding to a notification of an externally occurring automatically initiated event. Based on the determination that the automatically initiated event corresponds to an event initiated internally within the device, processing unit 1505 provides a second variant of the second tactile output corresponding to a notification of an internally generated automatically initiated event (e.g., using output variant providing unit 1530).

[0402] According to some embodiments, based on the determination that the warning condition is triggered by a manually initiated event, processing unit 1505 determines a measurement of urgency associated with the manually initiated event (e.g., using urgency determination unit 1535) and modulates the intensity of the first tactile output based on the measurement of urgency (e.g., using modulation unit 1545). According to some embodiments, based on the determination that the warning condition is triggered by a manually initiated event, processing unit 1505 determines one or more contextual attributes associated with the manually initiated event (e.g., using context determination unit 1540) and modulates the intensity of the first tactile output based on one or more contextual attributes (e.g., using modulation unit 1545).

[0403] According to some implementation schemes, based on the determination that the warning condition is triggered by an automatically initiated event, the processing unit 1505 analyzes the features associated with the warning condition to determine one or more contextual attributes associated with the automatically initiated event (e.g., with the aid of the context determination unit 1540) and modulates the intensity of the second tactile output based on one or more contextual attributes (e.g., with the aid of the modulation unit 1545).

[0404] According to some implementations, based on the determination of warning conditions triggered by an automatically initiated event, the processing unit 1505 determines the time change measurement of urgency associated with the automatically initiated event (e.g., by means of the urgency determination unit 1535) and modulates the intensity of a second tactile output indicating the time change measurement of urgency within a specified time window (e.g., by means of the modulation unit 154).

[0405] In some embodiments, a first tactile output is accompanied by a first audio output and a second tactile output is accompanied by a second audio output, wherein the first tactile output is the same as the second tactile output and the first audio output is different from the second audio output. In some embodiments, a first tactile output is accompanied by a first audio output and a second tactile output is accompanied by a second audio output, wherein the first audio output is similar to the second audio output and the first tactile output is different from the second tactile output. In some embodiments, a first tactile output is accompanied by a first audio output, while the second tactile output is not accompanied by an audio output. Optionally, the first tactile output is similar to the second tactile output.

[0406] According to some implementation schemes, a first tactile output corresponding to a manually initiated event notification is stronger than a second tactile output corresponding to an automatically initiated event notification. In some cases, the first tactile output is characterized by a first set of features including one or more of the following: the amplitude of the first tactile output, the duration of the first tactile output, the regularity associated with the first tactile output, the repetition frequency of tactile features in the first tactile output, and the selection of tactile features constituting the first tactile output; and the second tactile output is characterized by a second set of features including one or more of the following: the amplitude of the second tactile output, the duration of the second tactile output, the regularity associated with the second tactile output, the repetition frequency of tactile features in the second tactile output, and the selection of tactile features constituting the second tactile output.

[0407] According to some implementations, processing unit 1505 detects a first alert condition corresponding to an incoming email message from a first human sender (e.g., using detection unit 1510). In response to determining that the incoming email message corresponds to a manually initiated event, processing unit 1505 provides a first tactile output (e.g., using tactile providing unit 1520). Processing unit 1505 detects a second alert condition corresponding to an incoming text message from a second human sender (e.g., using detection unit 1510) and, in response to determining that the incoming text message corresponds to a manually initiated event, provides a first tactile output (e.g., using tactile providing unit 1520).

[0408] In some embodiments, the device has a touch-sensitive display (e.g., provided by touch-sensitive surface unit 1503) and a first tactile output and a second tactile output provided via a touch-sensitive display on a computing device (e.g., by means of tactile providing unit 1520).

[0409] According to some implementations, processing unit 1505 detects the occurrence of an alert condition associated with the application (e.g., using detection unit 1510), and in response to detecting the occurrence of the alert condition, determines whether the alert condition corresponds to personal communication from a sender in a contact list associated with the user (e.g., using personal communication determination unit 1550). Based on the determination that the alert condition corresponds to personal communication from a sender in a contact list associated with the user, processing unit 1505 provides a first tactile output corresponding to a personal alert notification (e.g., using tactile providing unit 1520), and based on the determination that the alert condition does not correspond to personal communication from a sender in a contact list associated with the user, processing unit 1505 provides a second tactile output corresponding to an automatic alert notification (e.g., using tactile providing unit 1520), wherein the amplitude of the first tactile output is greater than that of the second tactile output.

[0410] As mentioned above, refer to Figure 7 , Figures 9 to 13 The operation is optionally performed by Figures 14 to 19 The components described herein shall be used for implementation. Figure 16 An exemplary functional block diagram of a device 100 configured according to the principles of various described embodiments is shown. Those skilled in the art will understand that... Figure 16 The functional blocks described herein may be optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0411] like Figure 16 As shown, device 100 includes a display unit 1601 configured to display an application; a touch-sensitive surface unit 1603 configured to receive user contact; and a processing unit 1605 coupled to the display unit 1601 and the touch-sensitive surface unit 1603. In some embodiments, the processing unit 1605 includes a detection unit 1610, an output providing unit 1615, a modulation condition determining unit 1620, an output modification unit 1625, an input measurement unit 1630, a tactile input determining unit 1635, an output termination unit 1640, an environmental condition determining unit 1645, an output delay unit 1650, and a start providing unit 1655.

[0412] Processing unit 1605 is configured to detect a first alert condition associated with user input to a receiving application on a computing device (e.g., using detection unit 1610), and in response to detecting the first alert condition, to provide a first tactile output with a first intensity and corresponding to the user input to the application (e.g., using output unit 1615). After providing the first tactile output, processing unit 1605 detects a second alert condition associated with receiving a predetermined system event in the application (e.g., using detection unit 1610), and in response to detecting the second alert condition, to provide a second tactile output with a second intensity and corresponding to the predetermined system event (e.g., using output providing unit 1615), wherein the second intensity is greater than the first intensity.

[0413] The following paragraphs

[039] -

[0386] describe what can be made by Figure 16 Different implementations of the device 100 shown, either alone or in any combination.

[0414] In some implementations, the first haptic output is provided directly in response to user input and as feedback of the result of the user input. According to some implementations, the second alert condition corresponds to an event automatically triggered by the application, an automatically initiated event notification received by the application, or a manually initiated event notification received by the application.

[0415] According to some implementations, processing unit 1605 determines whether there is an input-based modulation condition (e.g., by means of modulation condition determination unit 1620), and in response to determining that the input-based modulation condition exists, processing unit 1605 modifies the first tactile output (e.g., by means of output modification unit 1625) before providing the first tactile output.

[0416] In some implementations, after detecting a corresponding warning condition, processing unit 1605 determines that the corresponding warning condition originates from a corresponding user input received on the computing device and a measurement identifying user participation associated with the detected user input (e.g., using participation measurement unit 1630). Processing unit 1605 modulates a first signal strength of a first tactile output based on the measurement of user participation (e.g., using output modification unit 1625).

[0417] In some implementations, after detecting a corresponding warning condition, processing unit 1605 determines that the corresponding warning condition originates from tactile user input received on the device (e.g., using tactile input determination unit 1635) and identifies the location on the device where the tactile user input was received. Processing unit 1605 modulates the spatial tactile energy distribution of the first tactile output (e.g., using output modification unit 1625) by providing a specified proportion of tactile energy greater than that of the first tactile output within a specified radius threshold of the location where the tactile user input was received on the device.

[0418] In some implementations, after detecting a corresponding warning condition, processing unit 1605 determines that the warning condition originates from receiving tactile user input on the device (e.g., using tactile input determination unit 1635), and modulates the spatial tactile energy distribution of the first tactile output (e.g., using output modification unit 1625) by reducing the amplitude of the tactile energy of the first tactile output to make it perceptible at the location where the tactile user input is received on the device. The location where the tactile user input is received on the device changes over time, and therefore the spatial tactile energy corresponding to the time-varying location where the tactile user input is received also changes over time.

[0419] In some embodiments, after detecting a corresponding warning condition, processing unit 1605 determines that the corresponding warning condition originates from received tactile user input and identifies the temporal variation morphological attributes of the tactile user input (e.g., using tactile input determination unit 1635). Processing unit 1605 modifies the morphological characteristics of the first tactile output over time to simulate the temporal variation morphological attributes of the tactile user input (e.g., using output modification unit 1625). According to some embodiments, the temporal variation morphological attributes include a time-based contact pressure distribution between the tactile user input and the touch-sensitive surface of the computing device, and modifications including modulating the energy distribution of the first tactile output over time to reflect the time-based contact pressure distribution between the tactile user input and the touch-sensitive surface of the computing device (e.g., using output modification unit 1625).

[0420] In some implementations, after detecting a corresponding warning condition, processing unit 1605 determines that the corresponding warning condition originates from continuous user input (e.g., using haptic input determination unit 1635) and, in response to the determination, initiates a first haptic output at the initiation time of the continuous user input (e.g., using output providing unit 1615). Processing unit 1605 detects the termination of the continuous user input and terminates the first haptic output at the termination time of the continuous user input (e.g., using output termination unit 1640).

[0421] Processing unit 1605 determines whether there is a modulation condition based on environmental conditions (e.g., by means of environmental condition determination unit 1645), and in response to determining that there is a modulation condition based on environmental conditions, modifies the second tactile output before providing the second tactile output (e.g., by means of output modification unit 1625).

[0422] In some implementations, after detecting a corresponding warning condition associated with a corresponding application, processing unit 1605 determines that the corresponding warning condition does not originate from receiving user input. In response to determining that the warning condition does not originate from receiving user input, processing unit 1605 determines whether one or more environmental conditions affecting the device exist as potential interference to the haptic output when the warning condition occurs (e.g., using environmental condition determination unit 1645). Based on the determination that no interfering environmental conditions exist for the device, processing unit 1605 provides a corresponding second haptic output based on the user's receiving state (e.g., using output providing unit 1615). Based on the determination that one or more environmental conditions affecting the device exist as potential interference to the haptic output, processing unit 1605 delays providing the corresponding second output to the user (e.g., using output delay unit 1650).

[0423] In some implementations, after detecting a corresponding warning condition associated with a corresponding application, processing unit 1605 determines that the corresponding warning condition does not originate from receiving user input. In response to determining that the corresponding warning condition does not originate from receiving user input, processing unit 1605 determines whether one or more environmental conditions affecting the device exist as potential interference to the haptic output when the corresponding warning condition occurs (e.g., using environmental condition determination unit 1645). Based on the determination that no interfering environmental conditions affect the device, processing unit 1605 provides the user with a first variant of the second haptic output (e.g., using output modification unit 1625), the first variant of the second haptic output having a first set of output characteristics. Based on the determination that one or more environmental conditions affecting the device exist as potential interference to the haptic output, processing unit 1605 provides the user with a second variant of the second haptic output (e.g., using output modification unit 1625), the second variant of the second haptic output having a second set of output characteristics, the second set of characteristics differing from the first set of characteristics, and the second variant of the second haptic output having a greater intensity than the first variant of the second haptic output.

[0424] In some embodiments, after detecting a corresponding warning condition associated with a corresponding application, processing unit 1605 determines that the corresponding warning condition does not originate from received user input. In response to determining that the warning condition does not originate from received user input, processing unit 1605 provides a trigger haptic output as a precursor to a second haptic output (e.g., using trigger providing unit 1655). The trigger haptic output increases the user's engagement level in operating the device to the level of the second haptic output, wherein at least a portion of the trigger haptic output is more pronounced than the second haptic output. Furthermore, processing unit 1605 provides the second haptic output after and within a specified time interval following the provision of the trigger haptic output (e.g., using output providing unit 1615). In some embodiments, the trigger haptic output includes a trigger haptic component having a time-varying waveform morphology characterized by a progressively increasing intensity of the haptic component over time. In some embodiments, the trigger haptic output includes a trigger haptic component having a time-varying waveform morphology characterized by an intensified haptic feature followed by a gradual decrease in the intensity of the haptic component.

[0425] In some implementations, in response to a corresponding alert condition derived from received user input, the second tactile output is initiated and the first tactile output is provided directly, without a corresponding prior initiation of the tactile output. In some implementations, the feature for initiating the tactile output is selected based on the urgency or context associated with the alert condition.

[0426] In some implementations, processing unit 1605 detects a first alert condition corresponding to a user selection of a user interface element displayed on an application user interface associated with the first application (e.g., using detection unit 1610). In response to the first alert condition, processing unit 1605 provides a corresponding first tactile output representing the user selection of the user interface element (e.g., using output providing unit 1615), the corresponding first tactile output including a first tactile component having a first tactile feature with a first intensity and a first duration. After providing the corresponding first tactile output, processing unit 1605 detects a second alert condition corresponding to an alert notification received by the first application (e.g., using detection unit 1610), and in response to the second alert condition, processing unit 1605 provides a corresponding second tactile output representing the receipt of the alert notification (e.g., using output providing unit 1615), the corresponding second tactile output including a second tactile component having a second tactile feature with a second intensity and a second duration, the corresponding second tactile output being different from and more pronounced than the corresponding first output based on the second tactile feature being more pronounced than the first tactile feature, the second intensity being greater than the first intensity, or the second duration being longer than the first duration.

[0427] In some implementations, processing unit 1605 detects a first alert condition associated with a first application, the first alert condition originating from a corresponding user input (e.g., using detection unit 1610), and provides a corresponding first tactile output representing the first alert condition (e.g., using output providing unit 1615). After providing the corresponding first tactile output, processing unit 1605 detects a second alert condition associated with the first application (e.g., using detection unit 1610), the second alert condition not originating from received user input, and provides a corresponding second tactile output representing the second alert condition (e.g., using output providing unit 1615), the corresponding second tactile output being different from the corresponding first output and having a greater intensity than the corresponding first output.

[0428] In some embodiments, the first tactile output has a first set of features including one or more of the following: the amplitude of the first tactile output, the duration of the first tactile output, the regularity associated with the first tactile output, the repetition frequency of tactile features in the first tactile output, and the selection of tactile features constituting the first tactile output; and the second tactile output has a second set of features including one or more of the following: the amplitude of the second tactile output, the duration of the tactile components associated with the second tactile output, the regularity associated with the second tactile output, the repetition frequency of tactile features in the second tactile output, and the selection of tactile features constituting the second tactile output.

[0429] In some embodiments, a first tactile output is accompanied by a first audio output and a second tactile output is accompanied by a second audio output, wherein the first tactile output is the same as the second tactile output and the first audio output is different from the second audio output. In some embodiments, a first tactile output is accompanied by a first audio output and a second tactile output is accompanied by a second audio output, wherein the first audio output is similar to the second audio output and the first tactile output is different from the second tactile output.

[0430] In some implementations, the first intensity of the first haptic output and the second intensity of the second haptic output are further adjusted, opposite to the volume settings on the device. According to some implementations, the second haptic output is accompanied by audio output, while the first haptic output is not accompanied by audio output.

[0431] In some embodiments, the computing device includes a touch-sensitive display for receiving user input (e.g., via touch-sensitive surface unit 1603) and a first tactile output and a second tactile output provided via the touch-sensitive display on the computing device (e.g., via output providing unit 1615).

[0432] As mentioned above, refer to Figure 7 , Figures 9 to 13 The operation is optionally performed by Figures 14 to 19 The components described herein shall be used for implementation. Figure 17 An exemplary functional block diagram of a device 100 configured according to the principles of various described embodiments is shown. Those skilled in the art will understand that... Figure 17 The functional blocks described herein may be optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0433] like Figure 17 As shown, device 100 includes a display unit 1701 configured to display an application; a touch-sensitive surface unit 1703 configured to receive user touch; and a processing unit 1705 coupled to the display unit 1701 and the touch-sensitive surface unit 1703. In some embodiments, the processing unit 1705 includes an input receiving unit 1710, an ongoing output initiation unit 1715, an ongoing output termination unit 1720, a feedback providing unit 1725, and an additional input receiving unit 1730.

[0434] Processing unit 1705 is configured to receive input corresponding to a first part of a multipart operation performed by an application running on a computing device (e.g., via input receiving unit 1710), and in response to receiving input corresponding to the first part of the multipart operation, to initiate an ongoing haptic output sequence (e.g., via ongoing output initiation unit 1715). After initiating the ongoing haptic output sequence, processing unit 1705 receives input corresponding to a second part of the multipart operation (e.g., via input receiving unit 1710), and in response to receiving input corresponding to the second part of the multipart operation, to terminate the ongoing haptic output sequence (e.g., via ongoing output termination unit 1720).

[0435] The following paragraphs

[0391] -

[0406] describe what can be made by Figure 17 Different implementations of the device 100 shown, either alone or in any combination.

[0436] In some embodiments, an input corresponding to a first part of a multi-part operation is detected on the touch-sensitive surface of the device (e.g., by means of touch-sensitive surface unit 1703) and an input corresponding to a second part of the multi-part operation is detected on the touch-sensitive surface of the device (e.g., by means of input receiving unit 1710). According to some embodiments, a tactile output sequence is provided via the touch-sensitive surface (e.g., by means of ongoing output initiation unit 1715).

[0437] In some embodiments, the device detects an input corresponding to a first part of the multipart operation (e.g., using the input receiving unit 1710) and the second device detects an input corresponding to a second part of the multipart operation. In some embodiments, the device detects both the first part of the multipart operation and the second part of the multipart operation (e.g., using the input receiving unit 1710).

[0438] According to some implementation schemes, a multi-part operation is a secure transaction processing mechanism. The first part of the multi-part operation includes equipping devices to authorize the secure transaction processing, while the second part of the multi-part operation includes authorizing the secure transaction processing. According to some implementation schemes, a multi-part operation is a secure transaction processing mechanism. The first part of the multi-part operation includes determining that user authentication is required to complete the secure transaction processing, while the second part of the multi-part operation includes receiving user authentication for the secure transaction processing.

[0439] In some implementations, the first part of the multipart operation includes closing the draft of the document, while the second part of the multipart operation includes returning to the draft of the document. In some implementations, the input corresponding to the first part of the multipart operation is a first user input interacting with an application running on the device, while the input corresponding to the second part of the multipart operation is a second user input interacting with the application, the second user input being different from the first user input.

[0440] According to some embodiments, detecting input corresponding to a first part of a multi-part operation includes detecting contact on a touch-sensitive surface (e.g., using input receiving unit 1710), and detecting input corresponding to a second part of a multi-part operation includes detecting movement of the contact on the touch-sensitive surface (e.g., using input receiving unit 1710). In some embodiments, detecting input corresponding to a first part of a multi-part operation includes detecting movement of the contact on the touch-sensitive surface (e.g., using input receiving unit 1710), and detecting input corresponding to a second part of a multi-part operation includes detecting lifting of the contact off the touch-sensitive surface (e.g., using input receiving unit 1710).

[0441] In some embodiments, the input corresponding to the first and second parts of the multipart operation includes a single gesture, wherein the input corresponding to the first part of the multipart operation is the initial part of the gesture and the input corresponding to the second part of the multipart operation is the subsequent part of the gesture. According to some embodiments, the gesture is initiated on a portion of the touch-sensitive surface corresponding to a first location in the user interface displayed on the device's display and terminated on a second portion of the touch-sensitive surface corresponding to a second location in the user interface different from the first location (e.g., by means of touch-sensitive surface unit 1703).

[0442] In some implementations, an input corresponding to the first part of a multipart operation initiates an event associated with the corresponding application, the event continues until an input corresponding to the second part of the multipart operation is received, and the input corresponding to the second part of the multipart operation terminates the event associated with the corresponding application (e.g., by means of the ongoing output termination unit 1720).

[0443] According to some embodiments, the application is a text editing application, and the input corresponding to the first part of the multi-part operation is the selection of a first user interface element that enables a text input mode for the text editing application. According to some embodiments, the multi-part operation includes one or more inputs for text input to the text editing application and events corresponding to text manipulations in the text editing document in response to the inputs for the text inputs. The input corresponding to the second part of the multi-part operation is the selection of a user interface element that disables a text input mode for the text editing application.

[0444] In some implementations, the input corresponding to the first part of a multipart operation is a selection of a first user interface for initiating a transaction, while the multipart operation includes one or more inputs for the input information required to process the transaction. In some implementations, the event corresponds to filling one or more data entry fields that receive the information required to process the transaction, and the input corresponding to the second part of the multipart operation is a selection of a user interface element for authorizing the completion of the transaction.

[0445] In some implementations, the input corresponding to the second part of the multipart operation follows the input corresponding to the first part of the multipart operation and is time-spaced from the input corresponding to the first part of the multipart operation by a specified time interval corresponding to the duration of the event, and an ongoing haptic output sequence is provided throughout the duration of the event and indicates the occurrence of the event. The characteristics of the ongoing haptic output sequence are optionally selected based on the event initiated by the input corresponding to the first part of the multipart operation.

[0446] In some embodiments, processing unit 1705 provides first tactile feedback (e.g., via feedback providing unit 1725) in response to receiving input corresponding to a first portion of a multi-part operation. This first tactile feedback is based on event selection and indicates the initiation of an event, and is distinct from the ongoing tactile output sequence. Processing unit 1705 also provides second tactile feedback (e.g., via feedback providing unit 1725) in response to receiving input corresponding to a second portion of the multi-part operation. This second tactile feedback is distinct from the ongoing tactile output sequence.

[0447] According to some implementation schemes, input corresponding to the first part of a multipart operation suspends ongoing events associated with the application, while input corresponding to the second part of the multipart operation resumes ongoing events associated with the application.

[0448] In some embodiments, the application is a voice communication application, and the event is an ongoing voice communication that occurs upon receiving input corresponding to the first part of a multi-part operation. In some embodiments, the input corresponding to the first part of the multi-part operation is a selection of a first user interface element of the voice communication application that pauses the ongoing voice communication, and the input corresponding to the second part of the multi-part operation is a selection of a second user interface element that resumes the voice communication.

[0449] In some embodiments, after initiating an ongoing haptic output sequence in response to a first part of the multipart operation and before receiving input corresponding to a second part of the multipart operation, processing unit 1705 receives one or more additional user inputs (e.g., via additional input receiving unit 1730) that are different from the input corresponding to the second part of the multipart operation. Processing unit 1705 continues to provide ongoing haptic output regardless of the additional user inputs (e.g., via ongoing output initiation unit 1715) until the input corresponding to the second part of the multipart operation is received. In some embodiments, the intensity of the ongoing haptic output sequence increases over time until a subsequent input corresponding to the multipart operation is received.

[0450] According to some implementations, in response to detecting an input corresponding to the first part of the multi-part operation (e.g., by means of the input receiving unit 1710), the processing unit 1705 initiates an ongoing audio output sequence to accompany the ongoing tactile sequence (e.g., by means of the ongoing output initiation unit 1715), and in response to detecting an input corresponding to the second part of the multi-part operation (e.g., by means of the input receiving unit 1710), terminates the ongoing audio output sequence (e.g., by means of the ongoing output termination unit 1720).

[0451] In some implementations, the time-varying haptic energy distribution of the haptic output sequence simulates the time-varying acoustic energy distribution of the audio output sequence. In some cases, in response to determining that the input corresponding to the first part of a multi-part operation does not pause an ongoing event but initiates an event associated with the corresponding application, processing unit 1705 provides a haptic output sequence without an accompanying audio output sequence (e.g., using the ongoing output initiation unit 1715), and in response to determining that the input corresponding to the first part of a multi-part operation pauses an ongoing event associated with the corresponding application, processing unit 1705 provides an ongoing audio output sequence accompanying the ongoing haptic output sequence (e.g., using the ongoing output initiation unit 1715).

[0452] As mentioned above, refer to Figure 7 , Figures 9 to 13 The operation is optionally performed by Figures 14 to 19 The components described herein shall be used for implementation. Figure 18 An exemplary functional block diagram of a device 100 configured according to the principles of various described embodiments is shown. Those skilled in the art will understand that... Figure 18 The functional blocks described herein may be optionally combined or separated into sub-blocks to implement the principles of the various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0453] like Figure 18 As shown, device 100 includes a display unit 1801 configured to display an application; a touch-sensitive surface unit 1803 configured to receive user contact; and a processing unit 1805 coupled to the display unit 1801 and the touch-sensitive surface unit 1803. In some embodiments, the processing unit 1805 includes an input detection unit 1810, an output providing unit 1815, an operation execution unit 1820, an image capture unit 1825, a secure transaction processing authorization unit 1830, and a save operation execution unit 1835.

[0454] Processing unit 1805 is configured to detect a first input corresponding to a request to perform a first operation (e.g., using input detection unit 1810) and, in response to detecting the first input, to provide a first output including a tactile component (e.g., using output providing unit 1815). Also in response to detecting the first input, processing unit 1805 is configured to perform the first operation (e.g., using operation execution unit 1820). After performing the first operation, processing unit 1805 is configured to detect a second input corresponding to a request to perform a second operation including the first operation and an auxiliary operation (e.g., using input detection unit 1810), and, in response to detecting the second input, to provide a second output including a tactile component (e.g., using output providing unit 1815), wherein the second output includes the first output and provides an auxiliary output corresponding to the auxiliary operation. Also in response to detecting the second input, processing unit 1805 is configured to perform the second operation (e.g., using operation execution unit 1820).

[0455] The following paragraphs

[0411] -

[0420] describe what can be made by Figure 18 Different implementations of the device 100 shown, either alone or in any combination.

[0456] In some implementations, in response to the detection of a second input, the processing unit 1805 performs a first operation and provides a first output, and performs an auxiliary operation and provides an auxiliary output (e.g., by means of the operation execution unit 1820 and the output providing unit 1815).

[0457] In some embodiments, in response to detecting a second input, processing unit 1805 performs a first operation (e.g., using operation execution unit 1820) before performing an auxiliary operation, and provides a first output (e.g., using output providing unit 1815) before providing an auxiliary output. In some embodiments, in response to detecting a second input, processing unit 1805 performs a first operation (e.g., using operation execution unit 1820) after performing an auxiliary operation, and provides a first output (e.g., using output providing unit 1815) after providing an auxiliary output.

[0458] In some implementations, in response to the detection of a second input, the processing unit 1805 concurrently executes at least a portion of the first operation with a portion of the auxiliary operation (e.g., by means of the operation execution unit 1820), and concurrently provides at least a portion of the first output with a portion of the auxiliary output (e.g., by means of the output providing unit 1815).

[0459] In some implementations, the first operation corresponds to capturing an image using a camera, while the second operation corresponds to capturing an image after a specified time interval. In some implementations, the first operation corresponds to a transaction-supporting operation that enables the device to authorize secure transaction processing, while the second operation corresponds to enabling the device to authorize secure transaction processing and authorizing secure transaction processing.

[0460] According to some implementations, the first operation corresponds to a save operation for saving content in an existing file, while the second operation corresponds to a save-as operation for saving the content in the existing file as a new file. According to some implementations, the first operation corresponds to a send operation for sending a reply to a message in the message inbox, while the second operation corresponds to a send and archive operation for sending a reply to a message in the message inbox and deleting the message from the message inbox. In some implementations, auxiliary operations have variable values, and auxiliary outputs have values ​​based on variable attributes.

[0461] In some embodiments, the tactile component corresponding to the second output includes a first tactile component corresponding to the first output and a second tactile component corresponding to the auxiliary output, the second tactile component being different from the first tactile component. In some cases, the auxiliary output includes non-tactile components. In some embodiments, the second output includes a text component that identifies the auxiliary operation as different from the first operation. And in some cases, the auxiliary output includes an audio component while the first output does not include an audio component.

[0462] According to some embodiments, the computing device includes a touch-sensitive display and receives first and second inputs via the touch-sensitive display (e.g., by means of a touch-sensitive surface unit 1803). In some embodiments, the tactile components of the first output and the second output are provided via the touch-sensitive display (e.g., by means of an output providing unit 1815).

[0463] In some implementations, processing unit 1805 detects a first input corresponding to a request to capture a first image using a camera (e.g., using input detection unit 1810), and in response to detecting the first input, processing unit 1805 provides a first output including a tactile component (e.g., using output providing unit 1815). Also in response to detecting the first input, processing unit 1805 captures the first image (e.g., using image capture unit 1825), and after capturing the first image, detects a second input corresponding to a request to capture a second image after a specified time interval (e.g., using input detection unit 1810). In response to detecting the second input, processing unit 1805 provides a second output including a tactile component (e.g., using output providing unit 1815), wherein the second output includes the first output and provides an auxiliary output corresponding to the measurement of the specified time interval expiring, and processing unit 1805 captures the second image after the specified time interval (e.g., using image capture unit 1825).

[0464] In some implementations, processing unit 1805 detects a first input corresponding to a request to perform a supporting transaction processing operation that enables the device to authorize secure transaction processing (e.g., using input detection unit 1810) and, in response to detecting the first input, provides a first output including a tactile component (e.g., using output providing unit 1815). Also in response to detecting the first input, processing unit 1805 enables the device to authorize secure transaction processing (e.g., using secure transaction processing authorization unit 1830). After performing the supporting transaction processing operation, processing unit 1805 detects a second input corresponding to a request to authorize secure transaction processing (e.g., using input detection unit 1810) and, in response to detecting the second input, provides a second output including a tactile component (e.g., using output providing unit 1815), wherein the second output includes the first output and provides an auxiliary output corresponding to the authorized secure transaction processing. Also in response to detecting the second input, processing unit 1805 authorizes secure transaction processing (e.g., using secure transaction processing authorization unit 1830).

[0465] In some implementations, processing unit 1805 detects a first input corresponding to a request to perform a save operation (e.g., using input detection unit 1810) and, in response to detecting the first input, provides a first output including a tactile component (e.g., using output providing unit 1815). Also in response to detecting the first input, processing unit 1805 performs a save operation to save content to an existing file (e.g., using save operation execution unit 1835), and after performing the save operation, processing unit 1805 detects a second input corresponding to a request to perform a save-as operation to save content from an existing file to a new file (e.g., using input detection unit 1810), and, in response to detecting the second input, provides a second output including a tactile component (e.g., using output providing unit 1815), wherein the second output includes the first output and provides an auxiliary output corresponding to the creation of a new file. Also in response to det...

Claims

1. A processor-implemented method executed on a computing device, the computing device including one or more input devices, the method comprising: A first alert condition is detected on the computing device, the first alert condition being associated with receiving user input, wherein the user input is input detected at one or more input devices of the computing device; as well as In response to detecting the first warning condition, and in association with user input received at the input device of the computing device based on the first warning condition, a first haptic output is provided on the computing device, the first haptic output having a first amplitude and corresponding to the user input on the computing device; After providing the first tactile output, a second alert condition is detected, which is associated with an automatically triggered notification in a set of predetermined automatically triggered notifications at the computing device; as well as In response to detecting the second alert condition, and in association with receiving an automatically triggered notification from the set of predetermined automatically triggered notifications based on the second alert condition, a second tactile output is provided on the computing device, the second tactile output having a second amplitude and corresponding to the automatically triggered notification from the set of predetermined automatically triggered notifications, wherein the second amplitude is greater than the first amplitude.

2. The method of claim 1, wherein the first tactile output is provided directly in response to the user input and as a result of the user input.

3. The method according to claim 1, wherein the set of predetermined automatically triggered notifications includes: Alerts automatically generated by applications on the computing device, and automatically initiated event notifications received by the applications.

4. The method according to any one of claims 1 to 3, further comprising: Determine whether there exists an input-based modulation condition corresponding to the user input; as well as In response to determining the existence of input-based modulation conditions, the first haptic output is modified before providing the first haptic output.

5. The method according to any one of claims 1 to 3, further comprising: Detect the corresponding warning conditions; The corresponding alert condition is determined to originate from the corresponding user input received on the computing device; Identify user-engaged measurements performed with the computing device or an application on the computing device, wherein the user-engaged measurements are associated with the received user input; as well as The intensity of the first signal of the first tactile output is modulated based on the measurement involving the user.

6. The method according to any one of claims 1 to 3, further comprising: Detect the corresponding warning conditions; It is determined that the corresponding warning condition originates from the user input received on the computing device; Identify the location where the user input is received on the computing device; as well as The spatial tactile energy distribution of the first tactile output is modulated by providing tactile energy greater than a specified proportion of the tactile energy of the first tactile output within a specified radius threshold of the location input by the user on the computing device.

7. The method according to any one of claims 1 to 3, further comprising: Detect the corresponding warning conditions; It is determined that the corresponding warning condition originates from user input received on the computing device; as well as By reducing the amplitude of the tactile energy of the first tactile output to make the position of the first tactile output on the computing device perceptible, the spatial tactile energy distribution of the first tactile output is modulated.

8. The method according to claim 7, wherein: The location, as input by the user, changes over time, received on the computing device; and The spatial tactile energy distribution of the first tactile output changes over time corresponding to the time-varying position at which the user input is received.

9. The method according to any one of claims 1 to 3, further comprising: Detect the corresponding warning conditions; It is determined that the corresponding warning condition originates from received user input; Identify the time-varying morphological attributes of the user input; as well as The morphological features of the first tactile output are modified over time to correspond to the time-varying morphological properties of the user input.

10. The method according to claim 9, wherein: The time-varying morphological properties include the time-based contact pressure distribution between the user input and the touch-sensitive surface of the computing device; and The modification includes modulating the energy distribution of the first tactile output over time to correspond to the time-based contact pressure distribution between the user input and the touch-sensitive surface of the computing device.

11. The method according to any one of claims 1 to 3, further comprising: Detect the corresponding warning conditions; It is determined that the corresponding alert conditions originate from continuous user input; In response to the determination, the first haptic output is initiated at the time of the continuous user input. Detect the termination of the continuous user input; as well as The first haptic output is terminated at the end time of the continuous user input.

12. The method according to any one of claims 1 to 3, further comprising: Determine whether there are modulation conditions based on environmental conditions; as well as In response to determining the presence of modulation conditions based on environmental conditions, the second tactile output is modified before providing the second tactile output.

13. The method according to any one of claims 1 to 3, further comprising: Detect the relevant warning conditions associated with the corresponding application; It is determined that the corresponding warning condition does not originate from receiving user input at one or more input devices of the computing device; as well as In response to determining that the corresponding warning condition does not originate from user input received at one or more input devices of the computing device: Determine whether one or more environmental conditions, for the computing device, exist at the time the corresponding warning condition occurs, as a potential interference with the tactile output; Based on the determination that there are no potential interferences to the tactile output from the environmental conditions of the computing device, the second tactile output is provided based on the user's reception state. Based on one or more environmental conditions of the computing device that are determined to be potential interference with the tactile output, the provision of the second tactile output to the user is delayed.

14. The method according to any one of claims 1 to 3, further comprising: Detect the relevant warning conditions associated with the corresponding application; It is determined that the corresponding warning condition does not originate from receiving user input at one or more input devices of the computing device; as well as In response to determining that the corresponding warning condition does not originate from user input received at one or more input devices of the computing device: Determine whether one or more environmental conditions, specifically for the computing device, exist at the time the corresponding warning condition occurs, as potential interference with tactile output. Based on environmental conditions for the computing device that determine there are no potential interferences to the tactile output, a first variant of the second tactile output is provided to the user, the first variant of the second tactile output having a first set of output features, and Based on one or more environmental conditions for the computing device that are determined to be potential interference with the tactile output, a second variant of the second tactile output is provided to the user, the second variant of the second tactile output having a second set of output features that are different from the first set of output features, and the second variant of the second tactile output having a greater intensity than the first variant of the second tactile output.

15. The method according to any one of claims 1 to 3, further comprising: Detect the relevant warning conditions associated with the corresponding application; It is determined that the corresponding warning condition does not originate from receiving user input at one or more input devices of the computing device; as well as In response to determining that the corresponding warning condition does not originate from user input received at one or more input devices of the computing device: A haptic trigger output is provided as a precursor to the second haptic output, the haptic trigger output increasing the level of user engagement in operating the computing device prior to the second haptic output, wherein at least a portion of the haptic trigger output is more noticeable than the second haptic output; as well as The second tactile output is provided during and after the specified time interval in which the initial tactile output is provided.

16. The method of claim 15, wherein the activation haptic output includes a haptic component having a time-varying waveform shape, the time-varying waveform shape being characterized by the increasing intensity of the haptic component over time.

17. The method of claim 15, wherein the activated haptic output includes a haptic component having a time-varying waveform morphology, the time-varying waveform morphology being characterized by an enhanced haptic feature of gradually decreasing intensity following the haptic component.

18. The method of claim 15, wherein: In response to a corresponding alert condition derived from the received user input, the first haptic output is provided directly in the absence of a corresponding previous haptic output.

19. The method of claim 15, wherein the feature for activating the haptic output is selected based on the urgency or context associated with the corresponding warning condition.

20. The method according to any one of claims 1 to 3, wherein: The first warning condition corresponds to a user selection of a user interface element displayed on the application user interface, which is associated with a first application on the computing device; The first tactile output includes a first tactile component, the first tactile component includes a first tactile feature, and the first tactile feature includes a first intensity and a first duration; The second warning condition corresponds to the warning notification received by the first application; The second tactile output represents the receipt of the warning notification. The second tactile output includes a second tactile component, the second tactile component includes a second tactile feature, and the second tactile feature includes a second intensity and a second duration. and The second tactile output is different from the first tactile output and is more obvious than the first tactile output, based on the fact that the second tactile feature is more obvious than the first tactile feature, the second intensity is greater than the first intensity, or the second duration is longer than the first duration.

21. The method according to claim 1, wherein: The first tactile output represents the first warning condition; and The second tactile output represents the second warning condition, and the second tactile output is different from the first tactile output and has a greater intensity than the first tactile output.

22. The method according to any one of claims 1 to 3, wherein: The first tactile output has a first set of features, which includes one or more of the following: the amplitude of the first tactile output, the duration of the first tactile output, the regularity associated with the first tactile output, the repetition frequency of tactile features in the first tactile output, and the selection of tactile features included in the first tactile output; and The second tactile output has a second set of features, which includes one or more of the following: the amplitude of the second tactile output, the duration of the tactile components associated with the second tactile output, the regularity associated with the second tactile output, the repetition frequency of the tactile features in the second tactile output, and the selection of tactile features included in the second tactile output.

23. The method according to any one of claims 1 to 3, wherein: The first tactile output is accompanied by the first audio output; The second tactile output is accompanied by a second audio output; and The first audio output is different from the second audio output.

24. The method according to any one of claims 1 to 3, wherein: The first tactile output is accompanied by the first audio output; The second tactile output is accompanied by a second audio output; The first audio output is similar to the second audio output; and The first tactile output is different from the second tactile output.

25. The method of claim 24, wherein the first amplitude of the first haptic output and the second amplitude of the second haptic output are further adjusted in contrast to the volume settings on the computing device.

26. The method according to any one of claims 1 to 3, wherein: The second tactile output is accompanied by audio output; and The first tactile output was not accompanied by audio output.

27. The method according to any one of claims 1 to 3, wherein: The one or more input devices of the computing device include a touch-sensitive display for receiving the user input; and The first tactile output and the second tactile output are provided via the touch-sensitive display on the computing device.

28. A computer-readable storage medium storing executable code, which, when executed by a processor of a computing device, causes the computing device to perform operations, the computing device including an output generator and one or more input devices, the operations including: A first alert condition is detected on the computing device, the first alert condition being associated with receiving user input, wherein the user input is input detected at one or more input devices of the computing device; as well as In response to detecting the first warning condition, and in association with user input received at the input device of the computing device based on the first warning condition, a first haptic output is provided on the computing device, the first haptic output having a first amplitude and corresponding to the user input on the computing device; After providing the first tactile output, a second alert condition is detected, which is associated with an automatically triggered notification in a set of predetermined automatically triggered notifications at the computing device; as well as In response to detecting the second alert condition, and in association with receiving an automatically triggered notification from the set of predetermined automatically triggered notifications based on the second alert condition, a second tactile output is provided on the computing device, the second tactile output having a second amplitude and corresponding to the automatically triggered notification from the set of predetermined automatically triggered notifications, wherein the second amplitude is greater than the first amplitude.

29. The computer-readable storage medium of claim 28, wherein the executable code includes instructions that, when executed by the processor of the computing device, cause the computing device to perform the method of any one of claims 2 to 27.

30. A computing device, comprising: Output generator; One or more input devices, including a touch-sensitive surface; as well as A memory that stores one or more programs, said one or more programs including instructions for: A first alert condition is detected on the computing device, the first alert condition being associated with receiving user input on the computing device, wherein the user input is input detected at one or more input devices of the computing device; as well as In response to detecting the first warning condition, and in association with user input received at the input device of the computing device based on the first warning condition, a first haptic output is provided on the computing device, the first haptic output having a first amplitude and corresponding to the user input on the computing device; After providing the first tactile output, a second alert condition is detected, which is associated with an automatically triggered notification in a set of predetermined automatically triggered notifications at the computing device; as well as In response to detecting the second alert condition, and in association with receiving an automatically triggered notification from the set of predetermined automatically triggered notifications based on the second alert condition, a second tactile output is provided on the computing device, the second tactile output having a second amplitude and corresponding to the automatically triggered notification from the set of predetermined automatically triggered notifications, wherein the second amplitude is greater than the first amplitude.

31. The computing device of claim 30, wherein the one or more programs include instructions for performing the method of any one of claims 2 to 27.

Images