Device, method, and user interface for providing voice notifications

Abstract

The present invention provides an improved method, device, and information processing device for providing voice notifications. [Solution] An electronic device communicating with one or more wearable audio output devices detects the occurrence of one or more first events while the wearable audio output devices are being worn by a user. In response, the electronic device outputs audio content corresponding to the first events via the wearable audio output devices. After outputting the audio content corresponding to the first events, the electronic device detects the movement of the wearable audio output devices and, in accordance with the detection of movement and the determination that the first movement of the wearable audio output devices satisfies a first movement criterion, outputs additional audio content corresponding to one or more events via the wearable audio output devices.

Description

[Technical Field]

[0001] This generally relates to electronic devices that have an audio output device, such as a wearable audio output device, including but not limited to electronic devices that provide voice notifications via an audio output device. [Background technology]

[0002] Voice notifications are widely used to communicate information to users in language. However, existing methods for providing voice notifications are cumbersome, inefficient, and limited. In some cases, existing methods require multiple inputs to generate or respond to a voice notification. In some cases, existing methods provide too little or too much information at once. In some cases, existing methods provide voice notifications in an intrusive and distracting manner. In addition, existing methods waste energy because they take more time than necessary and require user interaction. The latter problem is particularly critical in battery-powered devices. [Overview of the project]

[0003] Therefore, there is a need for electronic devices with improved methods and interfaces for providing voice notifications. Such methods and interfaces optionally complement or replace existing methods for providing voice notifications. Such methods and interfaces reduce the number, range, and / or nature of user input, creating a more efficient human-machine interface. With respect to battery-operated devices, such methods and interfaces conserve power and extend the time between battery charges.

[0004] The drawbacks and other problems associated with providing voice notifications described above are mitigated or eliminated by the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the device is a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the device has (and / or communicates with) a touchpad. In some embodiments, the device has (and / or communicates with) a touch-sensitive display (also known as a “touchscreen” or “touchscreen display”). In some embodiments, the device has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in memory for performing multiple functions. In some embodiments, the user interacts with the GUI in part through stylus and / or finger touch and gestures on the touch-sensitive surface. In some embodiments, the functions optionally include gameplay, image editing, drawing, presentation, word processing, spreadsheet creation, telephone, video conferencing, email, instant messaging, training support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. The executable instructions for performing those functions optionally include non-temporary computer-readable storage media or other computer program products configured to be executed by one or more processors.

[0005] According to some embodiments, the method is performed in an electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of one or more wearable audio output devices. The method includes detecting the occurrence of multiple events without outputting notifications corresponding to the events via one or more audio output devices while one or more wearable audio output devices are not in their respective positions relative to the user's ears. After detecting the occurrence of multiple events, the method includes detecting via one or more sensors that one or more wearable audio output devices are in their respective positions relative to the user's ears. In response to one or more wearable audio output devices being in their respective positions relative to the user's ears, the method includes outputting information via one or more wearable audio output devices, including one or more audio notifications corresponding to any one or more of the multiple events.

[0006] According to some embodiments, the method is performed in an electronic device having one or more audio output devices. The method includes detecting the occurrence of an event and, in response to detecting the occurrence of the event, outputting a voice notification via one or more audio output devices that includes audio content representing the event. According to a determination that the event is associated with a first characteristic of a first set of characteristics, the voice notification includes audio content representing the event and has a first audio output characteristic selected based on the first characteristic. According to a determination that the event is associated with a second characteristic of the first set of characteristics that is different from the first characteristic, the voice notification includes audio content representing the event and has a second audio output characteristic that is different from the first output characteristic, selected based on the second characteristic.

[0007] According to some embodiments, the method is performed in an electronic device that communicates with one or more wearable audio output devices and with one or more sensors for detecting the placement of one or more wearable audio output devices. The method includes detecting the occurrence of a first event and, after detecting the occurrence of the first event, outputting one or more audio notifications corresponding to the first event via one or more wearable audio output devices. The method includes detecting an input directed to one or more wearable audio output devices after starting to output one or more audio notifications. The method includes, in response to detecting an input directed to one or more wearable audio output devices, performing a first action according to a determination that an input was detected within a predetermined time period with respect to one or more audio notifications corresponding to the first event, wherein the first action is associated with one or more audio notifications corresponding to the first event, and performing a second action according to a determination that an input was detected after a predetermined time period has elapsed with respect to one or more audio notifications corresponding to the first event, wherein the second action is not associated with one or more audio notifications corresponding to the first event.

[0008] According to some embodiments, the method is performed in an electronic device communicating with one or more wearable audio output devices. The method includes detecting the occurrence of a first event and, after detecting the occurrence of the first event, outputting one or more audio notifications corresponding to the first event via one or more wearable audio output devices. The method includes, after outputting one or more audio notifications corresponding to the first event, outputting one or more separate audio outputs indicating that the electronic device is ready to receive a response to the first event. While outputting each audio output, the method includes detecting an input corresponding to a response to the first event and generating a response to the first event based on the input.

[0009] According to some embodiments, the method is performed in an electronic device communicating with one or more wearable audio output devices. The method includes detecting the occurrence of a first event while one or more wearable audio output devices are being worn by a user. In response to detecting the occurrence of the first event, the method includes outputting one or more audio notifications corresponding to the first event via one or more wearable audio output devices, which includes delaying the output of the one or more audio notifications corresponding to the first event until the conversation ends, according to a determination that the user of the electronic device is currently engrossed in a conversation, and outputting one or more audio notifications corresponding to the first event without delay, according to a determination that the user of the electronic device is not currently engrossed in a conversation.

[0010] According to some embodiments, the electronic device includes (and / or communicates with) one or more audio output devices (e.g., one or more wearable audio output devices), optionally one or more sensors for detecting the placement of the audio output devices, optionally a display and / or touch-sensitive surface, optionally one or more accelerometers and / or attitude sensors, optionally one or more additional input devices (e.g., a rotatable input mechanism), one or more processors, and one or more memory for storing (and / or communicating with) one or more programs, wherein one or more programs are configured to be executed by one or more processors, and one or more programs include instructions to perform or cause to perform any of the operations described herein. According to some embodiments, a computer-readable storage medium stores instructions that, when executed by an electronic device including (and / or communicating with) one or more audio output devices, optionally one or more sensors for detecting the placement of the audio output devices, optionally a display and / or touch-sensitive surface, optionally one or more accelerometers and / or attitude sensors, and optionally one or more additional input devices, cause the device to perform any of the operations described herein. According to some embodiments, a graphical user interface on a device including (and / or communicating with) one or more audio output devices, optionally one or more sensors for detecting the placement of the audio output devices, optionally a display and / or touch-sensitive surface, optionally one or more accelerometers and / or attitude sensors, and optionally one or more additional input devices, memory, and one or more processors for executing one or more programs stored in memory, includes any one or more elements displayed in any of the methods described herein, which are updated in response to input as described in any of the methods described herein.According to some embodiments, the electronic device includes (and / or communicates with) one or more audio output devices, optionally one or more sensors for detecting the placement of the audio output devices, optionally a display and / or touch-sensitive surface, optionally one or more accelerometers and / or attitude sensors, optionally one or more additional input devices, and means for performing or causing to perform any of the operations described herein. According to some embodiments, the information processing device for use with the electronic device including (and / or communicating with) one or more audio output devices, optionally one or more sensors for detecting the placement of the audio output devices, optionally a display and / or touch-sensitive surface, optionally one or more accelerometers and / or attitude sensors, and optionally one or more additional input devices, includes means for performing or causing to perform any of the operations described herein.

[0011] Accordingly, an electronic device having (and / or communicating with) one or more audio output devices, optionally one or more sensors for detecting the placement of the audio output devices, optionally a display and / or touch-sensitive surface, optionally one or more accelerometers and / or attitude sensors, and optionally one or more additional input devices, is provided with improved methods and interfaces for providing voice notifications, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can complement or replace existing methods for providing voice notifications. [Brief explanation of the drawing]

[0012] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings, and similar reference numbers throughout the following drawings refer to the corresponding parts.

[0013] [Figure 1A]A block diagram showing a portable multifunctional device having a touch sensing display, according to some embodiments.

[0014] [Figure 1B] A block diagram showing exemplary components for event processing, according to some embodiments.

[0015] [Figure 2] A portable multifunctional device having a touch screen is shown, according to some embodiments.

[0016] [Figure 3A] A block diagram of an exemplary multifunctional device having a display and a touch sensing surface, according to some embodiments.

[0017] [Figure 3B] A block diagram of an exemplary wearable audio output device, according to some embodiments.

[0018] [Figure 4A] An exemplary user interface for a menu of an application on a portable multifunctional device is shown, according to some embodiments.

[0019] [Figure 4B] An exemplary user interface for a multifunctional device having a touch sensing surface separated from a display is shown, according to some embodiments.

[0020] [Figure 4C] Examples of dynamic intensity thresholds are shown, according to some embodiments. [Figure 4D] Examples of dynamic intensity thresholds are shown, according to some embodiments. [Figure 4E] Examples of dynamic intensity thresholds are shown, according to some embodiments.

[0021] [Figure 5A] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5B] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5C] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5D] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5E] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5F] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5G] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5H] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5I] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5J] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5K] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5L] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5M]This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5N] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5O] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5P] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5Q] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5R] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5S] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5T] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5U] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5V] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5W] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5X] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5Y] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5Z] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5AA] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5AB] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5AC] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. [Figure 5AD] This document illustrates exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments.

[0022] [Figure 6A] This is a flowchart illustrating a process for providing voice notifications to an audio output device worn by a user, according to several embodiments. [Figure 6B] This is a flowchart illustrating a process for providing voice notifications to an audio output device worn by a user, according to several embodiments. [Figure 6C] This is a flowchart illustrating a process for providing voice notifications to an audio output device worn by a user, according to several embodiments. [Figure 6D] This is a flowchart illustrating a process for providing voice notifications to an audio output device worn by a user, according to several embodiments. [Figure 6E] This is a flowchart illustrating a process for providing voice notifications to an audio output device worn by a user, according to several embodiments.

[0023] [Figure 7A] This is a flowchart of a process that provides an audio context for conveying additional information about an event, according to several embodiments. [Figure 7B] This is a flowchart of a process that provides an audio context for conveying additional information about an event, according to several embodiments. [Figure 7C] This is a flowchart of a process that provides an audio context for conveying additional information about an event, according to several embodiments.

[0024] [Figure 8A] This is a flowchart of a process that performs an action based on the timing of user input in response to a voice notification, according to several embodiments. [Figure 8B] This is a flowchart of a process that performs an action based on the timing of user input in response to a voice notification, according to several embodiments. [Figure 8C] This is a flowchart of a process that performs an action based on the timing of user input in response to a voice notification, according to several embodiments. [Figure 8D] This is a flowchart of a process that performs an action based on the timing of user input in response to a voice notification, according to several embodiments.

[0025] [Figure 9A] This is a flowchart illustrating a process of stimulating a user and receiving a response to an audio notification, according to several embodiments. [Figure 9B] This is a flowchart illustrating a process of stimulating a user and receiving a response to an audio notification, according to several embodiments. [Figure 9C] This is a flowchart illustrating a process of stimulating a user and receiving a response to an audio notification, according to several embodiments.

[0026] [Figure 10A] This is a flowchart illustrating a process for conditionally outputting voice notifications based on user activity and context, according to several embodiments. [Figure 10B]This is a flowchart illustrating a process for conditionally outputting voice notifications based on user activity and context, according to several embodiments. [Modes for carrying out the invention]

[0027] As described above, voice notifications are useful for communicating information to users by conveying information in language and by allowing users to conveniently hear the information without having to see it. Existing methods for providing voice notifications are often limited in functionality or effectiveness. In some cases, existing methods for providing voice notifications require multiple inputs to generate or respond to a voice notification. In some cases, existing methods for providing voice notifications provide voice notifications in an intrusive and distracting manner. The devices, methods, and user interfaces described herein improve user interface interaction with voice notifications in several ways. For example, embodiments disclosed herein provide more efficient methods for electronic devices to provide voice notifications and receive responses to voice notifications. In addition, embodiments disclosed herein provide improved voice notifications that communicate information to users more efficiently.

[0028] Figures 1A-1B, 2, and 3A-3B below provide a description of exemplary devices. Figures 4A-4B show exemplary user interfaces of exemplary devices in which embodiments disclosed herein are implemented. Figures 5A-5AD show exemplary voice notifications and exemplary user interfaces, as well as interaction with voice notifications. Figures 6A-6E show flowcharts of a method for providing voice notifications to an audio output device worn by a user. Figures 7A-7C show flowcharts of a method for providing audio context to convey additional information about an event. Figures 8A-8D show flowcharts of a method for performing actions based on the timing of user input to a voice notification. Figures 9A-9C show flowcharts of a method for prompting a user to receive a response to a voice notification. Figures 10A-10B show flowcharts of a method for conditionally outputting voice notifications based on user activity and context. The user interfaces and interactions in Figures 5A-5AD are used to illustrate the processes in Figures 6A-6E, 7A-7C, 8A-8D, 9A-9C, and 10A-10B.

[0029] Example device Herein, a detailed reference is made to embodiments shown in the accompanying drawings. The following detailed description includes numerous specific details to provide a complete understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other examples, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the aspects of the embodiments.

[0030] In this specification, terms such as "first," "second," etc., are used to describe various elements in several embodiments, but it will be understood that these elements should not be limited by those terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the various embodiments described, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first and second contacts are contacts, but they are not the same contact unless the context explicitly indicates otherwise.

[0031] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and is not intended to be limiting. In the descriptions of the various embodiments and the accompanying claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless otherwise explicitly stated in the context. Furthermore, it should be understood that, as used herein, the term “and / or” refers to and includes any and all possible combinations of one or more of the enumerated items relating to the invention. It will be further understood that, as used herein, the terms “includes,” “comprises,” and / or “comprising,” specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0032] Where used herein, the term "if" is interpreted, at its discretion and in context, to mean "when," "upon," "in response to determining," or "in response to detecting." Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are interpreted, at its discretion and in context, to mean "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0033] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, which also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as laptop or tablet computers having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad).

[0034] The following discussion describes electronic devices including displays and touch-sensitive surfaces. However, please understand that the definition of an electronic device is optional and may include one or more other physical user interface devices such as a physical keyboard, mouse, and / or joystick.

[0035] The device typically supports a variety of applications, including one or more of the following: gaming applications, note-taking applications, drawing applications, presentation applications, word processing applications, spreadsheet applications, telephone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0036] Various applications running on this device optionally utilize at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed by the device, are optionally adjusted and / or modified on an application-by-application basis and / or within each application. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally supports a variety of applications with intuitive and transparent user interfaces for the user.

[0037] Here, we turn our attention to embodiments of portable devices having a touch-sensitive display. Figure 1A is a block diagram of a portable multifunction device 100 having a touch-sensitive display system 112 according to several embodiments. The touch-sensitive display system 112 may be referred to as a “touchscreen” for convenience, or simply as a touch-sensitive display. The device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164 (for example, as part of one or more cameras). The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (for example, a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 that generate tactile output on Device 100 (for example, on a touch-sensitive surface such as the touch-sensitive display system 112 of Device 100 or the touchpad 355 of Device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0038] As used herein and in the claims, the term “tactile output” refers to the physical displacement of a device relative to its previous position, the physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or the displacement of a component relative to the center of mass of a device, which will be detected by the user through the user’s sense of touch. For example, in a situation where a device or component of a device is in contact with a touch-sensitive user’s surface (e.g., the user’s fingers, palm, or other part of their hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or component of the device. For example, the movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may be optionally interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may feel a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user’s action. As another example, movement of a touch-sensitive surface may be interpreted or perceived by the user as "roughness" of that surface, even if there is no change in the smoothness of the touch-sensitive surface. Such user interpretations of touch depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception of a typical (or average) user.By providing haptic feedback to the user using tactile output, device usability is improved, the user device interface becomes more efficient (for example, by helping the user provide appropriate input when operating / interacting with the device and reducing user errors), and in addition, power consumption is reduced and the device's battery life is improved by enabling the user to use the device more quickly and efficiently.

[0039] It should be understood that device 100 is merely an example of a portable multifunction device, and that device 100 may optionally have more or fewer components than those shown, may optionally be a combination of two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in Figure 1A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application-specific integrated circuits.

[0040] Memory 102 optionally includes high-speed random-access memory and optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by CPU(s) 120 and other components of device 100 such as peripheral interface 118 is optionally controlled by memory controller 122.

[0041] A peripheral device interface 118 is used to connect the device's input and output peripherals to the CPU(s) 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and process data.

[0042] In some embodiments, the peripheral interface 118, the CPU(s) 120, and the memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0043] The RF (radio frequency) circuit 108 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 108 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, which include, but are not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, and memory. The RF circuit 108 optionally communicates wirelessly with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), as well as with other devices. Wireless communication is optional and includes Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), and Code Division Multiple Access (W-CDMA).Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant Messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE)), Instant Messaging and Presence Services Using any of several communication standards, communication protocols, and communication technologies, including, but not limited to, a Service (IMPS), and / or Short Message Service (SMS), or any other suitable communication protocol, including a communication protocol not yet developed as of the filing date of this document.

[0044] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives the electrical signal converted from the sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or RF circuit 108 by the peripheral interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212, Figure 2). The headset jack provides an interface between the audio circuit 110 and detachable audio input / output peripherals such as output-only headphones or headsets that have both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0045] The I / O subsystem 106 connects input / output peripherals on device 100, such as the touch-sensitive display system 112 and other input or control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from and transmit electrical signals to the other input or control devices 116. The other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller(s) 160 are optionally connected to (or not connected to) any of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., 208, Figure 2) optionally include up / down buttons for controlling the volume of speaker 111 and / or microphone 113. One or more buttons optionally include push buttons (e.g., 206, Figure 2).

[0046] The touch-sensitive display system 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays a visual output to the user. This visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively, “graphics”). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term “affordance” refers to a user-interactive graphical user interface object (for example, a graphical user interface object configured to respond to input directed toward the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0047] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accept user input based on tactile and / or haptic contact. The touch-sensitive display system 112 and the display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of contact) on the touch-sensitive display system 112 and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0048] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light-emitting polymer display) technology, or LED (light-emitting diode) technology, but other display technologies are used in other embodiments. The touch-sensitive display system 112 and the display controller 156 optionally use any of several currently known or future-developed touch-sensing technologies, including but not limited to capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more contact points with the touch-sensitive display system 112, to detect contact and any movement or interruption thereof. In some embodiments, projected mutual capacitance sensing technology is used, such as that found in iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California.

[0049] The touch-sensitive display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the video resolution of the touchscreen is greater than 400 dpi (e.g., 500 dpi, 800 dpi, or higher). The user optionally touches the touch-sensitive display system 112 using any suitable object or attachment such as a stylus or finger. In some embodiments, the user interface is designed to function with finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen than that of a stylus. In some embodiments, the device translates coarse finger input into a precise pointer / cursor position or command to perform an action desired by the user.

[0050] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0051] Device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with generating, managing, and distributing power within the portable device.

[0052] Device 100 also optionally includes one or more optical sensors 164 (for example, as part of one or more cameras). Figure 1A shows an optical sensor coupled with an optical sensor controller 158 in the I / O subsystem 106. The optical sensor(s) 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor(s) 164 receives light from the environment, projected through one or more lenses, and converts the light into data representing an image. In conjunction with the imaging module 143 (also called the camera module), the optical sensor(s) 164 optionally captures still images and / or video. In some embodiments, the optical sensors are located on the back of device 100, opposite the touch-sensitive display system 112 on the front of the device, so that a touchscreen can be used as a viewfinder for acquiring still images and / or video images. In some embodiments, a separate light sensor is positioned on the front of the device so that an image of the user can be captured (for example, for selfies, or for video conferencing while the user is viewing other video conference participants on the touchscreen).

[0053] Device 100 also optionally includes one or more contact strength sensors 165. Figure 1A shows a contact strength sensor coupled with a strength sensor controller 159 in the I / O subsystem 106. The contact strength sensor(s) 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other strength sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact strength sensor(s) 165 receives contact strength information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact strength sensor is positioned alongside or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112). In some embodiments, at least one contact strength sensor is located on the back of Device 100, opposite the touchscreen display system 112 which is located on the front of Device 100.

[0054] The device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows a proximity sensor 166 coupled to a peripheral interface 118. Alternatively, the proximity sensor 166 is coupled to an input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is positioned near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0055] Device 100 also optionally includes one or more tactile output generators 163. Figure 1A shows a tactile output generator coupled with a tactile feedback controller 161 in the I / O subsystem 106. In some embodiments, the tactile output generator(s) 163 includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile outputs on the device). The tactile output generator(s) 163 receives a tactile feedback generation command from the tactile feedback module 133 and generates a tactile output on the device 100 that can be sensed by the user of the device 100. In some embodiments, at least one tactile output generator is positioned alongside or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112) and optionally generates a tactile output by moving the touch-sensing surface vertically (e.g., inward / outward from the surface of device 100) or horizontally (e.g., forward / backward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is positioned on the back of device 100, opposite the touch-sensing display system 112 which is located on the front of device 100.

[0056] Device 100 also optionally includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (e.g., as part of an inertial measurement unit, IMU) for obtaining information about the device's position (e.g., attitude). Figure 1A shows sensors 167, 168, and 169 coupled to a peripheral interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to an input controller 160 in an I / O subsystem 106. In some embodiments, the information is displayed on a touchscreen display in a portrait or landscape view based on an analysis of data received from one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the device's location.

[0057] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, as shown in Figures 1A and 3, memory 102 stores device / global internal state 157. The device / global internal state 157 includes one or more of the following: an active application state indicating which application is active, if there is an application currently active; a display state indicating which application, view, or other information occupies various areas of the touch-sensitive display system 112; a sensor state including information obtained from various sensors and other input or control devices 116 of the device; and position and / or orientation information relating to the position and / or orientation of the device.

[0058] An operating system 126 (for example, embedded operating systems such as iOS®, Android, Darwin, RTXC, LINUX, UNIX®, OS X®, WINDOWS®, or VxWorks) includes various software components and / or drivers for controlling and managing overall system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware components and software components.

[0059] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) are adapted to connect to other devices directly or indirectly via a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors identical or similar to and / or compatible with the 30-pin connectors used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external ports are Lightning connectors identical or similar to and / or compatible with the Lightning connectors used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external port is a USB Type-C connector identical, similar to, and / or compatible with the USB Type-C connector used in some electronic devices from Apple Inc. in Cupertino, California.

[0060] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in cooperation with the display controller 156) and contact with other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger down event), determining the intensity of contact (e.g., the force or pressure of contact, or an alternative to the force or pressure of contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting a drag event of one or more fingers), and determining whether contact has stopped (e.g., detecting a finger up event or interruption of contact). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can optionally be applied to a single contact (e.g., a single finger or stylus contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0061] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motion, timing, and / or intensity of detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger up (lift-off) event at the same location (or substantially the same location) as the finger down event (e.g., at the icon location). In another embodiment, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger down event, followed by a drag event of one or more fingers, and then a finger up (lift-off) event. Similarly, taps, swipes, drags, and other gestures are optionally detected with respect to the stylus by detecting specific contact patterns with respect to the stylus.

[0062] In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger down event and detecting a finger up event, but not on the intensity of finger contact between the two events. In some embodiments, a tap gesture is detected according to the determination that the length of time between the finger down event and the finger up event is shorter than a predetermined value (e.g., shorter than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the intensity of finger contact between taps meets a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a light or deep pressure intensity threshold. Thus, a finger tap gesture can satisfy certain input criteria that do not require the characteristic intensity of contact to meet a given intensity threshold in order for certain input criteria to be met. To clarify, finger contact in a tap gesture generally needs to meet a nominal contact detection intensity threshold below which contact is not detected in order to detect a finger down event. A similar analysis applies to detecting tap gestures or other contacts by a stylus. In cases where the device can detect contact from a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold is optional and does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0063] In a similar manner, the same concept applies to other types of gestures. For example, swipe gestures, pinch gestures, de-pinch gestures, and / or long press gestures are optional and are detected based on whether they meet criteria that are either unrelated to the intensity of the contacts involved in the gesture, or do not require the contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, swipe gestures are detected based on the amount of movement of one or more contacts, pinch gestures are detected based on the movement of two or more contacts toward each other, de-pinch gestures are detected based on the movement of two or more contacts toward each other, and long press gestures are detected based on the duration of contact on the touch-sensitive surface that is less than a threshold amount of movement. Therefore, the statement that a particular gesture recognition criterion does not require the intensity of one or more contacts to meet a corresponding intensity threshold for that criterion to be satisfied means that a particular gesture recognition criterion can be satisfied when the contact(s) in the gesture do not reach the corresponding intensity threshold, and can also be satisfied when one or more of the contacts in the gesture reach or exceed the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on the determination that a finger-down event and a finger-up event were detected within a predetermined time period, regardless of whether the contacts are above or below their respective intensity thresholds during a predetermined time period, and a swipe gesture is detected based on the determination that the movement of the contact is greater than a predetermined magnitude, even if the contacts exceed their respective intensity thresholds at the end of the movement of the contacts. Even in embodiments where gesture detection is influenced by the intensity of the contact performing the gesture (for example, the device detects longer presses more quickly when the contact intensity exceeds an intensity threshold, or the device is slower to detect tap inputs when the contact intensity is higher), the detection of those gestures does not require the contact to reach a specific intensity threshold, as long as the criteria for recognizing the gesture can be met in situations where the contact does not reach a specific intensity threshold (for example, even if the amount of time required to recognize the gesture changes).

[0064] Contact intensity thresholds, duration thresholds, and movement thresholds are combined in various different combinations to create heuristics for distinguishing two or more different gestures directed towards the same input element or region in certain situations, thereby enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. The statement that a particular set of gesture recognition criteria does not require the intensity of a contact(s) to meet its respective intensity threshold for that particular gesture recognition criterion to be satisfied does not preclude simultaneously evaluating other intensity-dependent gesture recognition criteria to identify other gestures that have criteria that are satisfied when the gesture includes contact with an intensity exceeding its respective intensity threshold. For example, in some situations, a first gesture recognition criterion for a first gesture that does not require the intensity of a contact(s) to meet a corresponding intensity threshold for the first gesture recognition criterion to be satisfied is in competition with a second gesture recognition criterion for a second gesture that depends on the contact(s) reaching a corresponding intensity threshold. In such competition, a gesture is optional and will not be recognized as satisfying the first gesture recognition criterion for the first gesture if the second gesture recognition criterion for the second gesture is first satisfied. For example, if a contact reaches the corresponding intensity threshold before it moves a predetermined amount, a deep press gesture will be detected instead of a swipe gesture. Conversely, if a contact moves a predetermined amount before it reaches the corresponding intensity threshold, a swipe gesture will be detected instead of a deep press gesture. Even in such situations, the first gesture recognition criterion for the first gesture still does not require the intensity of the contact(s) to meet the corresponding intensity threshold for the first gesture recognition criterion to be satisfied, because if the contact remains below the corresponding intensity threshold until the end of the gesture (e.g., a swipe gesture with a contact that does not increase to an intensity above the corresponding intensity threshold), the gesture is recognized as a swipe gesture by the first gesture recognition criterion.In this way, a particular gesture recognition criterion that does not require the intensity of a contact(s) to meet a corresponding intensity threshold for that particular gesture recognition criterion to be satisfied still depends on the intensity of the contact(s) with respect to the intensity threshold in some situations, and / or (B) in some situations, the particular gesture recognition criterion (e.g., for a long press gesture) does not function if a competing set of intensity-dependent gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-dependent gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input (e.g., for a long press gesture competing with a deep press gesture for recognition).

[0065] In conjunction with the accelerometer 167, gyroscope 168, and / or magnetometer 169, the attitude module 131 selectively detects attitude information about the device, such as the device's attitude (e.g., roll, pitch, and / or yaw) within a specific coordinate system. The attitude module 131 includes software components for performing various actions related to detecting the device's position and changes in the device's attitude.

[0066] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other display, including components for modifying the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term “graphics” includes, but is not limited to, any object that can be displayed to a user, including, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, and animations.

[0067] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like, as needed, along with coordinate data and other graphic characteristic data, specifying the graphics to be displayed, and then generates screen image data to be output to the display controller 156.

[0068] The haptic feedback module 133 includes various software components that generate commands (for example, commands used by the haptic feedback controller 161) that create haptic outputs at one or more locations on the device 100 using a haptic output generator (one or more) 163 in response to user interaction with the device 100.

[0069] The text input module 134 is optionally a component of the graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other applications that require text input).

[0070] The GPS module 135 determines the device's location and provides this information for use in various applications (for example, to the phone 138 for location-based phone calls, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0071] The notification module 145 provides logic for processing detected events such as incoming messages or phone calls, calendar event invitations or reminders, application events, and system events. The notification module facilitates and / or controls the display of notifications and alerts associated with detected events (e.g., on a display, via audio output, or via another type of output). For example, the notification module 145 may interface with application 136 to present notifications associated with its application and / or with operating system 126 to present system notifications.

[0072] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof: ● Contact module 137 (sometimes called the address book or contact list), ●Telephone module 138, ●Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ●Training support module 142, ● Camera module 143 for still images and / or video images, ●Image management module 144, ● Browser module 147, ● Calendar module 148, ● A widget module 149 that optionally includes one or more of the following: weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, other widgets obtained by the user, and user-created widgets 149-6. ●Widget creation module 150 for creating user-created widget 149-6, ● Search module 151, ● A video and music player module 152, which optionally consists of a video player module and a music player module. ●Memo Module 153, ●Map module 154 and / or ● Online video module 155.

[0073] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java®-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0074] Together with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contact module 137 includes executable instructions for managing an address book or contact list (stored, for example, in the application internal state 192 of the contact module 137 in memory 102 or memory 370), which include adding names(s) to the address book, removing names(s) from the address book, associating telephone numbers(s) to names, email addresses(s) to names, physical addresses(s) to names, or other information, associating images to names, categorizing and sorting names, providing telephone numbers and / or email addresses to initiate and / or facilitate communication by telephone 138, video conferencing 139, email 140, or IM 141, and so on.

[0075] In conjunction with the RF circuit 108, voice circuit 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the telephone module 138 includes executable commands to input a series of characters corresponding to a telephone number, access one or more telephone numbers in the address book 137, change the entered telephone number, dial each telephone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication may optionally use any of several communication standards, protocols, and technologies.

[0076] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, the video conferencing module 139 includes executable commands to start, conduct, and end a video conference between the user and one or more other participants, in accordance with the user's commands.

[0077] In conjunction with the RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the email client module 140 includes executable commands for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with the image management module 144, the email client module 140 makes it extremely easy to create and send emails containing still or video images captured by the camera module 143.

[0078] In conjunction with the RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the instant messaging module 141 includes executable instructions for inputting a series of characters corresponding to an instant message, modifying previously entered characters, sending each instant message (for example, using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messaging, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0079] In conjunction with the RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, the training support module 142 includes executable commands for creating training (e.g., having time, distance, and / or calorie consumption targets), communicating with training sensors (in sports devices and smartwatches), receiving training sensor data, calibrating sensors used to monitor training, selecting and playing music for training, and displaying, storing, and transmitting training data.

[0080] Together with the touch-sensitive display system 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0081] Together with the touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, text input module 134, and camera module 143, the image management module 144 includes executable commands for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0082] Together with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, the browser module 147 includes executable commands for browsing the internet in accordance with user commands, including searching, linking, receiving, and displaying web pages or parts thereof, as well as attachments and other files linked to web pages.

[0083] Together with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, the calendar module 148 includes executable instructions that, in accordance with user commands, create, display, modify, and store a calendar and data associated with the calendar (e.g., calendar items, to-do lists, etc.).

[0084] Together with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphic module 132, text input module 134, and browser module 147, the widget module 149 is optionally a mini-application downloaded and used by the user (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or a mini-application created by the user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets®).

[0085] In conjunction with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, the widget creation module 150 includes executable commands for creating widgets (for example, converting user-specified portions of a web page into widgets).

[0086] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions to search for text, music, sounds, images, videos, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user commands.

[0087] In conjunction with the touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, the video and music player module 152 includes executable commands that enable the user to download and play recorded music or other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable commands to display, present, or otherwise play video (for example, on the touch-sensitive display system 112, or on an external display connected wirelessly or via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0088] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the memo module 153 includes executable commands that create and manage notes, to-do lists, and the like, according to user commands.

[0089] In conjunction with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 may be used to receive, display, modify, and store maps and map-associated data (e.g., driving directions, data about stores and other points of interest at or near a particular location, and other location-based data) in accordance with user commands.

[0090] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, the RF circuit 108, the text input module 134, the email client module 140, and the browser module 147, the online video module 155 includes executable instructions that enable the user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on the touchscreen 112, or on an external display connected wirelessly or via the external port 124), send emails containing links to specific online videos, and otherwise manage them. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos.

[0091] Each of the modules and applications identified above corresponds to an executable instruction set that performs one or more of the functions described above, as well as the methods described in this application (e.g., methods performed by a computer and other information processing methods described herein). These modules (i.e., instruction sets) do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules can be optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0092] In some embodiments, device 100 is a device in which the operation of a default set of functions in the device is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for device 100 to operate, the number of physical input control devices (push buttons, dials, etc.) on device 100 is optionally reduced.

[0093] A default set of functions, performed only through the touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, a “menu button” is implemented using a touch-sensitive surface. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touch-sensitive surface.

[0094] Figure 1B is a block diagram showing exemplary components for event processing according to several embodiments. In some embodiments, memory 102 (in Figure 1A) or 370 (in Figure 3A) includes an event sorting unit 170 (e.g., within the operating system 126) and each application 136-1 (e.g., any of the applications 136, 137-155, 380-390 described above).

[0095] The event sorter 170 receives event information and determines the application 136-1 that distributes the event information, and the application view 191 of application 136-1. The event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates the current application view(s) displayed on the touch-sensitive display system 112 when the application is active or running. In some embodiments, a device / global internal state 157 is used by the event sorter 170 to determine which application(s) are currently active, and the application internal state 192 is used by the event sorter 170 to determine the application view 191 from which the event information is distributed.

[0096] In some embodiments, the application internal state 192 includes additional information such as resume information to be used when the application 136-1 resumes execution, user interface state information that indicates or is ready to display information displayed by the application 136-1, a state queue that allows the user to return to a previous state or view of the application 136-1, and one or more redo / undo queues of previous actions performed by the user.

[0097] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (for example, a user touch on the touch-sensitive display system 112 as part of a multi-touch gesture). The peripheral interface 118 transmits information received from the I / O subsystem 106, or from sensors such as the proximity sensor 166, one or more accelerometers 167, and / or the microphone 113 (via the audio circuit 110). The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display system 112 or the touch-sensitive surface.

[0098] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when a significant event occurs (e.g., receiving input that exceeds a predetermined noise threshold and / or for a longer period than predetermined).

[0099] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognition determination module 173.

[0100] The hit view determination module 172 provides software procedures for determining where in one or more views a sub-event occurred when the touch-sensitive display system 112 displays two or more views. A view consists of control devices and other elements that the user can see on the display.

[0101] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of each application) in which a touch is detected optionally corresponds to the program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally called a hit view, and the set of events recognized as appropriate input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch gesture.

[0102] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchy from which sub-events should be processed. In most situations, the hit view is the lowest-level view from which the initiating sub-event (i.e., the first sub-event in a series of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source from which it was identified as the hit view.

[0103] The active event recognition determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition determination module 173 determines that all views, including the physical location of the sub-event, are actively involved views, and therefore all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is entirely confined to an area associated with one particular view, higher-level views in the hierarchy still remain actively involved views.

[0104] The event dispatcher module 174 dispatches event information to an event recognition unit (e.g., an event recognition unit 180). In embodiments including an active event recognition unit determination module 173, the event dispatcher module 174 distributes the event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information acquired by each event receiving unit module 182 in an event queue.

[0105] In some embodiments, the operating system 126 includes an event sorter 170. Alternatively, application 136-1 includes an event sorter 170. In yet another embodiment, the event sorter 170 is a standalone module or part of another module stored in memory 102, such as a contact / motion module 130.

[0106] In some embodiments, application 136-1 includes a plurality of event processing units 190 and one or more application views 191, each containing instructions for handling touch events occurring within each view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other characteristics. In some embodiments, each event processing unit 190 includes one or more of the following: a data update unit 176, an object update unit 177, a GUI update unit 178, and / or event data 179 received from an event sorter 170. The event processing unit 190 optionally utilizes or calls the data update unit 176, object update unit 177, or GUI update unit 178 to update the application's internal state 192. Alternatively, one or more of the application views 191 include one or more respective event processing units 190. In some embodiments, one or more of the data update unit 176, object update unit 177, and GUI update unit 178 are included in each application view 191.

[0107] Each event recognition unit 180 receives event information (e.g., event data 179) from the event sort 170 and identifies an event from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparison unit 184. In some embodiments, the event recognition unit 180 also includes metadata 183 and at least a subset of event distribution commands 188 (optionally including sub-event distribution commands).

[0108] The event receiving unit 182 receives event information from the event sorter 170. The event information includes information about sub-events, such as touches or the movement of touches. Depending on the sub-event, the event information also includes additional information such as the position of the sub-event. When the sub-event involves the movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, an event includes the rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also called the device's orientation).

[0109] The event comparison unit 184 compares event information with a predefined definition of an event or sub-event, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes definitions of events (e.g., a default sequence of sub-events), such as event 1 (187-1) and event 2 (187-2). In some embodiments, sub-events in event 187 include, for example, the start of a touch, the end of a touch, a move of a touch, a stop of a touch, and multiple touches. In one embodiment, the definition for event 1 (187-1) is a double tap on a displayed object. A double tap includes, for example, a first touch (start of touch) for a predetermined stage on the displayed object, a first lift-off (end of touch) for a predetermined stage, a second touch (start of touch) for a predetermined stage on the displayed object, and a second lift-off (end of touch) for a predetermined stage. In another embodiment, event 2(187-2) is defined as a drag on a displayed object. The drag includes, for example, touching (or contacting) a predetermined number of stages on the displayed object, moving the touch across the touch-sensitive display system 112, and lifting off the touch (end of the touch). In some embodiments, the event also includes information about one or more associated event processing units 190.

[0110] In some embodiments, the event definition 187 includes an event definition for each user interface object. In some embodiments, the event comparison unit 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display system 112, when a touch is detected on the touch-sensitive display system 112, the event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with its respective event processing unit 190, the event comparison unit uses the results of the hit test to determine which event processing unit 190 should be activated. For example, the event comparison unit 184 selects the sub-event and the event processing unit associated with the object that triggers the hit test.

[0111] In some embodiments, the definition of each event 187 also includes a delay action that delays the delivery of event information until it is determined whether a series of sub-events correspond to the event type of the event recognition unit.

[0112] If each event recognition unit 180 determines that a series of sub-events does not match any of the events in the event definition 186, each event recognition unit 180 enters an event impossible, event failed, or event terminated state and thereafter ignores subsequent sub-events of the touch-based gesture. In this situation, if there are other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch-based gesture.

[0113] In some embodiments, each event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists that indicate to the actively involved event recognition unit how the event distribution system should perform sub-event distribution. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how the event recognition units interact with each other, or how they can interact with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are distributed to various levels in the view hierarchy or program hierarchy.

[0114] In some embodiments, each event recognition unit 180 activates an event processing unit 190 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, each event recognition unit 180 delivers event information associated with the event to the event processing unit 190. Activating the event processing unit 190 is separate from sending (and delaying the sending of) sub-events to the respective hit view. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event processing unit 190 associated with that flag captures the flag and executes a default process.

[0115] In some embodiments, the event distribution command 188 includes a sub-event distribution command that distributes event information about a sub-event without activating an event processing unit. Instead, the sub-event distribution command distributes event information to an event processing unit associated with a set of sub-events, or to a view that is actively involved. The event processing unit associated with the set of sub-events or the view that is actively involved receives the event information and performs predetermined processing.

[0116] In some embodiments, the data update unit 176 creates and updates data used in application 136-1. For example, the data update unit 176 updates telephone numbers used in contact module 137 or stores video files used in video and music player module 152. In some embodiments, the object update unit 177 creates and updates objects used in application 136-1. For example, the object update unit 177 creates new user interface objects or updates the position of user interface objects. The GUI update unit 178 updates the GUI. For example, the GUI update unit 178 prepares display information and sends it to graphics module 132 for display on touch-sensitive display.

[0117] In some embodiments, the event processing unit(s) 190 includes or has access to a data update unit 176, an object update unit 177, and a GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in a single module of their respective applications 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0118] The foregoing description regarding the handling of user touch events on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 100 using input devices, but it should be understood that not all of these begin on the touchscreen. For example, mouse movement and mouse button presses, touch movements such as taps, drags, and scrolls on a touchpad, which are optionally linked to single or multiple presses or holds on a keyboard, pen stylus input, input based on real-time analysis of video images acquired by one or more cameras, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof may be optionally used as inputs corresponding to sub-events that define the events to be recognized.

[0119] Figure 2 shows a portable multifunctional device 100 having a touchscreen (e.g., a touch-sensitive display system 112, Figure 1A) according to several embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In these embodiments, and in embodiments described later, the user can select one or more of the graphics by making gestures on the graphics using, for example, one or more fingers 202 (not shown in the figure to an exact scale) or one or more styluses 203 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with that one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward), and / or rolling of a finger in contact with the device 100 (from right to left, left to right, upward and / or downward). In some implementations or situations, accidental contact with a graphic does not constitute a selection of that graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon does not arbitrarily select the corresponding application.

[0120] Device 100 optionally also includes one or more physical buttons, such as a "Home" or menu button 204. As previously mentioned, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally run on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on a touchscreen display.

[0121] In some embodiments, device 100 includes a touchscreen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for turning the device on / off and locking the device, one or more volume buttons 208, a subscriber identification module (SIM) card slot 210, a headset jack 212, and an external docking / charging port 124. The push button 206 is optionally used to turn the device on / off by pressing down and holding the button down for a predetermined period of time, to lock the device by pressing down and releasing the button before a predetermined period of time has elapsed, and / or to unlock the device or initiate an unlocking process. In some embodiments, device 100 also accepts verbal input through a microphone 113 to activate or deactivate certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output to the user of device 100.

[0122] Figure 3A is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to several embodiments. Device 300 does not need to be portable. In some embodiments, device 300 is a game system, laptop computer, desktop computer, tablet computer, multimedia playback device, navigation device, educational device (such as a children's learning toy), game system, or control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication buses 320 optionally include circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 300 includes an input / output (I / O) interface 330, which optionally includes a display 340, which is a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, a touchpad 355, a touch output generator 357 for generating tactile output on a device 300 (similar to the tactile output generator(s) 163 described above with reference to Figure 1A, for example), and a sensor 359 (e.g., an optical sensor, an accelerometer, a proximity sensor, a touch sensor, and / or a touch intensity sensor similar to the touch intensity sensor(s) 165 described above with reference to Figure 1A). The memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU(s) 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or subsets thereof, that are stored in memory 102 of the portable multifunction device 100 (Figure 1A). Furthermore, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of the portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while memory 102 of the portable multifunction device 100 (Figure 1A) optionally does not store those modules.

[0123] Each of the elements identified in Figure 3A above is optionally stored in one or more of the previously mentioned memory devices. Each of the modules identified above corresponds to an instruction set that performs the function described above. The modules or programs (i.e., instruction sets) identified above do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of those modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0124] Figure 3B is a block diagram of an exemplary wearable audio output device 301 according to several embodiments. In some embodiments, the wearable audio output device 301 is a single earphone. In some embodiments, the wearable audio output device 301 includes one or more pairs of earphones (e.g., one for each of the user's ears). In some embodiments, the wearable audio output device 301 is one or more in-ear earphones, one or more over-ear headphones, etc. In some embodiments, the wearable audio output device 301 includes one or more speakers 306 for providing audio output (e.g., to the user's ears). In some embodiments, the wearable audio output device 301 includes one or more placement sensors 304 for detecting the positioning or placement of the wearable audio output device 301 relative to the user's ear, such as for detecting the placement of the wearable audio output device 301 in the user's ear. In some embodiments, the wearable audio output device 301 conditionally outputs audio based on whether it is in or near the user's ear (for example, the wearable audio output device 301 may stop outputting audio when it is not in the user's ear to reduce power consumption). In some embodiments, the audio I / O logic 312 determines the positioning or placement of the wearable audio output device 301 relative to the user's ear based on information received from placement sensors 304, and in some embodiments, the audio I / O logic 312 controls the resulting conditional output of audio.

[0125] In some embodiments, the wearable audio output device 301 includes one or more microphones 302 for receiving audio input. In some embodiments, the microphones 302(s) detect speech from the user wearing the wearable audio output device 301 and / or ambient noise around the wearable audio output device 301. In some embodiments, audio I / O logic 312 detects or recognizes speech or ambient noise based on the information received from the microphones 302(s). In some embodiments, the wearable audio output device 301 includes one or more input devices 308, such as a touch-sensitive surface (for detecting touch input), accelerometers (s) and / or attitude sensors (for determining the attitude of the wearable audio output device 301 relative to the physical environment and / or changes in the attitude of the device), and / or other input devices that the user can use to interact with the wearable audio output device 301 and provide input. In some embodiments, inputs provided via input device(s) 308 are processed by audio I / O logic 312. In some embodiments, audio I / O logic 312 provides instructions or content for audio output and optionally communicates with a separate device (e.g., device 100 in Figure 1A) that receives and processes inputs (or information about inputs) provided via microphone(s) 302, placement sensor(s) 304, and / or input device(s) 308(s).

[0126] Here, we focus on embodiments of a user interface ("UI") that are optionally implemented on the portable multi-functional device 100.

[0127] Figure 4A shows an exemplary user interface for an application menu on a portable multifunction device 100 according to several embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular signals and Wi-Fi signals. ●Time, ●Bluetooth (registered trademark) indicator, ●Battery status indicator, ●Tray 408 containing icons for frequently used applications, as shown below. ○Optionally including an indicator 414 for the number of missed calls or voicemail messages, an icon 416 of the telephone module 138 labeled "Telephone", ○Optionally including an indicator 410 for the number of unread emails, an icon 418 of the email client module 140 labeled "Mail", ○ Icon 420 of browser module 147, labeled "Browser", and, ○ Icon 422 of the video and music player module 152 labeled "Music", and, ● Icons of other applications, such as the following: ○ Icon 424 of IM module 141, labeled "Message", ○ Icon 426 of calendar module 148, labeled "Calendar", ○ Icon 428 of image management module 144, labeled "Photo" ○ Icon 430 of camera module 143, labeled "Camera" ○ Icon 432 of online video module 155, labeled "online video" ○ Icon 434 of stock price widget 149-2, labeled "Stock Price" ○ Icon 436 of map module 154, labeled "Map" ○ Icon 438 of weather widget 149-1, labeled "Weather" ○ Icon 440 of the alarm clock widget 149-4, labeled "Clock" ○ Icon 442 of training support module 142, labeled "Training Support" ○ Icon 444 of memo module 153, labeled as "Memo," and, ○ An icon 446 labeled "Settings," which provides access to the settings of device 100 and its various applications 136, for a settings application or module.

[0128] Please note that the icon labels shown in Figure 4A are merely examples. For example, other labels may be used selectively for various application icons. In some embodiments, the label for each application icon includes the name of the application to which that application icon corresponds. In some embodiments, the label for a particular application icon is different from the name of the application to which that particular application icon corresponds.

[0129] Figure 4B shows an exemplary user interface on a device (e.g., device 300, Figure 3A) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355, Figure 3A) separate from the display 450. Many of the following embodiments are given by reference to input on the touchscreen display 112 (when the touch-sensitive surface and the display are combined), but in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in Figure 4B. In some embodiments, the touch-sensitive surface (e.g., 451 in Figure 4B) has a principal axis (e.g., 452 in Figure 4B) corresponding to a principal axis (e.g., 453 in Figure 4B) on the display (e.g., 450). According to those embodiments, the device detects contact with the touch-sensitive surface 451 (e.g., 460 and 462 in Figure 4B) at positions corresponding to each of the positions on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in Figure 4B). Thus, when the touch-sensitive surface is separated from the display, user input detected by the device on the touch-sensitive surface (e.g., 451 in Figure 4B) (e.g., touches 460 and 462, and their movement) is used by the device to operate the user interface on the display of the multifunction device (e.g., 450 in Figure 4B). It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0130] In some embodiments, the device's response to an input detected by the device depends on a criterion based on the contact intensity during the input. For example, for some "light press" inputs, a contact intensity exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the device's response to an input detected by the device depends on a criterion that includes both the contact intensity during the input and a time-based criterion. For example, for some "deep press" inputs, a contact intensity exceeding a second intensity threshold during an input greater than a first intensity threshold for light presses triggers a second response only if a delay time has elapsed between satisfying the first and second intensity thresholds. This delay time is typically shorter than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, and the delay time increases as the second intensity threshold increases). This delay time helps avoid accidental recognition of deep press inputs. As another example, for some "deep press" inputs, there is a period of reduced sensitivity that occurs after the time the first intensity threshold is satisfied. During the period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps to avoid accidental deep press inputs. For other deep press inputs, the response to the detection of deep press inputs is independent of time-based criteria.

[0131] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors, such as user settings, contact motion, input timing, the application being run, the rate at which intensity is applied, the number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), and the position of the focus selector. Exemplary factors are described in U.S. Patent Publications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0132] For example, FIG. 4C shows a dynamic intensity threshold 480 that changes over time, based in part on the intensity of the touch input 476 over time. The dynamic intensity threshold 480 is the sum of two components, a first component 474 that decays over time after a predetermined delay time p1 from when the touch input 476 is first detected, and a second component 478 that follows the trace of the intensity of the touch input 476 over time. The first high intensity threshold of the first component 474 reduces the accidental triggering of the "deep press" response while further enabling an immediate "deep press" response if the touch input 476 provides sufficient intensity. The second component 478 reduces the unintended triggering of the "deep press" response due to the gradual intensity variations of the touch input. In some embodiments, when the touch input 476 meets the dynamic intensity threshold 480 (e.g., at point 481 in FIG. 4C), a "deep press" response is triggered.

[0133] FIG. 4D shows another dynamic intensity threshold 486 (e.g., intensity threshold IT D ). FIG. 4D also shows two other intensity thresholds, a first intensity threshold IT H and a second intensity threshold IT L . In FIG. 4D, the touch input 484 meets the first intensity threshold IT H and the second intensity threshold IT L before time p2, but no response is provided until the delay time p2 has elapsed at time 482. Also, in FIG. 4D, the dynamic intensity threshold 486 decays over time with an attenuation that starts at time 488 after a predefined delay time p1 has elapsed from time 482 (when the response associated with the second intensity threshold IT L is triggered). This type of dynamic intensity threshold reduces the accidental triggering of the response associated with the dynamic intensity threshold IT H or the second intensity threshold IT L immediately after or simultaneously with the triggering of the response associated with a lower intensity threshold such as the first intensity threshold IT D .

[0134] FIG. 4E shows yet another dynamic intensity threshold 492 (e.g., intensity threshold IT D ). In FIG. 4E, the intensity threshold ITL The associated response is triggered after a delay time p2 has elapsed from the time the touch input 490 is first detected. Simultaneously, the dynamic intensity threshold 492 decays after a predetermined delay time p1 has elapsed from the time the touch input 490 is first detected. Therefore, without releasing the touch input 490, the intensity threshold IT is triggered after the increase in the intensity of the touch input 490. L The decrease in the intensity of touch input 490 after triggering the associated response is when the intensity of touch input 490 reaches another intensity threshold, e.g., intensity threshold IT L Even when it falls below (for example, at time 494), the intensity threshold IT D This can trigger the associated response.

[0135] User interface and related processes We now draw our attention to embodiments of user interfaces ("UI") and associated processes that may be implemented on an electronic device (e.g., portable multifunction device 100, Figure 1A, or device 300, Figure 3A) which includes (and / or communicates with) one or more audio output devices (e.g., one or more wearable audio output devices such as wearable audio output device 301, Figure 3B), optionally one or more sensors for detecting the placement of the audio output devices, optionally a display and / or touch-sensitive surface, optionally one or more accelerometers and / or attitude sensors, and optionally one or more additional input devices (e.g., rotatable input mechanisms).

[0136] Figures 5A–5AD illustrate exemplary voice notifications and user interfaces, as well as interactions with voice notifications, according to several embodiments. The user interfaces in these figures are used to illustrate the processes described below, including those in Figures 6A–6E, 7A–7C, 8A–8D, 9A–9C, and 10A–10B. For ease of explanation, some embodiments are discussed with reference to operations performed using a separate wearable output device that communicates with the device that detects events corresponding to voice notifications. However, in some cases, similar operations are optionally performed using an audio output device that is part of the device that detects the events.

[0137] Figures 5A to 5AD illustrate exemplary voice notifications and exemplary user interfaces (including language, oral, and tactile interfaces) according to several embodiments, as well as interactions with voice notifications.

[0138] Figure 5A illustrates the provision of voice notifications in response to a user 500 wearing one or more earphones 502. The earphones 502 can represent a single earphone, two earphones (one for each of the user 500's ears), one or more in-ear earphones, one or more over-ear headphones, etc. In some embodiments, the earphones 502 correspond to a wearable audio output device 301 (Figure 3B). In some embodiments, the earphones 502 communicate with a device (e.g., device 100, Figure 1A) which may be separate from or integrated with the earphones 502.

[0139] In Figure 5A, no audio output is provided through the earphone 502 while it is positioned away from the user 500's ear and outside the range 504 around the user 500's ear.

[0140] When the earphone 502 is placed in the ear of user 500 (or moved within its range 504), a voice summary 506 for the day is output via the earphone 502. In some embodiments, the voice summary 506 includes voice notifications (e.g., corresponding to events that have occurred, are scheduled to occur, and / or are expected to occur) in a device communicating with the earphone 502. In some embodiments, as shown in Figure 5A, the voice summary 506 is provided only in response to the initial detection of the placement of the earphone 502 in the ear of user 500 on a particular day. As shown in Figure 5A, the voice summary 506 includes any one or more of the following: ●The current date (for example, "Today is Monday, September 24th") ●The weather on that day (for example, "It is currently 65 degrees Celsius and sunny, with a high of 78 degrees Celsius"), ● A summary of events missed because earphone 502 was not in or within the user 500's ear (for example, the sentence, "You have 4 messages and 5 emails"), ● Commuting information (for example, "It currently takes me 25 minutes to get to work"), and also, ●The first calendar event coming up (for example, "Your next meeting is at 10:30 for the 'Project Status Update'").

[0141] Figure 5B shows an example of distinguishing voice notifications based on the properties of detected events. As shown in Figure 5B, the detected events correspond to received messages 508-1 through 508-4. Message 508-1 is received from "Harold Smith" and contains text content (e.g., "Hi, I'll be in town from next Wednesday through Sunday"). Next, messages 508-2 and 508-3 are received from "Barbara Lee (Mother)" (the user's mother) and contain text content (e.g., "Are you coming home this weekend?" and "It's Dad's birthday that day," respectively). Then, message 508-4 is also received from "Harold Smith" and contains text content (e.g., "Let's find some time to catch up").

[0142] In response to receiving message 508-1, a voice notification 510 containing the content of message 508-1 in a linguistic format (e.g., read aloud by converting the text of message 508-1 into speech) is provided to user 500 via earphone 502. In the example shown in Figure 5B, the voice notification 510 identifies the message sender, "Harold Smith," by his full name (e.g., both his first and last name). In some situations, including the contact's last name (e.g., "Smith") in addition to the contact's first name (e.g., "Harold"), as shown in Figure 5B, is because user 500 has not communicated with this contact for at least a predetermined period (e.g., at least one week, one month, one year, or other specified period). In some situations, including the contact's first and last name in a voice notification associated with a message or other event received from this contact is because user 500's address book contains another contact with the same name (e.g., user 500's address book contains another "Harold").

[0143] In response to receiving message 508-2, a voice notification 512 containing the content of message 508-2 in language format is provided to user 500 via earphone 502. In the example shown in Figure 5B, the voice notification 512 identifies the message sender "Barbara Lee" (user 500's mother) by her alias "Mother" (as specified by user 500 in his address book, for example) instead of her full name "Barbara Lee". The voice notification 512 does not identify "Mother" by her full name because user 500 has communicated with "Mother" within a predetermined period (e.g., a week, a month, a year, or other specified time period).

[0144] In addition, in response to receiving message 508-3, an audio notification 514 containing the content of message 508-3 in language format is provided to the user 500 via the earphone 502. In the example shown in Figure 5B, since message 508-3, which corresponds to audio notification 514, follows immediately after another message (for example, message 508-2, which corresponds to audio notification 512) also received from the same sender "Barbara Lee", audio notification 514 omits the name of the message sender "Barbara Lee".

[0145] In response to receiving message 508-4, an audio notification 516 containing the content of message 508-4 in language format is provided to user 500 via earphone 502. In the example shown in Figure 5B, since user 500 has received message 508-1 from the same "Harold Smith," and has communicated with "Harold Smith" within a predetermined period at this point, the audio notification 516 omits the last name of the message sender, "Harold Smith."

[0146] Figure 5C shows an example of outputting audio notifications at different pseudospatial locations based on the properties of the corresponding events. In some embodiments, multiple audio output devices are required to provide audio notifications at different pseudospatial locations. Thus, Figure 5C shows a user 500 wearing a pair of earphones 502. As shown in Figure 5C, multiple events, messages 508-4 through 508-6, have been received. As described with reference to Figure 5B, message 508-4 is a text message from "Harold Smith". Specifically, message 508-4 is sometimes referred to as a "personal message" in that it is sent from "Harold Smith" to user 500 and not to any other recipients. Messages 508-5 through 508-6 are both received as part of a group message session among three or more participants and are identified as a "team" and sometimes referred to as a "group message". Specifically, message 508-5 is received by participant "John Godfrey" in the "Team" group message and contains text content (e.g., "I'll be 10 minutes late, sorry"). Then, message 508-6 is received by another participant "Alice Young" in the same "Team" group message and contains text content (e.g., "It's okay, I'll look into other projects first").

[0147] In response to receiving message 508-4, an audio notification 518 containing the content of message 508-4 in language format is provided to user 500 via earphone 502. Since message 508-4 is a unique message, the audio notification 518 is output at a pseudo-spatial location 520 inside user 500's head so that it sounds as if it is coming from or being played inside user 500's head at spatial location 520. In some embodiments, unique messages such as audio notification 518 are output at one or more pseudo-spatial locations inside user 500's head. In some embodiments, different spatial locations are associated with different contacts. For example, an audio notification from a sender other than "Harold Smith" is output at a different pseudo-spatial location inside user 500's head than pseudo-spatial location 520 in some embodiments.

[0148] In response to receiving messages 508-5 to 508-6, voice notifications 522 and 526 are provided to user 500 via earphone 502, respectively. Voice notification 522 contains the content of message 508-5 in language format, beginning with an indication that the message is part of a group message and the identifier of the group message (for example, the group message begins with “in team group” and “team” is the group message identifier). In addition, voice notification 522 is output at a pseudo-spatial location 524 outside user 500's head so that it sounds as if it is coming from outside user 500's head or being played outside of it at spatial location 524 (for example, even if the sound is coming from earphone 502 in user 500's ear). Next, voice notification 526 contains the content of message 508-6 in language format. In some embodiments, as shown in the example in Figure 5C, the message 508-6 corresponding to the voice notification 526 follows immediately after another message (e.g., message 508-5 corresponding to voice notification 522) that was also received as part of the same group message (e.g., the "Team" group message), so the voice notification 526 omits the group message preface (e.g., "Within the Team Group"). The voice notification 526 is output using a different type of audio from the voice notification 522 (e.g., a different audio tone, or a different pseudo-voice) (e.g., as indicated by the italicized text representing the voice notification 526, as opposed to the normal text representing the voice notification 522). In addition, the voice notification 526 is output at a pseudo-spatial location 528 outside the user 500's head so that it sounds as if the voice notification 526 is coming from or being played outside the user 500's head at the spatial location 528.

[0149] In some embodiments, as shown in Figure 5C, messages 508-5 and 508-6 are group messages rather than individual messages, so voice notifications 522 and 526 are output at pseudospatial locations outside the user 500's head (e.g., pseudospatial locations 524 and 528, respectively). In addition, in some embodiments, as shown in Figure 5C, messages 508-5 and 508-6 are messages from different senders (e.g., different participants in a group message), so voice notification 522 is output at a different pseudospatial location from voice notification 526.

[0150] In some embodiments, messages for different types of events are distinguished by outputting corresponding voice notifications at different volumes. For example, as shown in Figure 5C, voice notification 518 is output at a higher volume (as indicated by bold text) than voice notifications 522 and 526. In some embodiments, individual messages (e.g., message 508-4 from "Harold Smith" to user 500 only) are output at a higher volume than group messages (e.g., messages 508-5 and 508-6 within the group "team" between user 500 and at least "John Godfrey" and "Alice Young"). Those skilled in the art will understand that there are many other ways to distinguish events, including, but not limited to, highlighting voice notifications from a particular contact to other contacts (for example, in response to user designation of a particular contact, such as through the creation of a contact “whitelist”), highlighting voice notifications associated with a particular application to other applications, highlighting voice notifications associated with a particular type of application to other types of applications, and highlighting voice notifications that meet certain geographical or temporal criteria (for example, highlighting voice notifications for a particular event while the user is in or near a particular geographical location, or during a particular time period).

[0151] Figure 5D illustrates an example of distinguishing voice notifications based on geographical context. As shown in Figure 5D, a first event 530 is detected (for example, on a device such as device 100 in Figure 1A, communicating with earphone 502). Event 530 is a notification of an upcoming calendar event, which is a meeting taking place at a specified time 532 (e.g., "10:30 AM") on a specified topic 531 (e.g., "Project Status Updates") at a specified location 533 (e.g., "Apollo Conference Room, Building 1, N. De Anza Blvd., Cupertino"). Event 530 is detected at a predetermined time for the calendar event (e.g., at the current time 534 "10:20 AM"), which is 10 minutes before the start of the upcoming calendar event. In response to the detection of event 530, a voice notification 538 is provided via earphone 502 to inform user 500 of the upcoming meeting. The voice notification 538 includes information about event 530, including instructions for the event type (e.g., "Meeting"), event time (e.g., "At 10:30"), and event theme (e.g., "Project Status Update").

[0152] As shown in Figure 5D, the voice notification 538 also includes event location information (e.g., "Apollo Conference Room"). In the example shown in Figure 5D, user 500's current location 536 is "Lobby, Building 1". Therefore, user 500 is already in the same building (e.g., "Building 1") as the meeting location associated with event 530. Thus, the voice notification 538 omits the building name (e.g., "Building 1"), as well as any description of a larger geographical area such as street and city names. The voice notification 538 includes only location information for a geographical area different from user 500's current location 536 (e.g., a geographical area within the building, such as a floor number or conference room name). In some embodiments, if user 500 is already in the same smallest geographical area as the upcoming event (e.g., the same conference room), event location information is not provided in the corresponding voice notification.

[0153] Figure 5D also shows that a second event 540 is detected (for example, by a device such as device 100 in Figure 1A, which is communicating with earphone 502). Event 540 is a notification of an upcoming calendar event, which is a meeting to take place at a specified time 542 (for example, "3:00 PM") on a specified topic 541 (for example, "Testbench Evaluation") at a specified location 543 (for example, "Montara Conference Room, Building 5, S. Main St., Sunnyvale"). Event 540 is detected at a predetermined time for the calendar event (for example, at the current time 544 "2:50 PM"), which is 10 minutes before the start of the upcoming calendar event. In response to the detection of event 540, an audio notification 548 is provided via earphone 502 to notify user 500 of the upcoming meeting. Voice notification 548 includes information about event 540, including instructions for the event type (e.g., "meeting"), event time (e.g., "at 3 o'clock"), and event theme (e.g., "testbench evaluation").

[0154] In some embodiments, the voice notification 548 also includes event location information (e.g., "Montara Conference Room, Building 5, S. Main St., Sunnyvale"). In the example shown in Figure 5D, user 500's current location 546 is "Lobby, Building 1". Thus, user 500 is in a different city (e.g., "Cupertino") than the city where event 540 takes place (e.g., "Sunnyvale"), although the two cities are in the same state. Thus, user 500 is in a different city than the city where event takes place (e.g., "Sunnyvale"), although the two cities are in the same state. Therefore, the voice notification 548 includes the name of the conference room (e.g., "Montara"), the name of the building (e.g., "Building 5"), the name of the street (e.g., "South Main Street"), and the name of the city (e.g., "Sunnyvale"), all of which are in different geographical areas than user 500's current location 546. Since user 500 is in the same state, voice notification 548 omits the name of the state where event 540 is taking place.

[0155] Figure 5E illustrates an example of distinguishing voice notifications based on the properties of the received event. Specifically, Figure 5E shows different sound effects and output volumes used to represent events indicating transactions of different values. In response to user 500 receiving a small payment from "Travis" (e.g., the cost of his "lunch"), voice notification 550 describing the transaction includes a first "[coin falling]" sound effect and is played at a low volume (as shown in lowercase text) through earphone 502. In response to user 500 receiving a large payment from "Jane" (e.g., the cost of her "share of the rent"), voice notification 552 describing the transaction includes a second "[clicking]" cash register sound effect, which differs from the first sound effect and is played at a high volume (as shown in bold uppercase text) through earphone 502.

[0156] Figure 5F shows exemplary behavior performed in response to user 500 interacting with earphone 502 after no notifications or events have been received for at least a predetermined period of time. Specifically, Figure 5F shows that no notifications have been received in the past 10 minutes (e.g., by a device communicating with earphone 502). Therefore, in response to the detection of a tap input 554 on earphone 502, earphone 502 begins playing music, as indicated by the audio output 556.

[0157] Figures 5G-5H show that the earphone 502 outputs condensed voice notifications in response to the detection of multiple events while it is not in the user 500's ear, and in the aggregate, the corresponding voice notifications for the detected events are numerous and / or long. Specifically, Figure 5G shows that multiple events are detected while the earphone 502 is not in the user 500's ear. The detected events correspond to messages 558-1 to 558-3 received from "Travis", messages 560-1 and 560-2 and event invitation 560-3 received from "Octavia", and messages 562-1 to 562-5 received from "Ellie". While the earphone 502 is not in the user 500's ear, voice notifications corresponding to the detected events are not output via the earphone 502.

[0158] Figure 5H shows that the earphone 502 is placed in the ear of user 500. In response, an audio notification corresponding to the detected event is provided. However, outputting the audio content associated with each of the detected events 558-1 to 558-3, 560-1 to 560-3, and 562-1 to 562-5 would require a longer time in the aggregate than necessary. Instead, a condensed audio notification is provided, as shown in Figure 5H. Specifically, audio notification 564 summarizes all messages 558-1 to 558-3 received from "Travis" into the sentence "Travis has sent you 3 messages." Similarly, audio notification 566 summarizes all messages 560-1 and 560-2 and event invitation 560-3 received from "Octavia" into the sentence "Octavia has sent you 2 messages and 1 event invitation." In the example in Figure 5H, audio notifications 564 and 566 are output at pseudo-spatial location 568.

[0159] Figures 5I–5K illustrate how the amount of content included in an audio notification (sometimes referred to as the “redundancy” of the audio notification) changes in response to user 500 interacting with earphone 502 via head movements. Figure 5I shows user 500 tilting their head towards a pseudo-spatial position 568 (e.g., towards the “source” of the sound) to hear more information about the audio notification just played (or, in some situations, the audio notification being played) after hearing an audio notification 566 summarizing events 560-1 through 560-3. In response, Figure 5J shows additional audio content associated with events 560-1 through 560-3 (e.g., an extended version of audio notification 566) being provided in the form of audio notifications 566-1 through 566-3. Audio notification 566-1 contains the content of a linguistic message 560-1 that begins only with the name of a sender “Octavia” with whom user 500 has communicated “recently” (e.g., within the last week, month, or year). Voice notification 566-2 contains the content of message 560-2 in language form, omitting any sender identifier (because message 560-2 follows immediately after message 560-1, which is another event from the same sender). Voice notification 566-3 describes the event invitation 560-3 (e.g., "Octavia has invited you to an event today at 4:30 PM"). In some embodiments, since the event invitation 560-3 follows immediately after another event (e.g., message 560-2) from the same sender (e.g., "Octavia"), voice notification 566-3 is rephrased to omit any sender identifier.

[0160] Figure 5K shows that after hearing voice notification 566-3, user 500 tilts their head away from the pseudo-spatial location 568 to hear less information about subsequent voice notifications (for example, to hear a condensed voice notification again). In response, a condensed voice notification 570 is provided at the pseudo-spatial location 568. Specifically, voice notification 570 summarizes all messages 562-1 through 562-5 received from "Ellie" into the sentence "Travis has sent you 5 messages."

[0161] Figures 5L-5R illustrate exemplary actions performed in response to user interaction with the earphone 502. In Figure 5L, while an audio notification 570 is provided, or within a predetermined period after the audio notification 570 is provided (e.g., within a few seconds), user 500 performs a tap input 572 on the earphone 502. In response to the detection of the tap input 572, additional audio content associated with messages 562-1 to 562-5 (e.g., extended versions of the audio notification 570) is provided in the form of audio notifications 570-1 to 570-5, as shown in Figures 5M-5P. In some embodiments, a (single) tap input directed to the earphone 502 during or "immediately" after an audio notification performs an action related to the audio notification (e.g., providing additional information, as shown in Figures 5L-5M), while a tap input outside the time cycle "immediately" after an audio notification performs a different action unrelated to any audio notification (e.g., starting or stopping music playback, as shown in Figure 5F).

[0162] In Figure 5M, voice notification 570-1 contains the content of a linguistic message 562-1, which begins only with the name "Ellie," the sender with whom user 500 has recently communicated. Voice notification 570-2 contains the content of a linguistic message 562-2. Specifically, since message 562-2 consists of a photograph, voice notification 570-2 contains a linguistic description of the photograph's content (e.g., a linguistic summary including one or more keywords associated with the photograph, determined using image processing). Thus, voice notification 570-2 contains the sentence, "Ellie sent a photograph of a cat," describing the photograph of a cat in message 562-2. In some embodiments, since message 562-2 follows immediately after another message (e.g., message 562-1) from the same sender (e.g., "Ellie"), voice notification 570-2 is rephrased to omit any sender identifier.

[0163] In Figure 5N, since message 562-3 follows immediately after message 562-2, which is also another event from the same sender (e.g., "Ellie"), the voice notification 570-3 contains the content of message 562-3 in language form and omits any sender identifier. After providing voice notification 570-3, earphone 502 begins providing the first part of voice notification 570-4, which contains the first part of the content of message 562-4 in language form (e.g., the partial sentence "You..."). While voice notification 570-4 is being provided (e.g., after the first part has been provided, but before the second subsequent part has been provided), user 500 directs the double-tap input 574 towards earphone 502, as shown in Figure 5O. In response, the earphone 502 skips the remainder of the voice notification 570-4 corresponding to the unread portion(s) of message 562-4, and then outputs the voice notification 570-5 corresponding to message 562-5, as shown in Figure 5P. In some embodiments, a (single) tap input directed to the earphone 502 during or "immediately" after the voice notification performs a different action with respect to the voice notification than a double tap input directed to the earphone 502 during or "immediately" after the voice notification (for example, skipping the remainder of the voice notification as shown in Figures 5O-5P) (for example, providing additional information as shown in Figures 5L-5M).

[0164] In addition, in Figure 5P, the voice notification 570-5 is provided at an initial time, as indicated by timer 576-1. In Figure 5Q, as indicated by timer 576-2, after a predetermined period (e.g., more than 7-10 seconds) has elapsed since the voice notification 570-5 was provided, user 500 directs a double-tap input 578 towards the earphone 502. In response to the double-tap input 578 detected after the predetermined period, the virtual assistant is invoked (e.g., on the earphone 502, or on a device communicating with the earphone 502, such as device 100 in Figure 1A). The invokement of the virtual assistant is indicated by an acoustic effect 580 (e.g., a virtual assistant chime or tone, or another indication that the virtual assistant is ready to receive voice commands). In some embodiments, a double-tap input directed at the earphone 502 during or immediately after an audio notification performs an action related to the audio notification (for example, skipping the rest of the audio notification as shown in Figures 5O-5P), while a tap input outside the time cycle immediately after an audio notification performs a different action unrelated to any audio notification (for example, invoking a virtual assistant as shown in Figures 5Q-5R).

[0165] Figures 5S-5AB illustrate how the user is stimulated to receive a response to an audio notification. In response to receiving event 560-3, which is a calendar event invitation from Octavia for a meeting at "4:30 PM", audio notification 566-3 is provided to user 500 via earphone 502. Audio notification 566-3 contains information about event 560-3 in linguistic form.

[0166] After providing the voice notification 566-3, the earphone 502 plays an acoustic effect 582 (e.g., white noise) to indicate that the earphone 502, or the electronic device communicating with the earphone 502, is ready to receive a response to the voice notification 566-3, and to urge the user 500 to provide a response. In some embodiments, the electronic device communicating with the earphone 502 is a smartwatch (as shown in Figures 5T-5Y). In some embodiments, the electronic device communicating with the earphone 502 is a smartphone (as shown in Figures 5Z-5AA).

[0167] Figure 5T shows the transition from Figure 5S. Figure 5T shows an electronic device 100 communicating with the earphone 502. Device 100 includes a touchscreen 112 (also called a “touch-sensitive display”). In response to hearing the sound effect 582 played in Figure 5S following the voice notification 566-3, user 500 provides a touch input 584 on the touchscreen 112. In response to detection of the touch input 584, the selection indicator 586 indicates the selection of the first of one or more response options for responding to the calendar event invitation 560-3. In response to detection of the touch input 584, device 100 begins audio playback of the response options for responding to the calendar event invitation 560-3. Specifically, Figure 5T shows that in response to a touch input 584 (or, more generally, a change in the touch input 584), the selection indicator 586 indicates the selection of an option (e.g., “Accept”) for accepting a calendar event invitation 560-3, and that an audio notification 588-1 with the response option “Accept” is played to the user 500 via the earphone 502. In some embodiments, the response options are displayed on the touchscreen 112 along with the audio playback of the response options, as in the example shown in Figure 5T. In some embodiments, to reduce power consumption by the touchscreen 112, the response options are not displayed on the touchscreen 112, and the user 500 hears the response options provided audibly without looking at the touchscreen 112. In some embodiments, (as described herein with reference to, for example, Figure 5AB) any one or more of the response options are automatically played after the audio notification 566-3 is played and before the sound effect 582 is played, to indicate that the device 100 is ready to receive a response.In some embodiments, not all response options are automatically played (for example, as shown in Figure 5AB), and any unplayed response options can be accessed and played in response to inputs directed to the earphone 502 (for example, tap input or double tap input of the earphone 502, or audio input to the earphone 502 via the microphone of the earphone 502), or to inputs to the device 100 (for example, touch input 584 as described herein with reference to Figures 5T-5V, or rotation of the dial 592 as described herein with reference to Figures 5W-5Y).

[0168] Next, in Figure 5U, in response to further movement of the touch input 584, the selection indicator 586 changes to indicate the selection of an option to decline the calendar event invitation 560-3 (e.g., "Deny"), and an audio notification 588-2 with the response option "Deny" is played to the user 500 via the earphone 502. Next, in response to further movement of the touch input 584, the selection indicator 586 changes to indicate the selection of an option to tentatively accept the calendar event invitation 560-3 (e.g., "Tentative"), and an audio notification 588-3 with the response option "Tentative" is played to the user 500 via the earphone 502.

[0169] Next, in Figure 5V, after voice notification 588-3 is provided to user 500 (and before different voice notifications with different response options are provided), and while selection indicator 586 indicates the selection of an option to tentatively accept calendar event invitation 560-3, touch input 584 is removed from touchscreen 112 at an initial time, as indicated by timer 590-1. After touch input 584 is removed from touchscreen 112 for at least a predetermined period (e.g., several seconds), as indicated by timer 590-2, the selection of an option to tentatively accept calendar event invitation 560-3 is confirmed, and voice notification 588-4 is provided to indicate that calendar event invitation 560-3 has been tentatively accepted. In some embodiments, the selection of each response option is confirmed in response to a second input (e.g., a tap input on touchscreen 112) detected after voice notification 566-3 is provided and before different voice notifications with different response options are provided. In some embodiments, where the selected response option is displayed on the touchscreen 112, the selection of the response option is confirmed when a second input is detected while the selection indicator 586 indicates the selection of the response option. In some embodiments, where one or more response options are displayed simultaneously on the touchscreen 112, each response option is selected and confirmed in response to a touch input (e.g., a tap input) on the touchscreen 112 at the location corresponding to each response option.

[0170] Figure 5W shows an alternative transition from Figure 5S. Figure 5W shows an electronic device 100 communicating with the earphone 502. Device 100 includes a dial 592 (a rotatable input mechanism, such as the crown of a watch) and a display 112 (optionally a touch-sensitive display, such as the touchscreen 112 in Figure 5U). In response to hearing the sound effect 582 played in Figure 5S following the voice notification 566-3, user 500 rotates the dial 592. In response to the detection of the rotation of the dial 592, the selection indicator 586 indicates the selection of the first response option from one or more response options for responding to the calendar event invitation 560-3. In response to the detection of the rotation of the dial 592, device 100 begins to play the voice of the response options for responding to the voice notification 560-3. Specifically, Figure 5W shows that in response to the rotation of the dial 592, the selection indicator 586 indicates the selection of an option (e.g., "Accept") for accepting the calendar event invitation 560-3, and that an audio notification 588-1 with the response option "Accept" is played to the user 500 via the earphone 502. In some embodiments, as in the example shown in Figure 5W, the response options are displayed on the display 112 along with the audio playback of the response options. In some embodiments, to reduce power consumption by the display 112, the response options are not displayed on the display 112, and the user 500 hears the response options provided audibly without looking at the display 112.

[0171] Next, in Figure 5X, in response to further rotation of the dial 592, the selection indicator 586 changes to indicate the selection of an option to decline the calendar event invitation 560-3 (e.g., "Declin"), and an audio notification 588-2 with the response option "Declin" is played to the user 500 via the earphone 502. Next, in response to further rotation of the dial 592, the selection indicator 586 changes to indicate the selection of an option to tentatively accept the calendar event invitation 560-3 (e.g., "Tentative"), and an audio notification 588-3 with the response option "Tentative" is played to the user 500 via the earphone 502.

[0172] Next, in Figure 5Y, when an audio notification 588-3 is provided to the user 500, an input 594 (e.g., a tap input or a press input) is detected via the dial 592 while the selection indicator 586 indicates the selection of an option to tentatively accept the calendar event invitation 560-3. In response to the press input 594 via the dial 592, the selection of an option to tentatively accept the calendar event invitation 560-3 is confirmed, and an audio notification 588-4 is provided to indicate that the calendar event invitation 560-3 has been tentatively accepted. In some embodiments, the selection of each response option is confirmed when no further input is detected via the dial 592 for at least a predetermined period (e.g., several seconds) after detecting the rotation of the dial 592 that caused the selection of each response option, or after hearing the playback of each response option. In some embodiments, where the selected response option is displayed on the display 112, the selection of the response option is confirmed when the selection indicator 586 remains on each response option for at least a predetermined period. In some embodiments, the device 100 includes a touch-sensitive surface, and if one or more response options are displayed simultaneously on the touchscreen 112, each response option is selected and confirmed in response to a touch input (e.g., a tap input) on the touch-sensitive surface at the location corresponding to each response option.

[0173] Figure 5Z shows an alternative transition from Figure 5S. Figure 5Z shows an electronic device 100 communicating with the earphone 502. The device 100 includes a display 112. In response to hearing the sound effect 582 played in Figure 5S following the voice notification 566-3, the user 500 raises the device 100 so as to change the orientation of the device 100 by at least a predetermined amount (for example, by a predetermined degree such as at least 30 degrees, from horizontal to at least a predetermined degree above horizontal such as 30 degrees above horizontal, or from within a predetermined range of horizontal such as within 30 degrees horizontal to outside a predetermined range) (for example, raising the upper edge 100a of the device 100 relative to the bottom edge 100b of the device 100).

[0174] In response to the detection of a change in the posture of device 100, a set of response options for responding to the voice notification 566-3 is displayed on the display 112 of device 100 (e.g., the options "Accept," "Reject," "Tentative," and "Skip"). In some embodiments, the set of response options is displayed according to the determination that device 100 has risen within a predetermined period (e.g., a few seconds) from the time when the earphone 502 began to output the sound effect 582.

[0175] Next, in Figure 5AA, after browsing the response options on the display 112, while the sound effect 582 is played via the earphone 502, the user 500 responds to the calendar event invitation 560-3 by aloud saying his desired response option, as indicated by the "acceptance" in the response (speech) 596. In response to detecting the response 596 (e.g., via the earphone 502), the device 100 receives and processes the selection of an option to accept the calendar event invitation 560-3. In response to this, an audio notification 598 is provided to indicate that the calendar event invitation 560-3 has been accepted.

[0176] In some embodiments, the earphone 502 stops playing the sound effect 582 after a predetermined period of time. In some embodiments, after the earphone 502 stops playing the sound effect 582, the device 100 no longer receives responses to the calendar event invitation 560-3 via the input mechanism described with reference to Figures 5T to 5AA. In some such embodiments, the user 500 may instead respond to the calendar event invitation 560-3 by interacting with one or more user interfaces of the device 100 to navigate to, view, and manually respond to a visual notification corresponding to the event 560-3 on the touch-sensitive display of the device 100.

[0177] Figure 5AB illustrates the adjustment of audio output volume in response to ambient noise level. In Figure 5AB, an audio notification 5100 containing information about event 560-3 in linguistic form is provided to the user 500 via the earphone 502. The ambient noise element 5102 indicates a high ambient noise level (for example, in contrast to Figure 5S, where the ambient noise element 5102 is absent). In some embodiments, the ambient noise level is detected via one or more microphones of the earphone 502 or an electronic device 100 communicating with the earphone 502. Since the ambient noise level in Figure 5AB is high, the output volume of the audio notification 5100 is increased so that the user 500 can hear the audio notification 5100 against the ambient noise (as shown in bold capital text, in contrast to the normal text of the audio notification 566-3 in Figure 5S).

[0178] In the example shown in Figure 5AB, a subset of response options (corresponding to the most likely response) for responding to the calendar event invitation 560-3 is also provided to the user 500 via the earphone 502. Specifically, after information about event 560-3 is provided, the user 500 is presented with a choice between two options: "Accept" and "Reject" (for example, the phrase "Accept or reject?").

[0179] After a subset of response options has been provided, earphone 502 plays an acoustic effect 5104 (e.g., white noise) to indicate that earphone 502 or an electronic device communicating with earphone 502 is ready to receive a response to the voice notification 5100, and to urge user 500 to provide a response. Since the ambient noise level is high, the output volume of the acoustic effect 5104 is also increased so that user 500 can hear the acoustic effect 5104 against the ambient noise (as shown in bold uppercase text, in contrast to the lowercase text of acoustic effect 582 in Figure 5S). User 500 may respond to the voice notification 5100 in any of the manners described herein, such as by speech (Figure 5AA). Alternatively, user 500 may first request to hear additional response options in any of the manners described herein, such as by providing input via device 100 (as described herein with reference to Figures 5T-5Z, for example), before providing or confirming a selection of response options.

[0180] Figure 5AC illustrates the delay in voice notifications while user 500 is engrossed in a conversation. At time t0, one or more events are detected (e.g., one or more notifications are received). Also at time t0, it is determined that user 500 is engrossed in a conversation (e.g., by detecting speech via one or more microphones of the earphone 502 or the electronic device 100 communicating with earphone 502). Therefore, while user 500 is engrossed in the conversation, the output of voice notifications corresponding to the detected events is delayed. At time t1, user 500 is no longer engrossed in the conversation (e.g., earphone 502 or device 100 detects that the speech has ended). Therefore, following the detection of the end of the speech, the delayed voice notification 5106 is provided to user 500 via earphone 502 at time t2. In some embodiments, as shown in Figure 5AC, the voice notification 5106 includes an audio summary (e.g., a summary of the content of one or more events) of one or more events detected while the user 500 is engrossed in a conversation (e.g., using simulated speech such as a text-to-speech conversation). In some embodiments, the voice notification 5106 includes a beep or audio tone without providing an audio summary of one or more events.

[0181] Figure 5AD illustrates the suppression of redundant voice notifications when the device includes or communicates with multiple audio output devices. In Figure 5AD, a map navigation event 5108 corresponding to turn-by-turn directions is detected. While the earphone 502 is not worn in the user 500's ears, voice notifications 5110, including linguistic instructions, are provided through one or more speakers of the device 100. Optionally, along with the provided voice notifications 5110, visual notifications 5112 are displayed on the display 112 of the device 100. While the earphone 502 is worn in the user 500's ears, voice notifications 5114 are provided through the earphone 502, along with the optionally displayed visual notifications 5112.

[0182] Figures 6A-6E are flowcharts illustrating a method 600 for providing voice notifications to an audio output device worn by a user, according to several embodiments. The method 600 is performed in an electronic device (e.g., a portable multifunction device 100, Figure 1A, or device 300, Figure 3A) that communicates (or includes) (e.g., via a wired or wireless connection) with one or more wearable audio output devices (e.g., wearable audio output devices 301, Figure 3B, such as in-ear earphones or over-ear headphones, separate from the electronic device) and one or more sensors (e.g., placement sensors (one or more), Figure 3B) for detecting the placement of the one or more wearable audio output devices. In some embodiments, the sensors are placed on the wearable audio output devices (e.g., as shown in Figure 3B). In some embodiments, the sensors are placed on the electronic device. In some embodiments, the device includes a display (e.g., touchscreen 112, Figure 1A, or display 340, Figure 3A) and a touch-sensitive surface (e.g., touchscreen 112, Figure 1A, or touchpad 355, Figure 3A). In some embodiments, the device communicates with or includes one or more additional audio output devices (e.g., speaker 111 on device 100, Figure 1A, or speaker(s) 306 on wearable audio output device 301, Figure 3B). In some embodiments, the device includes one or more audio input devices (e.g., microphone 113, Figure 1A, or microphone(s) 302, Figure 3B). In some embodiments, the device includes a rotatable input mechanism (e.g., as part of another input controller(s) 160 in Figure 1A), which may be used to provide input to the device by rotating an input mechanism (e.g., the crown of a watch). In some embodiments, the device includes one or more accelerometers or attitude sensors (e.g., one or more accelerometers 167, one or more gyroscopes 168, and / or one or more magnetometers 169, Figure 1A) for detecting the movement and / or changes in the orientation of the device.In some embodiments, the wearable audio output device(s) communicating with the device include one or more accelerometers or attitude sensors (e.g., as part of another input device(s) 306 in Figure 3B) for detecting changes in the movement and / or orientation of the wearable audio output device(s). Some operations in Method 600 are optionally combined, and / or the order of some operations is optionally changed.

[0183] As described below, Method 600 automatically provides a user wearing one or more wearable audio output devices (single or multiple headphones or in-ear earphones) with audio notifications of events occurring on related electronic devices while the user is not wearing the wearable audio output devices. By automatically providing audio notifications without requiring further user input when certain conditions are met (e.g., a specific arrangement of the wearable audio output devices), it makes it easier for the user to obtain information about events they may have missed, reduces the scope of device use and processing, thereby improving device usability, making the user device interface more efficient (e.g., by helping the user obtain desired information and reducing user errors), and improving the battery life of the device.

[0184] While one or more wearable audio output devices are not positioned relative to the user's ears, the device detects the occurrence of multiple events without outputting notifications corresponding to the events via one or more audio output devices (602).

[0185] After detecting the occurrence of multiple events, the device detects, via one or more (placement) sensors, that one or more wearable audio output devices are positioned relative to the user's ear (for example, inside the user's ear, on the user's ear, or near the user's ear, as described with reference to Figures 5A and 5G-5H) (604).

[0186] In response to one or more wearable audio output devices being positioned relative to the user's ears via one or more (placement) sensors, the device outputs information via one or more wearable audio output devices, including one or more voice notifications corresponding to any one or more of a plurality of events (606).

[0187] In some embodiments, one or more audio notifications include audio summaries of multiple events (e.g., audio summary 506, Figure 5A) (608). By providing audio summaries of events that occurred while the user was not wearing the wearable audio output device(s), the user can understand missed event information more quickly and easily, reducing the need for the user to look at visual notifications displayed on the electronic device's display, thereby reducing the number of inputs that need to be processed and the time the display needs to be powered on, thereby improving the usability of the device, reducing power usage, and improving the battery life of the device.

[0188] In some embodiments, the voice summary includes audio content according to the determination that the length of audio content representing multiple events does not exceed a predetermined length (610), and the voice summary does not include all of the audio content according to the determination that the length of audio content representing multiple events exceeds a predetermined length (e.g., voice notifications 564 and 566 that summarize the corresponding events rather than outputting all of the associated audio content, Figure 5H). By reducing the amount of content provided in the voice summary of an event when the total length of the audio content (e.g., the predicted total length) is too long (e.g., longer than a predetermined length), the intrusiveness of the voice summary is reduced, and event information is presented more efficiently while reducing the chances of user distraction. This reduces the number of inputs that need to be processed and the time the display needs to be powered on, thereby reducing the need for the user to check the notification one or more times, whether audibly or visually on the device's display, thereby improving device usability, reducing power usage, and improving device battery life.

[0189] In some embodiments, the information includes one or more audio notifications corresponding to one or more events (e.g., notifications received, time-based alerts, proximity-based alerts, calendar events, etc.) associated with each time period (e.g., during a particular day in which one or more wearable audio output devices are positioned relative to the user's ears) (e.g., during a particular day in which one or more wearable audio output devices are positioned relative to the user's ears) (612). In some embodiments, each time period is a time period that begins in the past (e.g., the start of the day, or the time when one or more wearable audio output devices were last detected in their respective positions relative to the user's ears) and continues to the present time (e.g., the time when one or more wearable audio output devices are positioned relative to the user's ears). In some embodiments, each time period includes (e.g., in addition to the time periods described above) a time period that begins in the present time (e.g., the time when one or more wearable audio output devices are positioned relative to the user's ears) and continues to a future time (e.g., the end of the day, etc.). For example, the voice summary 506 (Figure 5A) includes voice notifications corresponding to events that the user 500 missed before putting on the earphones 502, and voice notifications corresponding to events on a specific day, including future events.

[0190] By summarizing events associated with a given time period (such as a specific day when one or more audio output devices were placed in or near the user's ear, or a time period before or after the placement of one or more audio output devices in the user's ear), users can more quickly and easily understand event information for a given time period (such as missed events or upcoming events). This reduces the number of inputs that need to be processed and the time the display needs to be powered on, thereby reducing the need for the user to check missed notifications or upcoming events, whether audibly or visually on the device's display. This improves device usability, reduces power consumption, and improves the device's battery life.

[0191] In some embodiments, outputting one or more audio notifications corresponding to one or more events associated with each time period in response to detection that one or more wearable audio output devices are positioned in their respective locations relative to the user's ears is performed according to the determination that one or more wearable audio output devices are positioned in their respective locations relative to the user's ears at an early time between each time period (614) (for example, as described herein with reference to Figure 5A, one or more wearable audio output devices are not pre-positioned in their respective locations relative to the user's ears during each time period). In some embodiments, after detecting that one or more wearable audio output devices are positioned in their respective locations relative to the user's ears, the device detects via one or more sensors that one or more wearable audio output devices are positioned in their respective locations at a later time (or time) between each time period (for example, one or more wearable audio output devices are removed from their respective locations and then repositioned). In some embodiments, in response to detection via one or more sensors that one or more wearable audio output devices are positioned at a later time (or time) between each time period, the device stops outputting audio notifications corresponding to one or more events associated with each time period. For example, even if the earphone 502 is subsequently removed and placed back into the user 500's ear later that same day, the voice summary 506 (Figure 5A) will not be provided (however, voice notifications for events missed during the intervening time when the earphone 502 was not in the user 500's ear may still be provided).By providing an audio notification of an event only once at predetermined time intervals, the redundancy of event information provided to the user is reduced, thereby reducing the intrusiveness of the audio notification and reducing device processing (e.g., by reducing the number of times the audio notification needs to be provided to the user), thereby improving the usability of the device and making the user device interface more efficient (e.g., by helping the user achieve the intended result when operating / interacting with the device and reducing user errors), and further reducing power consumption and improving the device's battery life by enabling the user to use the device more quickly and efficiently.

[0192] In some embodiments, the information includes one or more audio notifications (e.g., upcoming events as shown in the audio summary 506 of Figure 5A) corresponding to one or more events that are scheduled to occur or are expected to occur within a predetermined period following the placement of one or more wearable audio output devices to the respective positions relative to the user's ears (616). In some embodiments, the predetermined period corresponds to the remainder of a particular day in which one or more wearable audio output devices are placed to the respective positions relative to the user's ears. By automatically providing audio notifications of upcoming events, the need for the user to provide input to the device to confirm event information is reduced, and the need for the user to visually view event information on the electronic device's display is reduced, thus decreasing the number of inputs that need to be processed and the time that the display needs to be powered on, thereby improving the usability of the device, reducing power usage, and improving the battery life of the device.

[0193] In some embodiments, outputting one or more audio notifications includes outputting a first audio notification corresponding to a first event among a plurality of events (618). In some embodiments, according to the determination that the electronic device is in a first context, the first audio notification includes first audio content corresponding to the first context, and according to the determination that the electronic device is in a second context different from the first context, the first audio notification includes second audio content corresponding to the second context, which is different from the first audio content. For example, as described herein with respect to Figure 5D and discussed further below, the first and second contexts may be a first geographic location and, with respect to a geographic area including the first geographic location, separate first and second geographic locations or first and second geographic areas. Another example is described below. Context-dependent presentation of event information within voice notifications provides users with information that is useful in a given context of the electronic device (and may not be useful in a different device context), thus reducing redundancy in the event information provided to the user while simultaneously providing sufficient information, thereby providing users with improved feedback, enhancing device usability, and making the user device interface more efficient (for example, by helping users achieve intended results when operating / interacting with the device and reducing user errors), and further reducing power consumption and improving device battery life by enabling users to use the device more quickly and efficiently.

[0194] In some embodiments, the first event corresponds to a calendar event occurring at a first geographic location (and, in some embodiments, having an event time and / or event title) (620), where the first geographic location includes location information for each of several hierarchical levels in the location hierarchy. In some embodiments, the determination that an electronic device is in a first context includes determining that the location of the electronic device is within the same first geographical area as the first geographic location (e.g., within the same building, within the same block, within the same city, etc.), where the first geographical area corresponds to a first hierarchical level in the location hierarchy. In some embodiments, the first audio content includes location information for each of the hierarchical levels lower than the first hierarchical level in the location hierarchy. In some embodiments, the first range is the minimum range surrounding the first geographic location, which also includes the location of the electronic device. In some embodiments, the first audio content includes an event time and / or event title. In some embodiments, the first audio content excludes location information for each of the hierarchical levels above the first hierarchical level.

[0195] In some embodiments, determining that an electronic device is in a second context includes determining that the electronic device's location is within the same second geographical region as the first geographical location, the second geographical region being distinct from the first geographical region and corresponding to a second hierarchical level in the location hierarchy that is different from the first hierarchical level. In some embodiments, the second audio content includes location information for each hierarchical level lower than the second hierarchical level in the location hierarchy. In some embodiments, the second geographical region is the minimum range surrounding the first geographical location, which also includes the location of the electronic device. In some embodiments, the second audio content includes event time and / or event title in addition to the aforementioned location information for each hierarchical level lower than the second hierarchical level in the location hierarchy. In some embodiments, the second audio content excludes location information for each hierarchical level higher than the second hierarchical level.

[0196] When a device user is within the same specific geographical area as their destination (such as a meeting place), providing information useful to the user for navigating to the destination without providing external location information for a larger geographical area reduces redundancy in the event information provided to the user while simultaneously providing sufficient information. This provides the user with improved feedback, enhances device usability, and makes the user device interface more efficient (for example, by helping the user achieve intended results when operating / interacting with the device and reducing user errors). Furthermore, this reduces power consumption and improves the device's battery life by enabling the user to use the device more quickly and efficiently. The context-dependent presentation of geographical information associated with events is described in more detail herein with reference to Figure 5D.

[0197] In some embodiments, a first event corresponds to a first message received from a first contact (622). In some embodiments, the determination that an electronic device is in a first context includes determining that a first part of the identifier of the first contact (e.g., nickname or given name) is the same as a first part of the identifier of at least one other contact in a list of contacts stored in the electronic device (or, in some embodiments, that the user has not communicated with the first contact within a previously defined time, such as within one year). In some embodiments, the first audio content includes a first part of the identifier of the first contact, a second part of the identifier of the first contact (e.g., last name or surname), and the content of the first message. In some embodiments, the determination that an electronic device is in a second context includes determining that a first part of the identifier of the first contact (e.g., nickname or given name) is not the same as a first part of the identifier of any other contact in the list of contacts. In some embodiments, the second audio content includes the first part of the identifier of the first contact and the content of the first message, but without the second part of the identifier.

[0198] In some embodiments, the determination that an electronic device is in a first context further includes determining that the user of the electronic device has communicated with a first contact within at least a previously defined time (e.g., the user has sent or received a message to a first contact within the last year). In some embodiments, the first audio content excludes the second part of the identifier of the first contact (e.g., last name) if there is no other name in the user's contact list that has the same first part of the user's contact list identifier (e.g., nickname or given name). When the identity of the message sender can be inferred from specific identifying information and context about the sender (e.g., because the user has recently communicated with that sender and / or because the sender has a unique name in the user's address book or another form of contact list or another source(s) of contact information applicable to the user), providing only specific information about the sender without providing irrelevant identifying information reduces redundancy in the event information provided to the user while still providing sufficient information, thereby reducing device processing, thereby improving device usability, reducing power consumption, and improving the device's battery life. The context-dependent inclusion / omission of some contact names is described in more detail herein, with reference to at least Figures 5B-5C.

[0199] In some embodiments, or in some circumstances, the first event corresponds to a first message from a first contact (for example, the device determines that the first event corresponds to each message from the first contact) (624). In some embodiments, the determination that an electronic device is in a first context includes determining that the first event does not follow a second event of a plurality of events that corresponds to a second message from the first contact (for example, a second message sent or received before the first message). In some embodiments, the first audio content includes the identifier of the first contact and the content of the first message. In some embodiments, or in some circumstances, the first event does not follow a second event of a plurality of events that corresponds to a second message from the first contact for the following reasons (for example, the device determines as follows): The first event is the first of several events, the first event is followed by a second event, but the second event does not correspond to a second message, or both the first and second (e.g., previous) events correspond to messages from the first contact, but one or more intervening events are detected between the first and second events. In some embodiments, as described herein with reference to operation 622, the identifier of each contact includes a first part and optionally a second part.

[0200] In some embodiments, determining that an electronic device is in a second context includes determining that a first event does not follow a second event in a group of events corresponding to a second message from a first contact (e.g., a second message sent or received before the first message) (e.g., no intervening event is detected). In some embodiments, the second audio content includes the content of the first message but does not include the identifier of the first contact. When the identity of the sender of a given message can be inferred from the identification information provided in a previous message immediately preceding the given message (e.g., before an intervening event occurred), omitting the sender's identification information of a given message reduces the redundancy of event information provided to the user, thereby reducing device processing, improving device usability, reducing power consumption, and improving device battery life. The context-dependent inclusion / omission of some contact names is described in more detail herein with reference to at least Figures 5B-5C.

[0201] In some embodiments, or in some circumstances, the first event corresponds to a first message that is part of a group message session for each group (626) (for example, the device determines so). Examples of contexts determined at least partially on group messages are described herein with reference to Figure 5C. In some embodiments, the determination that an electronic device is in a first context includes determining that the first event does not follow a second event of a plurality of events that corresponds to a second message that is part of a group message session (for example, a second message as part of a group message session that is sent or received before or after the first message). In some embodiments, the first audio content includes the identifier for each group or group message session, and the content of the first message. In some embodiments, the first event does not follow a second event of a plurality of events that corresponds to a second message that is part of a group message session for the following reasons (for example, the device determines as follows): The first event is the first of several events; the first event is followed by the second event, but the second event does not correspond to a message that is part of a group message session; or both the first and second (e.g., previous) events correspond to messages that are part of a group message session, but one or more intervening events are detected between the first and second events.

[0202] In some embodiments, the group message session identifier includes the name of the group message session. In some embodiments, the group message session identifier includes the identifiers of one or more (or all) participants in the group message session. In some embodiments, the first message is from each contact, and the first audio content includes the identifier of the first contact. In some embodiments, as described herein with reference to operation 622, the identifier of each contact includes a first part and optionally a second part.

[0203] In some embodiments, the determination that an electronic device is in a second context includes determining that the first event is successive to a second event among a group of events (for example, no intervening event is detected) that corresponds to a second message that is part of a group message session (for example, a second message sent or received before or after the first message as part of a group message session). In some embodiments, the second audio content includes the content of the first message and does not include the identifier of the respective group or group message session. In some embodiments, the first message is from a first contact and the second message is from a second contact different from the first contact, and therefore the first audio content includes the identifier of the first contact (for example, including a first part and optionally a second part) as described herein with reference to operation 622. In some embodiments, the first and second messages are from the same respective contact, and therefore the first audio content does not include the identifier of the first contact, as described herein with reference to operation 622.

[0204] When a given message is received as part of a group message session, and the group's identity can be inferred from the identification information provided in a preceding message immediately preceding the given message (e.g., before the intervening event occurred), omitting the group's identification information in the given message reduces the redundancy of event information provided to the user, thereby reducing device processing, improving device usability, reducing power consumption, and improving device battery life. Context-dependent inclusion / omission of group identifiers is described in more detail herein, with reference to at least Figures 5B-5C.

[0205] In some embodiments, outputting one or more audio notifications includes outputting a first audio notification corresponding to each of a first event among a plurality of events (628). In some embodiments, each of the first events corresponds to a first application associated with a first notification class, and each of the first audio notifications has a first structure defined by the first notification class. In some embodiments, the application specifies each notification class to which events associated with the application are associated. In some embodiments, each structure associated with each notification class is defined by an application-independent module (e.g., by the operating system of an electronic device or by an application programming interface). In some embodiments, the first structure specifies a first set of audio content to be included in each audio notification. In some embodiments, according to the determination that a second application associated with each of the second events is associated with a second notification class different from the first notification class, each of the second audio notifications has a different second structure (e.g., a second set of audio content specified to be included in each of the second audio notifications) defined by the second notification class.

[0206] For example, the voice notifications 538 and 548 for calendar event invitations 530 and 540 may have a notification structure defined by a calendar event notification class that defines the notification structure for calendar type events. Outputting voice notifications for events using the structure defined by the notification class to which the event is associated provides consistency in how event information is presented so that users can more easily understand the event information. In addition, using a normalized notification structure for voice notifications associated with a given notification class reduces the computation required to determine what information to include and output voice notifications, thereby improving device usability, reducing power usage, and improving device battery life.

[0207] In some embodiments, outputting one or more audio notifications includes outputting a second audio notification corresponding to each of the second events among a plurality of events (630). In some embodiments, each of the second events corresponds to a second application, and each of the second audio notifications has a second structure defined by the second application. In some embodiments, the second structure specifies a second set of audio content to be included in each of the second audio notifications. In some embodiments, according to the determination that each of the third events among the plurality of events is associated with a third application different from the second application, each of the third audio notifications has a different third structure defined by the third application (e.g., a third set of audio content specified to be included in each of the third audio notifications).

[0208] For example, the voice notifications 538 and 548 for calendar event invitations 530 and 540 may have a notification structure defined by the calendar application to which the event is associated. If the event is associated with different calendar applications, the event may have different notification structures defined separately by each calendar application. Outputting voice notifications using a structure defined or specified by the application to which the event is associated provides individual applications with greater flexibility in determining what information is present in the associated voice notification and takes into account the preferences and judgments of application developers regarding what information is useful to the user in the voice notification. In addition, using an application-specified structure for a given application's voice notification reduces the computation required to determine what information to include and output the voice notification, thereby improving device usability, reducing power consumption, and improving device battery life.

[0209] It should be understood that the specific order in which the operations are described in Figures 6A to 6E is merely an example, and is not intended to indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein. In addition, it should be noted that details of other processes described herein in relation to one or more other methods described herein (e.g., methods 700, 800, 900, and / or 1000) are also applicable in a similar manner to method 600 described above in relation to Figures 6A to 6E. For example, the devices, or their components, events, voice notifications, and inputs described herein with reference to method 600 may optionally have any one or more characteristics of the devices, or their components, events, voice notifications, and inputs described herein with reference to any one or more of the other methods described herein (e.g., methods 700, 800, 900, and / or 1000). For brevity, those details will not be repeated here.

[0210] Figures 7A-7C are flowcharts illustrating a method 700 that provides an audio context for conveying additional information about an event, according to several embodiments. The method 700 is performed in an electronic device (e.g., a portable multifunction device 100, Figure 1A, or device 300, Figure 3A, such as a telephone, computer, headphones, or one or more earbuds) that communicates with (or includes) one or more audio output devices (e.g., one or more speakers, such as the speaker 111 of device 100, Figure 1A, or the speaker(s) 306 of wearable audio output device 301, Figure 3B) (e.g., via a wired or wireless connection). In some embodiments, the device is one or more wearable audio output devices (e.g., a wearable audio output device 301, Figure 3B, such as in-ear earphones, over-ear headphones, etc., separate from the electronic device). In some embodiments, the device includes one or more sensors for detecting the placement of one or more audio output devices (e.g., one or more placement sensors 304, Figure 3B). In some embodiments, the device includes a display (e.g., touchscreen 112, Figure 1A, or display 340, Figure 3A) and / or a touch-sensitive surface (e.g., touchscreen 112, Figure 1A, or touchpad 355, Figure 3A). In some embodiments, the device includes one or more audio input devices (e.g., microphone 113, Figure 1A, or microphone(s) 302, Figure 3B). In some embodiments, the device includes a rotatable input mechanism (e.g., as part of another input controller(s) 160 in Figure 1A), which may be used to provide input to the device by rotating an input mechanism (e.g., the crown of a watch). In some embodiments, the device includes one or more accelerometers or attitude sensors (e.g., accelerometer(s) 167, gyroscope(s) 168, and / or magnetometer(s) 169, Figure 1A) for detecting the movement and / or changes in the orientation of the device.In some embodiments where the audio output device(s) are separate from the device and communicate with the device, the audio output device(s) include one or more accelerometers or attitude sensors (e.g., as part of another input device(s) 306 in Figure 3B) for detecting changes in the movement and / or orientation of the wearable audio output device(s). Some operations in Method 700 are optionally combined, and / or the order of some operations is optionally changed.

[0211] As described below, Method 700 modifies the audio output characteristics associated with different types of voice notifications (e.g., different spatial location, volume, content, etc.) associated with events having different characteristics (e.g., different sender / initiator, different value, etc.). Note that spatial location, sometimes called pseudo-spatial location, is a perceived characteristic of the audio output. Spatial location can be controlled or modified using well-known speech synthesis techniques so that the audio output is perceived as coming from a specific spatial location in three-dimensional space, distinct from the physical location of the speaker producing the audio output. Generally, at least two speakers are required to modify the spatial location of the audio output. By modifying the audio output of voice notifications based on the characteristics (one or more) of the associated event, it helps the user distinguish between different types of events and different event information. By providing improved feedback, the usability of the device is improved, and the user-device interface becomes more efficient (e.g., by helping the user achieve the intended result and by reducing user failures when operating / interacting with the device), which in turn allows the user to use the device more quickly and efficiently, reducing device power consumption and improving battery life.

[0212] The device detects the occurrence of an event (e.g., a request to play recent notifications, received messages, etc.) (702).

[0213] In response to detecting the occurrence of an event, the device outputs an audio notification via one or more audio output devices that includes audio content representing the event (e.g., audio content including the content of an incoming message, instructions for a user-to-user transaction, etc.) (704).

[0214] According to a determination that an event is associated with a first characteristic of a first set of characteristics (706) (e.g., the event was sent or initiated by a first user of multiple users, or the event is associated with a first value in a range of values ​​(e.g., a first transaction value)), the voice notification includes audio content representing the event and has first audio output characteristics selected based on the first characteristic (e.g., a first spatial location, a first volume, a first sound effect, etc.). According to a determination that an event is associated with a second characteristic of a first set of characteristics different from the first characteristic (e.g., a second user of multiple users, a second value in a range of values, etc.), the voice notification includes audio content representing the event and has second audio output characteristics selected based on the second characteristic and different from the first output characteristics (e.g., a second spatial location, a second volume, a second sound effect, etc.). For example, as described herein with reference to Figure 5C, voice notifications for messages from different senders may be associated with different spatial locations and output there to help users distinguish between the messages.

[0215] In some embodiments, the determination that an event is associated with a first characteristic of a first characteristic group includes the determination that the event was initiated by a first user among multiple users (708), and the determination that an event is associated with a second characteristic of the first characteristic group includes the determination that the event was initiated by a second user among multiple users. In some embodiments, the first audio output characteristic includes a first spatial position associated with the first user. In some embodiments, the first spatial position is independent of (e.g., selected independently of) the physical position of the first user. For example, the first spatial position may be a position relative to the position of a pair of earphones 502, as shown in Figure 5C. In some embodiments, the second audio output characteristic includes a second spatial position associated with a second user, different from the first position. In some embodiments, the second spatial position is independent of (e.g., selected independently of) the physical position of the second user. In some embodiments, the spatial relationship between a first spatial position and a second spatial position is independent of the spatial relationship between the physical position of the first user and the physical position of the second user. For example, as described herein with reference to Figure 5C, audio notifications of messages from participants in a group message may be associated with different spatial positions relative to the position of a pair of earphones 502 and output there, as shown in Figure 5C, in order to help the user distinguish between messages.

[0216] By associating events initiated by different users (e.g., different senders of an incoming message) with different spatial locations (e.g., different pseudo-spatial locations) and outputting corresponding audio notifications for events at these different spatial locations, it helps users (e.g., recipient users) identify and distinguish the initiators of the audio notifications. In addition, in some cases, using different spatial locations for different event initiators / senders reduces the amount of information that needs to be provided to the user linguistically, visually, or otherwise to help the user identify the sender of a particular event or message (e.g., one or more names of the sender may be omitted). These benefits reduce device processing by providing improved feedback to the user (e.g., by reducing the amount of information that needs to be provided to the user), thereby improving device usability, making the user-device interface more efficient (e.g., by helping users achieve intended results when operating / interacting with the device and reducing user errors), and also reduce power consumption and improve device battery life by allowing users to use the device more quickly and efficiently.

[0217] In some embodiments, the determination that an event is associated with a first characteristic of a first characteristic group includes the determination that the event was initiated by a first user of a plurality of users (710), and the determination that an event is associated with a second characteristic of a first characteristic group includes the determination that the event was initiated by a second user of a plurality of users. In some embodiments, the first audio output characteristic includes a first audio tone associated with the first user, and the second audio output characteristic includes a second audio tone, which is different from the first audio tone and is associated with the second user. Distinguishing voice notifications from different senders by varying the audio tone used to output voice notifications is described herein with reference to Figure 5C. By associating different audio tones with events initiated by different users (e.g., different senders of incoming messages) and incorporating these different audio tones into the corresponding voice notifications for events, it helps users identify and distinguish between the initiaters of voice notifications. In addition, in some cases, using different audio tones for different event initiators / senders reduces the amount of information that needs to be provided to the user verbally, visually, or otherwise to help the user identify the sender of a particular event or message (e.g., one or more names of the sender may be omitted). These benefits reduce device processing by providing improved feedback to the user (e.g., by reducing the amount of information that needs to be provided to the user), thereby improving device usability, making the user-device interface more efficient (e.g., by helping the user achieve the intended result when operating / interacting with the device and reducing user errors), and also reduce power consumption and improve device battery life by allowing the user to use the device more quickly and efficiently.

[0218] In some embodiments, the determination that an event is associated with a first characteristic of a first characteristic group includes the determination that the event was initiated by a first user of a plurality of users (712), and the determination that an event is associated with a second characteristic of a first characteristic group includes the determination that the event was initiated by a second user of a plurality of users. In some embodiments, outputting a voice notification having a first audio output characteristic that includes audio content representing an event includes outputting a voice notification that includes audio content representing an event using a first pseudo-voice associated with the first user (for example, regardless of whether the detected event includes voice communication from the first user). In some embodiments, outputting a voice notification having a second audio output characteristic that includes audio content representing an event includes outputting a voice notification that includes audio content representing an event using a second pseudo-voice, which is associated with the second user and differs from the first pseudo-voice (for example, regardless of whether the detected event includes voice communication from the second user). Distinguishing voice notifications from different senders by changing the pseudo-voice used to output the voice notification is described herein with reference to Figure 5C.

[0219] By associating different pseudo-voices with events initiated by different users (e.g., different senders of incoming messages) and using different pseudo-voices to output corresponding voice notifications, we help users identify and distinguish the initiators of voice notifications. In addition, in some cases, using different pseudo-voices for different event initiators / senders reduces the amount of information that needs to be provided to the user verbally, visually, or otherwise to help the user identify the sender of a particular event or message (e.g., one or more names of the sender may be omitted). These benefits reduce device processing by providing improved feedback to the user (e.g., by reducing the amount of information that needs to be provided to the user), thereby improving device usability, making the user-device interface more efficient (e.g., by helping the user achieve the intended result when operating / interacting with the device and reducing user errors), and also reduce power consumption and improve device battery life by allowing users to use the device more quickly and efficiently.

[0220] In some embodiments, the determination that an event is associated with a first characteristic of a first set of characteristics includes the determination that the event is associated with a first transaction value (e.g., a transaction between a user of a device and one or more other users or devices) (714), and the determination that an event is associated with a second characteristic of a first set of characteristics includes the determination that the event is associated with a second transaction value different from the first transaction value. In some embodiments, a first audio output characteristic includes a first sound effect representing a first transaction value, and a second audio output characteristic includes a second sound effect representing a second transaction value, different from the first sound effect. In some embodiments, the first sound effect includes each sound effect played at a first volume representing a first transaction value, and the second sound effect includes the same each sound effect played at a second volume representing a second transaction value. For example, a larger transaction value is represented by a sound effect played at a higher volume, and a smaller transaction value is represented by the same sound effect played at a lower volume. In some embodiments, the second sound effect includes a different type of sound effect than the first sound effect. Distinguishing between voice notifications for transactions of different values ​​by changing the sound effect and / or volume of the corresponding voice notification is described herein with reference to Figure 5E.

[0221] By associating different sound effects and / or different volumes with events associated with different transaction values, and outputting corresponding audio notifications using different sound effects and / or different volumes, the system helps the user identify and distinguish between received transaction values ​​(or relative values). In addition, in some cases, by using appropriate sound effects and / or volumes to communicate information about received transaction values ​​to the user, the system reduces the amount of information that needs to be provided to the user verbally, visually, or otherwise to help the user identify the transaction values ​​(e.g., details of the transaction values ​​may be omitted). These benefits reduce device processing by providing the user with improved feedback (e.g., by reducing the amount of information that needs to be provided to the user), thereby improving device usability, making the user-device interface more efficient (e.g., by helping the user achieve intended results when operating / interacting with the device and reducing user errors), and reducing power consumption and improving device battery life by allowing the user to use the device more quickly and efficiently.

[0222] In some embodiments, the determination that an event is associated with a first characteristic of a first characteristic group includes the determination that the event is a first type of notification (e.g., a first type of message, or a notification associated with a first application or group of applications) (716), and the determination that an event is associated with a second characteristic of a first characteristic group includes the determination that the event is a second type of notification different from the first type of notification (e.g., a second type of message, or a notification associated with a second application or group of applications). In some embodiments, the first audio output characteristic includes a first background audio associated with a first type of notification, and the second audio output characteristic includes a second background audio, different from the first background audio, associated with a second type of notification. By associating different background noises with different types of notifications and incorporating these different background noises into the corresponding voice notifications, it is possible to help the user identify and distinguish between the types of notifications. By providing users with improved feedback, device usability is enhanced, the user-device interface becomes more efficient (for example, by assisting users in achieving their intended results and by reducing user errors during device operation / interaction), which in turn allows users to use the device more quickly and efficiently, reducing device power consumption and improving battery life.

[0223] In some embodiments, the determination that an event is associated with a first characteristic of a first characteristic group includes the determination that the event is associated with a first type of message (e.g., a message between three or more users, sometimes referred to as a “group message”) (718), and the determination that an event is associated with a second characteristic of a first characteristic group includes the determination that the event is associated with a second type of message (e.g., a message between two users, sometimes referred to as a “individual message”), which is different from the first type of message. In some embodiments, the first audio output characteristic includes a first spatial location (e.g., a first pseudo-spatial location) associated with a first type of message, and the second audio output characteristic includes a second spatial location (e.g., a second pseudo-spatial location) associated with a second type of message, which is different from the first location. Distinguishing message types (e.g., individual vs. group messages) by outputting corresponding voice notifications at different spatial locations is described herein with reference to Figure 5C. Associating different types of messages with different spatial locations (e.g., different pseudo-spatial locations) and outputting corresponding voice notifications at different spatial locations helps users identify and distinguish between message types. By providing users with improved feedback, device usability is enhanced, the user-device interface becomes more efficient (for example, by assisting users in achieving their intended results and by reducing user errors during device operation / interaction), which in turn allows users to use the device more quickly and efficiently, reducing device power consumption and improving battery life.

[0224] In some embodiments, a first type of message corresponds to a message between the user of the electronic device and at least two other users (e.g., a "group message") (720), and a second type of message corresponds to a message between the user of the electronic device and up to one other user (e.g., an "individual message").

[0225] In some embodiments, the first spatial location is a pseudo-spatial location outside the user's head (722), and the second spatial location is a pseudo-spatial location inside the user's head. In some embodiments, outputting a voice notification at a pseudo-spatial location outside the user's head includes outputting the relevant audio content of the voice notification so that it sounds as if the audio content is coming from outside the user's head (or at least a predetermined distance from the user's head or ears) or being played outside. In some embodiments, outputting a voice notification at a pseudo-spatial location inside the user's head includes outputting the relevant audio content of the voice notification so that it sounds as if the audio content is being played inside the user's head.

[0226] Distinguishing individual messages from group messages by outputting corresponding audio notifications at pseudo-spatial locations "inside" and "outside" the user's head, respectively, is described herein with reference to Figure 5C. By presenting group messages so that they sound as if they are being played "outside" the user's head, while presenting individual messages at pseudo-locations so that they sound as if they are being played "inside" the user's head, the user is helped to distinguish between different types of messages, to focus more on messages directed at the user in particular, and to reduce distractions from messages in group message sessions that may not be directed at the user in particular. By providing the user with improved feedback, the usability of the device is improved, and the user-device interface becomes more efficient (for example, by helping the user achieve the intended result and by reducing user failures when operating / interacting with the device), which in turn allows the user to use the device more quickly and efficiently, reducing the device's power consumption and improving battery life.

[0227] In some embodiments, the electronic device includes a display and a touch-sensitive surface, and the device displays a user interface on the display (724). The user interface includes a representation of one or more audio output devices, and a first user interface element that represents a first type of message and is displayed at a first position in the user interface, the first position in the user interface defining a first spatial location associated with the first type of message, and a second user interface element that represents a second type of message and is displayed at a second position in the user interface, the second position in the user interface defining a second spatial location associated with the second type of message. In some embodiments, the device accepts input on the touch-sensitive surface that corresponds to the first user interface element (e.g., drag input, including touch at a position on the touch-sensitive surface corresponding to the first user interface element, and subsequent movement of the touch along the touch-sensitive surface). In some embodiments, in response to receiving input, the device moves a first user interface element to a third position within the user interface and associates a first type of message with a third spatial position defined by the third position within the user interface (and, in some embodiments, decouples a first type of message from a first spatial position defined by the first position within the user interface).

[0228] In some embodiments, outputting an audio notification containing audio content representing an event associated with a first type of message, after associating a first type of message with a third spatial location, includes outputting the audio content in such a way that it sounds as if it were coming from the third spatial location. In some embodiments, more generally, a user interface is provided that displays a user interface element for each event characteristic of each characteristic group, and each of the user interface elements is associated with its respective audio output characteristic. In some embodiments, the user may provide inputs directed to each user interface element on the user interface to modify the respective audio output characteristics associated with each event characteristic. For example, the user may modify the spatial location, volume, and / or type of sound effect associated with a particular event characteristic (e.g., initiating user, transaction value, message type, etc.). (Unless otherwise stated, “spatial location” as discussed herein is a pseudo-spatial location corresponding to the audio output characteristics of each audio notification.)

[0229] By providing a user interface that allows users to add or modify the spatial location associated with different types of messages, additional control options are offered, allowing users to customize their audio experience to be more familiar and recognizable, and reducing the amount of information that needs to be provided to the user in other ways to help them understand the information presented within voice notifications. These benefits reduce device processing by providing users with improved feedback (e.g., by reducing the amount of information that needs to be provided to the user), thereby improving device usability, making the user-device interface more efficient (e.g., by helping users achieve intended results when operating / interacting with the device and reducing user errors), and in addition, reducing power consumption and improving device battery life by allowing users to use the device more quickly and efficiently.

[0230] It should be understood that the specific order in which the operations are described in Figures 7A to 7C is merely an example, and is not intended to indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein. In addition, it should be noted that details of other processes described herein in relation to one or more other methods described herein (e.g., methods 600, 800, 900, and / or 1000) are also applicable in a similar manner to method 700 described above in relation to Figures 7A to 7C. For example, the devices, or their components, events, voice notifications, and inputs described above with reference to method 700 may optionally have any one or more characteristics of the devices, or their components, events, voice notifications, and inputs described herein with reference to one or more other methods described herein (e.g., methods 600, 800, 900, and / or 1000). For brevity, those details will not be repeated here.

[0231] Figures 8A-8D are flowcharts of a method 800 that performs an action based on the timing of user input to an audio notification, according to several embodiments. The method 800 is performed in an electronic device (e.g., a portable multifunction device 100, Figure 1A, or device 300, Figure 3A) that communicates with (or includes) (e.g., via a wired or wireless connection) one or more wearable audio output devices (e.g., wearable audio output devices 301, Figure 3B, such as in-ear earphones or over-ear headphones, separate from the electronic device) and one or more sensors (e.g., placement sensors (one or more), Figure 3B) for detecting the placement of the one or more wearable audio output devices. In some embodiments, the sensors are placed on the wearable audio output devices (e.g., as shown in Figure 3B). In some embodiments, the sensors are placed on the electronic device. In some embodiments, the device includes a display (e.g., touchscreen 112, Figure 1A, or display 340, Figure 3A) and a touch-sensitive surface (e.g., touchscreen 112, Figure 1A, or touchpad 355, Figure 3A). In some embodiments, the device communicates with or includes one or more audio output devices (e.g., speaker 111 on device 100, Figure 1A, or speaker(s) 306 on wearable audio output device 301, Figure 3B). In some embodiments, the device includes one or more audio input devices (e.g., microphone 113, Figure 1A, or microphone(s) 302, Figure 3B). In some embodiments, the device includes a rotatable input mechanism (e.g., as part of another input controller(s) 160 in Figure 1A), which may be used to provide input to the device by rotating an input mechanism (e.g., the crown of a wristwatch). In some embodiments, the device includes one or more accelerometers or attitude sensors (e.g., one or more accelerometers 167, one or more gyroscopes 168, and / or one or more magnetometers 169, Figure 1A) for detecting the movement and / or changes in the orientation of the device.In some embodiments, the wearable audio output device(s) communicating with the device include one or more accelerometers or attitude sensors (e.g., as part of another input device(s) 306 in Figure 3B) for detecting changes in the movement and / or orientation of the wearable audio output device(s). Some operations of Method 800 are optionally combined and / or the order of some operations is optionally changed.

[0232] As described below, Method 800 performs different actions in response to input directed to one or more wearable audio output devices based on whether an event was detected within a predetermined time period or subsequently in response to one or more audio notifications output in response to the detection of the event. In addition, by performing actions associated with an audio notification in response to detecting input within a predetermined time period in response to an audio notification, the user can quickly respond to or otherwise interact with the audio notification or the corresponding event. By performing different actions based on the timing of input directed to one or more wearable audio output devices and enabling the user to quickly respond to notifications, additional control options are provided to the user without requiring additional audio or visual output to notify the user of additional control options, thereby improving the usability of the device, making the user device interface more efficient (for example, by helping the user achieve intended results when operating / interacting with one or more wearable audio output devices and electronic devices, and by reducing user errors), and in addition, reducing power consumption and improving the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0233] The device detects the occurrence of the first event (802).

[0234] After detecting the occurrence of the first event, the device outputs one or more audio notifications corresponding to the first event via one or more wearable audio output devices (804).

[0235] After beginning to output one or more audio notifications, the device detects input directed to one or more wearable audio output devices (e.g., language input, touch input, device movement input such as accelerometer motion) (806).

[0236] In response to detecting an input directed to one or more wearable audio output devices (808), the device performs a first action according to a determination that an input was detected within a predetermined time period with respect to one or more audio notifications corresponding to a first event, the first action being associated with one or more audio notifications corresponding to a first event, and the device performs a second action according to a determination that an input was detected after a predetermined time period has elapsed with respect to one or more audio notifications corresponding to a first event, the second action being not associated with one or more audio notifications corresponding to a first event. For example, a tap input 554 (Figure 5F) detected outside a predetermined time period with respect to any audio notification causes the device to perform a different action (e.g., play music) than the action (e.g., listen to more information) caused by a tap input 572 (Figure 5L) detected within a predetermined time period with respect to audio notification 570.

[0237] In some embodiments, the device detects (810) a second input directed to one or more wearable audio output devices. In some embodiments, in response to detecting a second input directed to one or more wearable audio output devices, according to a determination that the second input was detected before starting to output one or more audio notifications corresponding to a first event, and according to a determination that a predetermined time period has elapsed with respect to one or more audio notifications corresponding to a second event (e.g., an event prior to the first event) that were output before the one or more audio notifications corresponding to the first event, the device performs a second operation, and the second operation is not associated with one or more audio notifications corresponding to the second event.

[0238] For example, additional tap inputs, such as tap input 554 (FIG. 5F), are detected outside of a predetermined time period with respect to any audio notification and cause the same operation to be performed (e.g., play music) as tap input 554. By performing an operation other than an operation regarding a specific audio notification in response to an input directed to the wearable audio output device(s) outside of a predetermined time period with respect to the specific audio notification, additional control options outside of the scope of notification control are provided to the user (e.g., to interact with other functions or applications of the electronic device) using the same input device, thereby improving the operability of the device, (e.g., making it possible for the user to achieve more intended results when operating / interacting with the wearable audio output device(s) and the electronic device and reducing user errors), making the user device interface more efficient, and in addition, reducing power usage and improving the battery life of the device by making it possible for the user to use the device more quickly and efficiently.

[0239] In some embodiments, after detecting the occurrence of a first event, the device detects the occurrence of a third event (812). In some embodiments, after detecting the occurrence of a third event and outputting one or more notifications corresponding to the first event, the device outputs one or more audio notifications corresponding to the third event via one or more wearable audio output devices. In some embodiments, in response to detecting input directed to one or more wearable audio output devices and according to the determination that input has been detected after the device has begun outputting one or more audio notifications corresponding to the third event, and within a predetermined time period with respect to one or more audio notifications corresponding to the third event, the device performs a first action, the first action being associated with one or more audio notifications corresponding to the third event.

[0240] For example, both head movements (Figure 5I) and tap inputs (Figure 5L) of user 500 detected within a predetermined time period for each voice notification (e.g., voice notifications 566 and 570, respectively) cause the same action (e.g., listen to more information) to be performed for each of the respective voice notifications. By performing the action associated with a particular voice notification in response to input directed to the wearable audio output device(s) within a predetermined time period for a particular voice notification (e.g., while the particular notification is being output), the user is provided with additional control options for responding to or interacting with a particular voice notification using the same input device, even if the voice notification is just one of a set of voice notifications being provided. By providing the user with improved control options, the usability of the device is improved, the user-device interface is made more efficient (e.g., by enabling the user to achieve more intended results when operating / interacting with the wearable audio output device(s) and electronic devices, and by reducing user errors), and in addition, power consumption is reduced and the battery life of the device is improved by enabling the user to use the device more quickly and efficiently.

[0241] In some embodiments, outputting one or more voice notifications corresponding to each event includes outputting a part of the audio content corresponding to each event (814), and performing a first action associated with one or more voice notifications corresponding to each event includes refraining from outputting the remaining part of one or more voice notifications corresponding to each event. Skipping the remaining part of the voice notification is described herein with reference to FIGS. 5N - 5P. By enabling the user to skip the remaining part of one or more voice notifications after listening to the first part, the possibility of presenting unnecessary or undesirable information to the user is reduced, and the time during which the audio output device(s) and / or electronic device(s) must be active can also be reduced, thereby improving the operability of the device and making the user device interface more efficient (e.g., by helping the user achieve the intended result when operating / interacting with the device and reducing user errors), and further by enabling the user to use the device more quickly and efficiently, reducing power usage and improving the battery life of the device.

[0242] In some embodiments, outputting one or more audio notifications corresponding to each event includes outputting a first portion of audio content corresponding to each event (816), and performing a first action associated with one or more audio notifications corresponding to each event includes outputting a second (e.g., additional) portion different from the first portion of audio content corresponding to each event. In some embodiments, the first portion of audio content corresponds to a shortened form of the audio notification corresponding to each event, and the second portion of audio content corresponds to an extended form of the audio notification, including audio content excluded from the shortened audio notification (such as a summarized, condensed, or truncated audio notification as described herein with reference to Method 600). Playing an extended form of an audio notification initially presented in a shortened form in response to user input directed to a wearable audio device while a condensed audio notification is being played or "immediately" afterward (e.g., within a predetermined time relative to the audio notification) is described herein with reference to Figures 5I-5J and 5L-5M.

[0243] By enabling users to hear an enhanced form of voice notification in response to user input directed at one or more wearable audio output devices, users can quickly obtain additional information about detected events without needing to view information on the electronic device's display. This improves device usability by reducing the number of inputs required to hear more information and the time the display needs to be powered on, making the user-device interface more efficient (for example, by helping users achieve intended results more quickly when operating / interacting with the device and reducing user errors), and also reduces power consumption and improves device battery life by enabling users to use the device more quickly and efficiently.

[0244] In some embodiments, performing a second action not associated with one or more voice notifications corresponding to each event includes, as described herein with reference to Figures 5F and 5Q-5R, invoking a virtual assistant or controlling media content playback (e.g., playing, pausing, resuming, stopping, rewinding, or fast-forwarding media content separate from one or more voice notifications corresponding to each event) (818). Performing an action other than the action associated with a particular voice notification in response to an input detected outside a predetermined time period with respect to a particular voice notification provides the user with additional control options outside the scope of notification control using the same input device (e.g., to interact with other functions or applications of the electronic device), thereby improving the usability of the device, making the user device interface more efficient (e.g., by enabling the user to achieve more intended results when operating / interacting with wearable audio output devices / electronic devices and reducing user errors), and in addition, reducing power consumption and improving the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0245] In some embodiments, the first action associated with one or more voice notifications corresponding to each event is based on the application associated with each event (e.g., specified in the application settings of the application) (820). By basing the actions performed for voice notifications on the application associated with the event corresponding to the voice notification, the user can provide relevant responses to events without requiring them to view the events on the electronic device's display. This provides the user with additional control options, reduces the time the display needs to be powered on, thereby improving the usability of the device, making the user-device interface more efficient (e.g., by helping the user achieve more intended results when operating / interacting with the device and reducing user errors), and in addition, reduces power consumption and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0246] In some embodiments, detecting each event includes receiving media content other than text content (822), and performing a first action associated with one or more audio notifications corresponding to each event includes providing a linguistic description of the media content other than text content. For example, in response to the reception of an image, the device provides one or more audio notifications, including a linguistic description of the image, as described herein with reference to Figure 5M. Providing linguistic descriptions of content other than text content (such as image content) provides the user with information about an additional type of content without requiring the user to view the content on the display of the electronic device, which is beneficial for visually impaired users. It improves the usability of the device by providing the user with improved audio feedback and reducing the time the display needs to be powered on, making the user-device interface more efficient (for example, by helping the user achieve the intended result when operating / interacting with the device and reducing user errors), and in addition, it reduces power consumption and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

[0247] In some embodiments, a first action is performed on a first voice notification according to a determination that input has been detected while the first voice notification of one or more voice notifications is being output (824). In some embodiments, input is detected after the first part of one or more voice notifications has been output, and in response, the device refrains from outputting the remainder of one or more voice notifications (e.g., skips them) (e.g., as described herein with reference to action 814). In some embodiments, a first action is performed on a second voice notification according to a determination that input has been detected while the second voice notification of one or more voice notifications different from the first voice notification is being output.

[0248] For example, if user 500's head movement (Figure 5I) is detected while the corresponding voice notification 566 is being output, it triggers the execution of an action for voice notification 566 (e.g., resulting in an action), while a tap input 572 (Figure 5L) is detected while the corresponding voice notification 570 is being output, it triggers the execution of an action for voice notification 570 (e.g., resulting in an action). By performing actions for a particular voice notification in response to input directed to the wearable audio output device(s) during that particular voice notification, the user can provide a response or otherwise interact with a particular voice notification, even if it is just one of a series of voice notifications that are being output. By providing the user with improved control options, the usability of the device is enhanced, the user-device interface is made more efficient (e.g., by helping the user achieve the intended result when operating / interacting with the wearable audio output device(s) and electronic devices, and by reducing user errors), and power consumption is reduced and the battery life of the device is improved by allowing the user to use the device more quickly and efficiently.

[0249] In some embodiments, according to the determination that the input is a first type of input (e.g., a single tap), the first action is a first type of action (e.g., a reply action or a “play more” action) (826), and according to the determination that the input is a second type of input (e.g., a double tap) that is different from the first type of input (e.g., a single tap), the first action is a second type of action that is different from the first type of action (e.g., a “skip” action). Performing different types of actions in response to different types of input received in response to a voice notification over a predetermined period of time or within a predetermined period of time is described herein with reference to Figures 5L-5M and 5N-5P. By performing different types of actions associated with a particular audio notification based on the type of input directed to the wearable audio output device(s) within a predetermined time period in response to a specific notification (e.g., while the specific notification is being output), the system provides the user with additional control options for responding to or interacting with a particular notification using the same input device, thereby improving the usability of the device, making the user device interface more efficient (e.g., by enabling the user to achieve more intended results when operating / interacting with wearable audio output devices(s) and electronic devices, and by reducing user errors), and in addition, reducing power consumption and improving the device's battery life by enabling the user to use the device more quickly and efficiently.

[0250] In some embodiments, according to the determination that the input is a first type of input (e.g., a single tap) (828), the second action is a third type of action (e.g., a "play / pause" action), and according to the determination that the input is a second type of input (e.g., a double tap) that is different from the first type of input (e.g., a single tap), the second action is a fourth type of action that is different from the third type of action (e.g., a "call virtual assistant" action). Performing different types of actions in response to different types of input received during or outside a predetermined period in response to any voice notification is described herein with reference to Figures 5F and 5Q-5R. By performing different types of actions other than those associated with a specific voice notification based on input directed to a wearable audio output device(s) outside of a predetermined time cycle for a particular voice notification, the system provides the user with additional control options outside the scope of notification control using the same input device (e.g., to interact with other functions or applications of the electronic device), thereby improving the usability of the device, making the user device interface more efficient (e.g., by enabling the user to achieve more intended results when operating / interacting with a wearable audio output device(s) and electronic devices, and by reducing user errors), and in addition, reducing power consumption and improving the device's battery life by enabling the user to use the device more quickly and efficiently.

[0251] In some embodiments, detecting input directed to one or more wearable audio output devices includes detecting a tap input (e.g., a first tap input) on one or more wearable audio output devices (830). In some embodiments, detecting input includes detecting a double tap input (e.g., a second tap input within a predetermined time period, ending with a first tap input). Performing actions in response to one or more tap inputs directed to one or more wearable audio output devices provides the user with control options using intuitive user interaction, thereby improving the usability of the device, making the user-device interface more efficient (e.g., by helping the user achieve intended results when operating / interacting with wearable audio output devices and electronic devices, and by reducing user errors), and in addition, reducing power consumption and improving the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0252] In some embodiments, where one or more wearable audio output devices include one or more accelerometers, detecting input directed to one or more wearable audio output devices includes detecting the movement of one or more wearable audio output devices via one or more accelerometers (832). Performing actions in response to accelerometer inputs is described herein with reference to Figures 5I–5K. Performing actions in response to accelerometer inputs provided via the movement of one or more wearable audio output devices provides the user with control options using intuitive user interaction (e.g., head movements of the user while wearing one or more wearable audio output devices), thereby improving the usability of the device, making the user-device interface more efficient (e.g., by helping the user achieve intended results when operating / interacting with wearable audio output devices and electronic devices, and by reducing user errors), and in addition, reducing power consumption and improving the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0253] In some embodiments, outputting one or more audio notifications corresponding to each event includes outputting a first portion of audio content corresponding to each event (834). In some embodiments, performing a first action in response to detecting movement of one or more wearable audio output devices in a first direction via one or more accelerometers (e.g., backward movement or tilt, or away from the pseudospatial location from which one or more audio notifications are output) includes ceasing to output a second portion of audio content corresponding to each event. In some embodiments, performing a first action in response to detecting movement of one or more wearable audio output devices in a second direction opposite to the first direction via one or more accelerometers (e.g., forward movement or tilt, or towards the pseudospatial location from which one or more audio notifications are output) includes outputting a second portion of audio content corresponding to each event. In some embodiments, the second portion of audio content includes additional audio content to the audio content that would have been output if no input directed to one or more wearable audio devices had occurred. Providing different amounts of audio content in response to the user's head movements toward or away from the pseudo-spatial location of sound is described herein with reference to Figures 5I-5K.

[0254] By varying the amount of audio content delivered based on the direction of movement of one or more wearable audio output devices, the system provides users with control options to hear additional information using intuitive user interaction (e.g., providing additional audio content when the user tilts in a particular direction, such as towards a simulated location / source of an audio notification, and less audio content when the user tilts in a different direction, such as away from the simulated location / source), thereby improving the usability of the device, making the user-device interface more efficient (e.g., by helping users achieve intended results when operating / interacting with wearable audio output devices and electronic devices, and by reducing user errors), and in addition, reducing power consumption and improving the battery life of the device by enabling users to use the device more quickly and efficiently.

[0255] In some embodiments, after detecting the occurrence of each event, the device refrains from outputting one or more notifications (e.g., audio, visual, and / or tactile) corresponding to each event, other than one or more audio notifications output via one or more wearable audio output devices (836). In some embodiments, the device includes or communicates with one or more audio output devices other than one or more wearable audio output devices, and the device refrains from outputting audio notifications corresponding to the first event via one or more other audio output devices. In some embodiments, the device includes one or more tactile output generators, and the device refrains from outputting tactile output notifications corresponding to the first event via one or more tactile output generators. In some embodiments, the device includes a display, and the device refrains from outputting visual notifications corresponding to the first event. Suppression of redundant audio notifications is described herein with reference to Figure 5AD.

[0256] By suppressing other types of notifications for events for which voice notifications are provided or will be provided in the future, redundant feedback that could distract or confuse the user is avoided, thereby improving device usability and making the user-device interface more efficient (for example, by helping the user achieve the intended results when operating / interacting with wearable audio output devices / electronic devices and reducing user errors), and in addition, reducing power consumption and improving device battery life by enabling the user to use the device more quickly and efficiently.

[0257] It should be understood that the specific order in which the operations are described in Figures 8A to 8D is merely an example, and is not intended to indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein. In addition, it should be noted that details of other processes described herein in relation to one or more other methods described herein (e.g., methods 600, 700, 900, and / or 1000) are also applicable in a similar manner to method 800 described above in relation to Figures 8A to 8D. For example, the devices, or their components, events, voice notifications, and inputs described above with reference to method 800 may optionally have any one or more characteristics of the devices, or their components, events, voice notifications, and inputs described herein with reference to one or more other methods described herein (e.g., methods 600, 700, 900, and / or 1000). For brevity, those details will not be repeated here.

[0258] Figures 9A-9C are flowcharts illustrating a method 900 for stimulating a user to receive a response to an audio notification, according to several embodiments. The method 900 is performed in an electronic device (e.g., a portable multifunction device 100, Figure 1A, or device 300, Figure 3A) that communicates with (or includes) (e.g., via a wired or wireless connection) one or more wearable audio output devices (e.g., wearable audio output devices 301, Figure 3B, such as in-ear earphones or over-ear headphones, separate from the electronic device) and one or more sensors (e.g., placement sensors (one or more), Figure 3B) for detecting the placement of the one or more wearable audio output devices. In some embodiments, the sensors are placed on the wearable audio output devices (e.g., as shown in Figure 3B). In some embodiments, the sensors are placed on the electronic device. In some embodiments, the device includes a display (e.g., touchscreen 112, Figure 1A, or display 340, Figure 3A) and a touch-sensitive surface (e.g., touchscreen 112, Figure 1A, or touchpad 355, Figure 3A). In some embodiments, the device communicates with or includes one or more audio output devices (e.g., speaker 111 on device 100, Figure 1A, or speaker(s) 306 on wearable audio output device 301, Figure 3B). In some embodiments, the device includes one or more audio input devices (e.g., microphone 113, Figure 1A, or microphone(s) 302, Figure 3B). In some embodiments, the device includes a rotatable input mechanism (e.g., as part of another input controller(s) 160 in Figure 1A), which may be used to provide input to the device by rotating an input mechanism (e.g., the crown of a wristwatch). In some embodiments, the device includes one or more accelerometers or attitude sensors (e.g., one or more accelerometers 167, one or more gyroscopes 168, and / or one or more magnetometers 169, Figure 1A) for detecting the movement and / or changes in the orientation of the device.In some embodiments, the wearable audio output device(s) communicating with the device include one or more accelerometers or attitude sensors (e.g., as part of another input device(s) 306 in Figure 3B) for detecting changes in the movement and / or orientation of the wearable audio output device(s). Some operations in Method 900 are combined at will, and / or the order of some operations is changed at will.

[0259] As described below, Method 900 outputs one or more voice notifications corresponding to events detected on the relevant electronic device, and subsequently a voice instruction that the electronic device is ready to receive the detected events, via one or more wearable audio output devices (e.g., worn by the user). By outputting voice notifications and voice instructions via one or more wearable audio output devices, the user is prompted to respond, and it is also shown that the response may be provided via one or more wearable audio output devices, thereby providing an alternative input mechanism that reduces the need for the user to interact with the electronic device, in particular by reducing the need for the user to view and respond to detected events using the device's display. This reduces the number of inputs that need to be processed and the time that the display needs to be powered on, thereby improving the usability of the device and making the user-device interface more efficient (e.g., by helping the user achieve the intended results when operating / interacting with the device and reducing user errors), and in addition, reducing power consumption and improving the battery life of the device by allowing the user to use the device more quickly and efficiently.

[0260] The device detects the occurrence of the first event (902).

[0261] After detecting the occurrence of the first event, the device outputs one or more voice notifications corresponding to the first event via one or more wearable audio output devices (904).

[0262] After outputting one or more voice notifications corresponding to the first event, the electronic device outputs a respective audio output separate from the one or more voice notifications indicating that it is ready to receive a response to the first event (906).

[0263] While outputting each respective audio output, the device detects an input corresponding to a response to the first event and generates a response to the first event based on the input (908).

[0264] In some embodiments, after outputting (or starting to output) one or more voice notifications corresponding to the first event, the device detects a second input corresponding to a request to respond to the first event (e.g., a tap input directed to the electronic device or one or more wearable audio output devices) via the electronic device or one or more wearable audio output devices (910). In response to the detection of the second input, a respective audio output indicating that the electronic device is ready to receive a response to the first event is output. In some embodiments, each respective audio output is output in accordance with a determination that the second input was detected within a predetermined time period while one or more voice notifications corresponding to the first event are being output or in relation to the one or more voice notifications (e.g., in response to the start of output of the one or more voice notifications or in relation to the volume of output of the one or more voice notifications).

[0265] By requiring the user to provide input corresponding to a request to respond to a first event, confirmation is provided that the user intends to provide a response to the first event. If confirmation is not received, careless input is not processed unnecessarily, thus preventing unintended actions from being performed on the electronic device. By providing improved processing and control options, device usability is enhanced, and the user-device interface becomes more efficient (for example, by assisting the user in achieving intended results and by reducing user failures during device operation / interaction), which in turn allows the user to use the device more quickly and efficiently, reducing device power consumption and improving battery life.

[0266] In some embodiments, each audio output includes white noise (for example, as described herein with reference to sound effects 582 and 5104 in Figures 5S-5AB) (912). In some embodiments, each audio output consists of white noise. By providing white noise to indicate that the device is ready to receive a response to an event for which an audio notification is output, the user is encouraged to respond using an intuitive voice, thereby improving the usability of the device and making the user-device interface more efficient (for example, by helping the user provide requested input when operating / interacting with the device and reducing user errors), and in addition, reducing power consumption and improving the battery life of the device by allowing the user to use the device more quickly and efficiently.

[0267] In some embodiments, where an electronic device communicates with one or more audio input devices (e.g., one or more microphones which are part of the electronic device or part of one or more wearable audio output devices), receiving input corresponding to a response to a first event includes detecting audio input via one or more audio input devices (as described herein with reference to response 596 in Figure 5AA, for example) (914). Receiving a response to an event detected via one or more audio input devices provides the user with an alternative input mechanism that accepts linguistic input (e.g., speech), which can be a more intuitive and natural way to provide input, and reduces the need to view and respond to detected events using another input device such as a touch-sensitive surface or display of the electronic device, thereby reducing the time these devices must be powered on and active, assisting visually impaired users, thereby improving the usability of the device, making the user device interface more efficient (e.g., by helping the user provide appropriate input to achieve intended results when operating / interacting with the device and reducing user errors), and further reducing power consumption and improving the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0268] In some embodiments, where the electronic device includes a touch-sensitive surface, while outputting each audio output and before detecting an input corresponding to a response to a first event (916), the device detects a first touch input via the touch-sensitive surface and, in response to the detection of the first touch input via the touch-sensitive screen, outputs one or more response options to the first event via one or more wearable audio output devices. In some embodiments, the input is a drag input, and different response options are output as the input moves across the touch-sensitive surface. In some embodiments, detecting an input corresponding to a response to a first event includes detecting a second touch input (e.g., a tap input) via the touch-sensitive surface that selects each of the one or more response options. In some embodiments, the second input selects each response option if the second input is detected while outputting each response option or within a specified time after outputting (and, in some embodiments, before starting to output subsequent response options). In some embodiments, the second input is the lift-off of the first touch input combined with the end of a predetermined period following the lift-off of the first touch input, during which no subsequent touch inputs are detected for a predetermined period. In some embodiments, the response generated for the first event corresponds to each selected response option. Outputting response options in response to touch inputs is described herein with reference to Figures 5T-5V.

[0269] By outputting response options to events detected via a wearable audio output device(s) in response to input on the touch-sensitive surface of an electronic device, users can hear the response options without needing to view them on the electronic device's display before responding. This reduces the time the display needs to be powered on, assisting visually impaired users, thereby improving device usability and making the user device interface more efficient (for example, by helping users achieve intended results when operating / interacting with the device and reducing user errors), and further reducing power consumption and improving the device's battery life by enabling users to use the device more quickly and efficiently.

[0270] In some embodiments, where the electronic device includes a rotatable input mechanism, while outputting each audio output and before detecting an input corresponding to a response to a first event (918), the device detects a first input (e.g., rotation of the rotatable input mechanism) via the rotatable input mechanism and, in response to detecting the first input via the rotatable input mechanism, outputs one or more response options to the first event via one or more wearable audio output devices. In some embodiments, detecting an input corresponding to a response to a first event includes detecting a second input (e.g., a tap input) via the rotatable input mechanism (or via other input devices or sensors of the electronic device, such as a touch-sensitive surface or an accelerometer) that selects each of the one or more response options. In some embodiments, the second input selects each response option if the second input is detected while outputting each response option or within a specified time after outputting (and, in some embodiments, before starting to output subsequent response options). In some embodiments, the responses generated to the first event correspond to each selected response option. Outputting response options in response to rotation of the rotatable input mechanism is described herein with reference to Figures 5W-5Y.

[0271] By responding to input via a rotatable input mechanism and outputting response options to respond to detected events via a wearable audio output device(s), the user can hear the response options without needing to view them on the electronic device's display before responding, and in some cases, this reduces the time the display needs to be powered on, assisting visually impaired users, thereby improving the usability of the device and making the user device interface more efficient (for example, by helping the user achieve the intended result when operating / interacting with the device and reducing user errors), and further by reducing power consumption and improving the device's battery life by enabling the user to use the device more quickly and efficiently.

[0272] In some embodiments, where the electronic device includes a display and one or more attitude sensors (e.g., one or more cameras, gyroscopes, inertial measurement units, or other sensors for detecting changes in the orientation of the electronic device), the device detects a change in the attitude (e.g., orientation) of the electronic device via one or more attitude sensors (corresponding to an upward tilt of the electronic device, such that the upper edge of the electronic device is raised relative to the bottom edge of the electronic device) (920). In some embodiments, in response to the detection of a change in the attitude of the electronic device, and according to the determination that a change in the attitude of the electronic device has been detected within a predetermined time period for one or more audio notifications corresponding to the first event (e.g., while one or more audio notifications are detected, or within a specified time after detection), the device displays one or more response options for the first event on the display (e.g., as described herein with reference to Figures 5Z-5AA). In some embodiments, detecting an input corresponding to a response to the first event includes detecting an input to select each of the one or more response options.

[0273] By displaying response options on the electronic device's display in response to detected events when the electronic device is lifted, users can quickly browse response options without having to wait for the options to be read aloud sequentially via a wearable audio output device(s). This reduces the number of inputs that need to be processed and the time that the wearable audio output device(s)(s) must be active, thereby improving the usability of the device and making the user-device interface more efficient (for example, by helping users achieve their intended results more quickly when operating / interacting with the device and reducing user errors), and in addition, reducing power consumption and improving the device's battery life by allowing users to use the device more quickly and efficiently.

[0274] In some embodiments, after outputting one or more voice notifications corresponding to a first event and before outputting each audio output indicating that the electronic device is ready to receive a response to the first event, the device outputs a number of response options via one or more wearable audio output devices (as described herein with reference to, for example, voice notification 5100 in Figure 5AB) that prompt the user of the electronic device to provide a response to the first event (922). By automatically outputting response options for responding to an event detected via one or more wearable audio output devices after outputting a corresponding voice notification, the number of inputs that need to be processed is reduced. By outputting each audio output indicating that the device is ready to receive a response after outputting the response options, the user can hear the response options before being prompted to respond, thereby reducing the need for the user to browse the response options on the device's display, thereby reducing the time the device's display needs to be powered on, which assists visually impaired users. These benefits improve the usability of the device, making the user device interface more efficient (for example, by helping users achieve their intended results when operating / interacting with the device and reducing user errors), which in turn reduces the device's power consumption and improves battery life by allowing users to use the device more quickly and efficiently.

[0275] In some embodiments, the first event is a calendar event invitation, and the multiple response options include an option to accept the calendar event invitation and an option to decline the calendar event invitation (for example, as described herein with reference to the voice notification 5100 in Figure 5AB). In some embodiments, the multiple response options further include an option to postpone the invitation (e.g., postpone responding to the invitation until a later time) and / or tentatively accept the invitation. In some embodiments, the detected input corresponding to the response to the first event includes a selection of each option among the multiple response options (e.g., recitation).

[0276] In some embodiments in which an electronic device communicates with one or more audio input devices, the device detects the volume of ambient noise to one or more audio input devices (926) and adjusts the output volume of one or more voice notifications based on the volume of ambient noise (as described herein with reference to, for example, Figure 5AB). By adjusting the output volume of voice notifications (and / or audio output indicating that the device is ready to receive a response to a corresponding event) based on the volume of ambient noise, the device helps the user hear the audio output by increasing the output volume in noisy environments, while reducing damage to the user's hearing by lowering the output volume in quiet environments where a high volume is not required for the user to hear the audio output against ambient noise, thereby providing the user with improved audio feedback. In addition, by lowering the output volume when a high volume is not required, the power usage of the audio output device is reduced. These benefits improve the usability of the device and make the user device interface more efficient (for example, by helping the user achieve intended results when operating / interacting with the device and reducing user errors), which further reduces the power usage of the device and improves battery life by making the device more quickly and efficiently usable by the user.

[0277] It should be understood that the specific order in which the operations are described in Figures 9A to 9C is merely an example, and is not intended to indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein. In addition, it should be noted that details of other processes described herein in relation to one or more other methods described herein (e.g., methods 600, 700, 800, and / or 1000) are also applicable in a similar manner to method 900 described above in relation to Figures 9A to 9C. For example, the devices, or their components, events, voice notifications, and inputs described above with reference to method 900 may optionally have any one or more characteristics of the devices, or their components, events, voice notifications, and inputs described herein with reference to one or more other methods described herein (e.g., methods 600, 700, 800, and / or 1000). For brevity, those details will not be repeated here.

[0278] Figures 10A-10B are flowcharts illustrating a method 1000 for conditionally outputting voice notifications based on user activity and context, according to several embodiments. The method 1000 is performed in an electronic device (e.g., a portable multifunction device 100, Figure 1A, or device 300, Figure 3A) that communicates with (or includes) (e.g., via a wired or wireless connection) one or more wearable audio output devices (e.g., wearable audio output devices 301, Figure 3B, such as in-ear earphones or over-ear headphones, separate from the electronic device) and one or more sensors (e.g., placement sensors (one or more), Figure 3B) for detecting the placement of the one or more wearable audio output devices. In some embodiments, the sensors are placed on the wearable audio output devices (e.g., as shown in Figure 3B). In some embodiments, the sensors are placed on the electronic device. In some embodiments, the device includes a display (e.g., touchscreen 112, Figure 1A, or display 340, Figure 3A) and a touch-sensitive surface (e.g., touchscreen 112, Figure 1A, or touchpad 355, Figure 3A). In some embodiments, the device communicates with or includes one or more audio output devices (e.g., one or more speakers such as speaker 111 on device 100, Figure 1A, or speaker(s) 306 on wearable audio output device 301, Figure 3B). In some embodiments, the device includes one or more audio input devices (e.g., microphone 113, Figure 1A, or microphone(s) 302, Figure 3B). In some embodiments, the device includes a rotatable input mechanism (e.g., as part of another input controller(s) 160 in Figure 1A), which may be used to provide input to the device by rotating an input mechanism (e.g., the crown of a wristwatch). In some embodiments, the device includes one or more accelerometers or attitude sensors (e.g., one or more accelerometers 167, one or more gyroscopes 168, and / or one or more magnetometers 169, Figure 1A) for detecting the movement and / or changes in the orientation of the device.In some embodiments, the wearable audio output device(s) communicating with the device include one or more accelerometers or attitude sensors (e.g., as part of another input device(s) 306 in Figure 3B) for detecting changes in the movement and / or orientation of the wearable audio output device(s). Some operations in Method 1000 are combined at will, and / or the order of some operations is changed at will.

[0279] As described below, Method 1000 delays the output of an audio notification from an electronic device when the user is currently engrossed in a conversation (to be detected on the electronic device via a microphone or on one or more wearable output devices communicating with the electronic device). This reduces the intrusiveness of the audio notification and increases the likelihood that the user will hear and understand the content of the audio notification, thereby reducing the need for the user to check the notification one or more times, whether audibly or visually on the device's display, thus reducing the number of inputs that need to be processed and the time that the display needs to be powered on, thereby improving the usability of the device, reducing power usage, and improving the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0280] The device detects the occurrence of a first event while one or more wearable audio output devices are being worn by the user (1002).

[0281] In response to detecting the occurrence of a first event, the device outputs one or more audio notifications corresponding to the first event via one or more wearable audio output devices (1004), which include delaying the output of one or more audio notifications corresponding to the first event until the conversation ends, according to a determination that the user of the electronic device is currently engrossed in a conversation (for example, based on detected speech by the user or another user), and outputting one or more audio notifications corresponding to the first event without delay, according to a determination that the user of the electronic device is not currently engrossed in a conversation. Delaying the output of audio notifications based on whether the user is engrossed in a conversation is described herein with reference to Figure 5AC.

[0282] In some embodiments where an electronic device communicates with one or more audio input devices, determining whether the user of the electronic device is currently engrossed in a conversation includes detecting speech via one or more audio input devices (as described herein with reference to, for example, Figure 5AC) (1006). In some embodiments, the one or more audio input devices include one or more microphones. In some embodiments, the one or more audio input devices are part of the electronic device. In some embodiments, the one or more audio input devices are part of one or more wearable audio output devices. By automatically detecting speech via one or more audio input devices such as microphones, the device can quickly and easily determine whether the user is engrossed in a conversation without requiring the use of alternative methods that do not require the device to prompt the user to provide one or more inputs. Performing speech detection automatically reduces the number of inputs required, thereby improving the usability of the device, reducing power usage, and improving the battery life of the device.

[0283] In some embodiments, outputting one or more voice notifications includes outputting audio content corresponding to a first event using pseudo-speech (as described herein with reference to voice notification 5106 in Figure 5AC, for example) (1008). Outputting voice notifications using pseudo-speech to output audio content corresponding to a detected event provides the user with substantial information about the detected event (e.g., the sender or initiator of the event, the content of the received message, the calendar or other event invitation time and location information, etc.) without requiring the user to provide additional input to hear the substantial information or to visually confirm the event information on the device's display. This reduces the number of inputs that need to be processed and the time that the display needs to be powered on, thereby improving the usability of the device, reducing power usage, and improving the battery life of the device.

[0284] In some embodiments, outputting one or more voice notifications includes outputting one or more audio tones (as described herein with reference to voice notification 5106 in Figure 5AC, for example) (1010). Outputting voice notifications using one or more audio tones (without outputting audio content using, for example, pseudo-speech) notifies the user that an event has been detected while reducing the intrusiveness and length of the audio content of the output notification, thereby reducing the amount of time the wearable audio output device(s) must be active or powered on, thereby improving the usability of the device, reducing power usage, and improving the battery life of the device.

[0285] In some embodiments, the electronic device includes or communicates with one or more second audio output devices (e.g., one or more speakers that are part of the electronic device) separate from one or more wearable audio output devices. In some embodiments, in conjunction with outputting verbalized information corresponding to a second event via one or more second audio output devices, the device refrains from outputting voice notifications corresponding to a second event via one or more wearable audio output devices (1012). In some embodiments, in conjunction with outputting one or more voice notifications corresponding to a first event via one or more wearable audio output devices, the device refrains from outputting verbalized information (e.g., voice notifications) corresponding to a first event via one or more second audio output devices. In other words, in some embodiments, the device outputs verbalized information (e.g., voice notifications) via one or more wearable audio output devices or via one or more second audio output devices, but not via both.

[0286] For example, in turn-by-turn navigation, while outputting language instructions via one or more speakers on or communicating with the electronic device, the device ceases outputting language instructions via one or more wearable audio output devices. In another example, while outputting language instructions via one or more wearable audio output devices, the device ceases outputting language instructions via one or more speakers on or communicating with the device. These examples are further described herein with reference to Figure 5AD.

[0287] By preventing the output of voice notifications through multiple audio output devices while the same voice notification is being output through other audio output devices, redundant audio output is avoided. This reduces overall power consumption across various audio output devices, reduces user confusion, improves device usability, reduces user errors, and thus reduces overall power consumption and improves the device's battery life.

[0288] In some embodiments, outputting one or more voice notifications corresponding to a first event is performed according to a determination that the metadata of the first event associates the first event with a voice notification (1014). In some embodiments, the metadata specifies that the notification corresponding to the first event should be presented to the user in language. In some embodiments, according to a determination that the metadata of the first event does not associate the first event with a voice notification, the device refrains from outputting one or more voice notifications corresponding to the first event. In some embodiments, the metadata of the first event specifies one or more properties of the first event (e.g., application, application type, event type, and / or contact for the first event), and metadata or settings stored in the electronic device associate each property of the first event with a voice notification.

[0289] In some embodiments, metadata associates the application associated with the first event with the voice notification (for example, the metadata specifies that a notification corresponding to a particular application should be presented to the user in a specific language). In some embodiments, the first event is associated with an application that is a specific type of application, and the metadata associates the application type of the application with the voice notification (for example, the metadata specifies that a notification corresponding to a particular type of application should be presented to the user in a specific language). For example, the first event is associated with a messaging application, and the metadata specifies that a notification corresponding to the messaging application should be presented to the user in a specific language (for example, via one or more wearable audio output devices). Other application types include calendar applications, fitness applications, game applications, and so on.

[0290] In some embodiments, the first event is a first type of event, and the metadata associates the event type of the event with a voice notification. For example, the first event is a calendar event, and the metadata specifies that a notification corresponding to the calendar event should be presented to the user in language. In some embodiments, the first event corresponds to a notification from a specific contact, and the metadata associates the notification from the specific contact with a voice notification. For example, the first event corresponds to a message received from a specific sender, and the metadata specifies that a message from the specific sender should be presented to the user in language. In some embodiments, the specific contact is one of several contacts on a contact whitelist stored on an electronic device.

[0291] By conditionally outputting voice notifications for detected events based on metadata of detected events indicating that an event is associated with a voice notification, and by requiring each event to specify that a voice notification should be provided for that event, the intrusiveness and frequency of voice notifications are reduced, and flexibility is provided to enable or disable voice notifications for specific events. By providing improved control over the presentation of voice notifications, device usability is improved, power usage is reduced, and device battery life is improved.

[0292] In some embodiments, outputting one or more voice notifications corresponding to a first event is performed according to a determination that the electronic device is in each mode that enables voice notifications (1016). In some embodiments, according to a determination that the electronic device is in each mode that does not enable voice notifications, the device either stops outputting one or more voice notifications corresponding to a first event or delays outputting one or more voice notifications corresponding to a first event until the device is in a mode that enables voice notifications.

[0293] In some embodiments, one or more device modes may be defined within a configuration user interface on an electronic device. In some embodiments, one or more device modes are defined by the user (for example, the user specifies a mode based on situations in which they want or do not want to receive voice notifications). In some embodiments, one or more device modes are defined based on volume settings, time of day, the device's geographical location, and / or the user's calendar availability, etc. An exemplary mode that enables voice notifications is a “work” mode defined based on business hours and / or the user’s workplace location or the presence of a device nearby. An exemplary mode that does not enable voice notifications is a “gym” mode defined based on an appointment time and / or the user’s gym location or the presence of a device nearby. In some embodiments, a particular mode enables a particular type of voice notification without enabling other types of voice notifications (for example, in “work” mode, calendar notifications are verbalized but game notifications are not).

[0294] By conditionally outputting voice notifications for detected events based on a determination that the current mode or context of the electronic device enables voice notifications, it reduces the intrusiveness and frequency of voice notifications in specific device modes, while allowing voice notifications to be provided in other modes as needed in specific device modes; provides users with the flexibility to enable or disable voice notifications in specific modes (e.g., time, location, notification type); provides users with improved control over the presentation of voice notifications; improves device usability; reduces power consumption; and improves device battery life.

[0295] In some embodiments, outputting one or more audio notifications corresponding to a first event is performed based on previous user behavior monitored by a virtual assistant running on the electronic device (1018). In some embodiments, the virtual assistant monitors user behavior on the electronic device to determine the circumstances under which the user has allowed the output of audio notifications (e.g., geographical location, time, volume setting, sender, etc.) and the circumstances under which the user has suppressed the output of audio notifications (e.g., by choosing not to wear one or more wearable audio output devices, by skipping presented audio notifications, etc.). In some embodiments, outputting one or more audio notifications corresponding to a first event is based on the determination that the device and / or one or more wearable audio output devices are present in a situation in which the user has previously allowed audio notifications. In some embodiments, in accordance with the determination that the device and / or one or more wearable audio output devices are present in a situation in which the user has previously allowed audio notifications, the device refrains from outputting one or more audio notifications corresponding to a first event.

[0296] By conditionally outputting voice notifications for events automatically detected based on previous user behavior learned by the virtual assistant, the number and range of control options that need to be presented to the user via the displayed settings interface are reduced, and the number of inputs the user needs to provide to the electronic device to specify the desired notification behavior is reduced, thereby improving device usability, reducing power consumption, and improving device battery life.

[0297] In some embodiments, where an electronic device communicates with one or more audio input devices, the device detects language input via one or more audio input devices indicating each event property (e.g., event type, application, application type, etc.) of interest to the user before outputting one or more voice notifications corresponding to a first event (1020). In some embodiments, outputting one or more voice notifications corresponding to a first event is performed according to a determination that the first event is associated with each event property. For example, in response to language input from the user indicating that the user is interested in transportation-related notifications, the device outputs one or more voice notifications if the first event includes transportation information (e.g., public transport schedule information) or if the first event is associated with a navigation application. In some embodiments, the event properties of interest to the user are inferred from the language input by the electronic device (e.g., by a virtual assistant running on the device).

[0298] By conditionally outputting audio notifications for detected events based on events associated with event properties specified by the user in previous language input, the system provides users with the flexibility to enable or disable (or emphasize or de-emphasize) notifications associated with specific types of events without requiring them to specify the target events via one or more displayed control options, thereby reducing the time the display needs to be powered on, improving device usability, reducing power consumption, and improving device battery life.

[0299] It should be understood that the specific order in which the operations are described in Figures 10A and 10B is merely an example, and is not intended to indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein. In addition, it should be noted that details of other processes described herein in relation to one or more other methods described herein (e.g., methods 600, 700, 800, and / or 900) are also applicable in a similar manner to method 1000 described above in relation to Figures 10A and 10B. For example, the devices, or their components, events, voice notifications, and inputs described herein with reference to method 1000 may optionally have any one or more characteristics of the devices, or their components, events, voice notifications, and inputs described herein with reference to one or more other methods described herein (e.g., methods 600, 700, 800, and / or 900). For brevity, those details will not be repeated here.

[0300] The above is written with reference to specific embodiments for illustrative purposes. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the invention and its practical applications, thereby enabling other persons skilled in the art to best use the invention and the various described embodiments with various modifications suitable for specific applications that may be conceived.

[0301] As described above, one aspect of the technology involves collecting and using data available from various sources to provide users with voice notifications in order to inform them of events that may be of interest to them. The disclosure considers that in some examples, such collected data may include personal information data that uniquely identifies a particular person, or personal information data that can be used to contact a particular person or to locate them. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter® IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), birth dates, or any other identifying or personal information.

[0302] This disclosure acknowledges that such use of personal data in the technology may be in the user's best interest. For example, personal data may be used to deliver event information as part of audio notifications that are more interesting to the user, while reducing the volume of event information that is not of interest or value to the user. Thus, such use of personal data enables the delivery of event information to the user that is tailored to the user's interests and takes into account information already known to the user. Furthermore, other uses of personal data that are in the user's best interest are also considered in this disclosure. For example, health and fitness data may be used to provide insights into the user's overall wellness or as positive feedback to individuals using the technology to pursue wellness goals.

[0303] This disclosure considers that entities involved in the collection, analysis, disclosure, transfer, storage, or other use of such personal data should adhere to a robust privacy policy and / or privacy practice. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for the strict confidentiality of personal data. Such policies should be readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected for the lawful and legitimate use of the entity and should not be shared or sold for any other purpose. Furthermore, such collection / sharing should be carried out only after informing and obtaining the user's consent. Furthermore, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others who have access to such personal data comply with those privacy policies and procedures. Furthermore, such entities may undergo third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal data collected and / or accessed, and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to and should be addressed accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0304] Notwithstanding the foregoing, this disclosure also considers embodiments that allow users to selectively prevent the use of or access to personal data. Specifically, this disclosure considers that hardware and / or software elements can be provided to prevent or prevent access to such personal data. For example, in the case of an event notification delivery service, the technology may be configured to allow a user to choose to “opt in” or “opt out” of participating in the collection of personal data during or at any time thereafter when registering for the service. In another example, a user may choose not to provide contact identification data, location data, data on received messages (such as phone calls, text messages, emails, calendar invitations, or events) for a target event notification delivery service. In yet another example, a user may choose to limit the period for which such data is retained or to completely prohibit the development of a user activity profile based on such data. In addition to providing “opt-in” and “opt-out” options, this disclosure also considers providing notices regarding access to or use of personal data. For example, a user could be notified when downloading an app that will access their personal data, and then warned again immediately before the app accesses their personal data.

[0305] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. This risk can be minimized by limiting data collection and deleting data when it is no longer needed. Additionally, where applicable, data de-identification can be used in certain health-related applications to protect user privacy. De-identification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means.

[0306] Therefore, while this disclosure broadly covers the use of personal data to implement one or more of the disclosed embodiments, we also consider that it is possible to implement those embodiments without requiring access to such personal data. In other words, the various embodiments of the technology are not rendered inoperable by the absence of all or part of such personal data. For example, event notifications can be selectively delivered to a user by inferring preferences based on only a minimal amount of non-personal data or personal information, such as content requested by a device associated with the user, other non-personal data available in the event notification delivery service, or publicly available information.

Claims

1. It is a method, An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, To detect the occurrence of multiple events without outputting notifications corresponding to the events via the wearable audio output devices while the one or more wearable audio output devices are not positioned relative to the user's ears, After detecting the occurrence of the multiple events, the system detects, via one or more sensors, that one or more wearable audio output devices are positioned relative to the user's ears at the respective locations. In response to the placement of one or more wearable audio output devices in the respective positions relative to the user's ears via one or more sensors, the wearable audio output devices output information including one or more voice notifications corresponding to any one or more of the multiple events. A method that includes this.

2. The method according to claim 1, wherein the one or more voice notifications include a voice summary of the plurality of events.

3. In accordance with the determination that the length of the audio content representing the multiple events does not exceed a predetermined length, the audio summary includes the audio content. In accordance with the determination that the length of the audio content representing the multiple events exceeds the predetermined length, the audio summary does not include all of the audio content. The method according to claim 2.

4. The method according to any one of claims 1 to 3, wherein the information includes one or more voice notifications corresponding to one or more events associated with each time period.

5. In response to the detection that one or more wearable audio output devices are positioned relative to the user's ear at their respective locations, outputting one or more audio notifications corresponding to one or more events associated with each time period is performed according to the determination that one or more wearable audio output devices are positioned relative to the user's ear at their respective locations during the initial time between each time period. The aforementioned method, After detecting that one or more wearable audio output devices are positioned relative to the user's ears at the respective locations, the system detects, via one or more sensors, that the one or more wearable audio output devices are positioned at the respective locations at a later time during each of the respective time periods. In response to detecting, via one or more sensors, that one or more wearable audio output devices are positioned at the respective locations at a later time between each of the time periods, the output of an audio notification corresponding to one or more events associated with each of the time periods is discontinued. The method according to claim 4, including the method described in claim 4.

6. The method according to any one of claims 1 to 3, wherein the information includes one or more voice notifications corresponding to one or more events that are scheduled to occur or are expected to occur within a predetermined period following the placement of the one or more wearable audio output devices at each of the positions relative to the user's ears.

7. Outputting one or more voice notifications includes outputting a first voice notification corresponding to the first event among the multiple events, In accordance with the determination that the electronic device is in a first context, the first voice notification includes first audio content corresponding to the first context. In accordance with the determination that the electronic device is in a second context different from the first context, the first voice notification includes a second audio content that is different from the first audio content and corresponds to the second context. The method according to any one of claims 1 to 6.

8. The aforementioned first event corresponds to a calendar event occurring at a first geographic location, and the aforementioned first geographic location includes location information for each of multiple hierarchical levels within the location hierarchy. The determination that the electronic device is within the first context includes determining that the location of the electronic device is within the same first geographical region as the first geographical location, wherein the first geographical region corresponds to a first hierarchical level in the location hierarchy, and the first audio content includes location information for each hierarchical level lower than the first hierarchical level in the location hierarchy. The determination that the electronic device is within the second context includes determining that the location of the electronic device is within the same second geographic region as the first geographic location, wherein the second geographic region is different from the first geographic region and corresponds to a second hierarchical level different from the first hierarchical level in the location hierarchy, and the second audio content includes location information for each hierarchical level lower than the second hierarchical level in the location hierarchy. The method according to claim 7.

9. The aforementioned first event corresponds to the first message from the first contact, The determination that the electronic device is in the first context includes determining that the first portion of the identifier of the first contact is the same as the first portion of the identifier of at least one other contact in the list of contacts stored in the electronic device, and the first audio content includes the first portion of the identifier of the first contact, the second portion of the identifier of the first contact, and the content of the first message. Determining that the electronic device is in the second context includes determining that the first portion of the identifier of the first contact is not the first portion of the identifier of any other contact in the list of contacts, and the second audio content includes the first portion of the identifier of the first contact and the content of the first message, without including the second portion of the identifier. The method according to claim 7.

10. The aforementioned first event corresponds to the first message from the first contact, The determination that the electronic device is within the first context includes determining that the first event does not follow consecutively to the second event among the plurality of events corresponding to the second message from the first contact, and the first audio content includes the identifier of the first contact and the content of the first message. The determination that the electronic device is in the second context includes determining that the first event follows the second event of the plurality of events corresponding to the second message from the first contact, and the second audio content includes the content of the first message but does not include the identifier of the first contact. The method according to claim 7.

11. The aforementioned first event corresponds to the first message, which is part of the group message session for each group. The determination that the electronic device is within the first context includes determining that the first event does not follow consecutively to the second event among the plurality of events that corresponds to the second message which is part of the group message session, and the first audio content includes the identifier of each of the groups or the group message session, and the content of the first message. The determination that the electronic device is in the second context includes determining that the first event follows a second event among a plurality of events that corresponds to a second message which is part of the group message session, and the second audio content includes the content of the first message but does not include the identifier of the respective group or the group message session. The method according to claim 7.

12. The method according to any one of claims 1 to 11, wherein outputting one or more audio notifications includes outputting a first audio notification corresponding to each of the first events among the plurality of events, each of the first events corresponds to a first application associated with a first notification class, and each of the first audio notifications has a first structure defined by the first notification class.

13. The method according to any one of claims 1 to 12, wherein outputting one or more audio notifications includes outputting a first audio notification corresponding to each of the second events among the plurality of events, each of the second events corresponds to a second application, and each of the second audio notifications has a second structure defined by the second application.

14. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is One or more processors, A memory storing one or more programs is provided, and the one or more programs are configured to be executed by the one or more processors, While one or more wearable audio output devices are not positioned relative to the user's ears, the system detects the occurrence of multiple events without outputting notifications corresponding to the events via the wearable audio output devices. After detecting the occurrence of the multiple events, the system detects, via one or more sensors, that one or more wearable audio output devices are positioned relative to the user's ears at the respective locations. In response to the placement of one or more wearable audio output devices in the respective positions relative to the user's ears via the one or more sensors, the wearable audio output devices output information including one or more voice notifications corresponding to any one or more of the multiple events. An electronic device that includes instructions for use.

15. A computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and the instructions are executed by an electronic device that communicates with one or more wearable audio output devices and one or more sensors for detecting the placement of the one or more wearable audio output devices, While one or more wearable audio output devices are not positioned relative to the user's ears, the system detects the occurrence of multiple events without outputting notifications corresponding to the events via the wearable audio output devices. After detecting the occurrence of the multiple events, the system detects, via one or more sensors, that one or more wearable audio output devices are positioned relative to the user's ears at the respective locations. In response to the placement of one or more wearable audio output devices at their respective positions relative to the user's ears via the one or more sensors, the wearable audio output devices are configured to output information including one or more voice notifications corresponding to any one or more of the multiple events. Computer-readable storage medium.

16. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is Means that are enabled to detect the occurrence of multiple events without outputting notifications corresponding to the events via the wearable audio output devices while the one or more wearable audio output devices are not positioned relative to the user's ears, After detecting the occurrence of the multiple events, means are enabled to detect, via one or more sensors, that one or more wearable audio output devices are positioned relative to the user's ears at the respective locations. Means that, in response to the placement of the one or more wearable audio output devices in the respective positions relative to the user's ears via the one or more sensors, are enabled to output information via the one or more wearable audio output devices, including one or more voice notifications corresponding to any one or more of the multiple events. Electronic devices, including those mentioned above.

17. An information processing device for use in an electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the arrangement of the one or more wearable audio output devices, wherein the information processing device is Means that are enabled to detect the occurrence of multiple events without outputting notifications corresponding to the events via the wearable audio output devices while the one or more wearable audio output devices are not positioned relative to the user's ears, After detecting the occurrence of the multiple events, means are enabled to detect, via one or more sensors, that one or more wearable audio output devices are positioned relative to the user's ears at the respective locations. Means that, in response to the placement of the one or more wearable audio output devices in the respective positions relative to the user's ears via the one or more sensors, are enabled to output information via the one or more wearable audio output devices, including one or more voice notifications corresponding to any one or more of the multiple events. Information processing device, including

18. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is One or more processors, A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs perform the method according to any one of claims 1 to 13. An electronic device that includes instructions for use.

19. A computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device communicating with one or more wearable audio output devices and one or more sensors for detecting the placement of the one or more wearable audio output devices, the computer-readable storage medium causes the electronic device to perform the method according to any one of claims 1 to 13.

20. A graphical user interface on an electronic device communicating with one or more sensors for detecting the placement of one or more wearable audio output devices, wherein the electronic device includes a memory and one or more processors for executing one or more programs stored in the memory, and the graphical user interface includes a user interface displayed according to the method of any one of claims 1 to 13.

21. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is An electronic device comprising means for performing the method described in any one of claims 1 to 13.

22. An information processing device for use in an electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the arrangement of the one or more wearable audio output devices, wherein the information processing device is An information processing apparatus comprising means for performing the method described in any one of claims 1 to 13.

23. It is a method, In an electronic device having one or more audio output devices, Detecting the occurrence of an event, The process includes, in response to detecting the occurrence of the event, outputting an audio notification containing audio content representing the event via one or more audio output devices, In accordance with the determination that the event is associated with a first characteristic of a first characteristic group, the voice notification includes the audio content representing the event and has a first audio output characteristic selected based on the first characteristic. A method wherein, in accordance with the determination that the event is associated with a second characteristic of the first set of characteristics which is different from the first characteristic, the voice notification includes the audio content representing the event and has a second audio output characteristic which is different from the first output characteristic, selected based on the second characteristic.

24. The determination that the event is associated with the first characteristic of the first set of characteristics includes the determination that the event was initiated by the first user of a group of users. The determination that the event is associated with the second characteristic of the first set of characteristics includes the determination that the event was initiated by the second user of the plurality of users, The first audio output characteristics include a first spatial position associated with the first user, The second audio output characteristics, unlike the first position, include a second spatial position associated with the second user. The method according to claim 23.

25. The determination that the event is associated with the first characteristic of the first set of characteristics includes the determination that the event was initiated by the first user of a group of users. The determination that the event is associated with the second characteristic of the first set of characteristics includes the determination that the event was initiated by the second user of the plurality of users, The first audio output characteristic includes a first audio tone associated with the first user, The second audio output characteristic differs from the first audio tone and includes a second audio tone associated with the second user. The method according to claim 23.

26. The determination that the event is associated with the first characteristic of the first set of characteristics includes the determination that the event was initiated by the first user of a group of users. The determination that the event is associated with the second characteristic of the first set of characteristics includes the determination that the event was initiated by the second user of the plurality of users, Outputting a voice notification having the first audio output characteristics, which includes the audio content representing the event, includes outputting a voice notification including the audio content representing the event using a first pseudo-voice associated with the first user, Outputting a voice notification having the second audio output characteristics and including the audio content representing the event includes, unlike the first pseudo-voice, outputting a voice notification including the audio content representing the event using a second pseudo-voice associated with the second user. The method according to claim 23.

27. The determination that the event is associated with the first characteristic of the first set of characteristics includes the determination that the event is associated with the first transaction value. The determination that the event is associated with the second characteristic of the first set of characteristics includes the determination that the event is associated with a second transaction value different from the first transaction value, The first audio output characteristic includes a first sound effect representing the first transaction value, The second audio output characteristic differs from the first sound effect and includes a second sound effect that represents the second transaction value. The method according to claim 23.

28. The determination that the event is associated with the first characteristic of the first set of characteristics includes the determination that the event is a notification of a first type, The determination that the event is associated with the second characteristic of the first set of characteristics includes the determination that the event is a second type of notification, which is different from the first type of notification. The first audio output characteristic includes a first background audio associated with the first type of notification, The second audio output characteristic differs from the first background audio and includes a second background audio associated with the second type of notification. The method according to claim 23.

29. The determination that the event is associated with the first characteristic of the first set of characteristics includes the determination that the event is associated with a message of a first type, The determination that the event is associated with the second characteristic of the first set of characteristics includes the determination that the event is associated with a second type of message that is different from the first type of message, The first audio output characteristic includes a first spatial position associated with the first type of message, The second audio output characteristic, unlike the first position, includes a second spatial position associated with the second type of message, The method according to claim 23.

30. The first type of message corresponds to a message between the user of the electronic device and at least two other users, The second type of message corresponds to a message between the user of the electronic device and up to one other user. The method according to claim 29.

31. The method according to claim 30, wherein the first spatial position is a pseudo-spatial position outside the user's head, and the second spatial position is a pseudo-spatial position inside the user's head.

32. The electronic device includes a display and a touch-sensing surface, and the method is, Displaying a user interface on the aforementioned display, wherein the user interface is The representation of one or more audio output devices, A first user interface element that represents the first type of message and is displayed at the first position in the user interface, wherein the first position in the user interface defines the first spatial position associated with the first type of message, A second user interface that represents the second type of message and is displayed at the second position within the user interface, wherein the second position of the user interface defines the second spatial position associated with the second type of message, and includes a second user interface element. Including, displaying, The touch-sensitive surface receives input corresponding to the first user interface, In response to receiving the aforementioned input, Moving the first user interface element to a third position within the user interface, Associating the first type of message with a third spatial location defined by the third location within the user interface The method according to claim 29, including the method described in claim 29.

33. One or more audio output devices, One or more processors, An electronic device comprising a memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, Detect the occurrence of an event, In response to the detection of the occurrence of the aforementioned event, one or more audio output devices will output an audio notification containing audio content representing the event. Includes instructions for, In accordance with the determination that the event is associated with a first characteristic of a first characteristic group, the voice notification includes the audio content representing the event and has a first audio output characteristic selected based on the first characteristic. In accordance with the determination that the event is associated with a second characteristic of the first set of characteristics which is different from the first characteristic, the voice notification includes the audio content representing the event and has a second audio output characteristic which is different from the first output characteristic, selected based on the second characteristic. Electronic devices.

34. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device having one or more audio output devices, the electronic device has the following properties: To detect the occurrence of an event, In response to the detection of the occurrence of the event, an audio notification including audio content representing the event is output via one or more audio output devices. In accordance with the determination that the event is associated with a first characteristic of a first characteristic group, the voice notification includes the audio content representing the event and has a first audio output characteristic selected based on the first characteristic. In accordance with the determination that the event is associated with a second characteristic of the first set of characteristics which is different from the first characteristic, the voice notification includes the audio content representing the event and has a second audio output characteristic which is different from the first output characteristic, selected based on the second characteristic. Computer-readable storage medium.

35. It is an electronic device, One or more audio output devices Means for detecting the occurrence of an event, In response to the detection of the occurrence of the aforementioned event, means for outputting an audio notification including audio content representing the event via one or more audio output devices, Includes, In accordance with the determination that the event is associated with a first characteristic of a first characteristic group, the voice notification includes the audio content representing the event and has a first audio output characteristic selected based on the first characteristic. In accordance with the determination that the event is associated with a second characteristic of the first set of characteristics which is different from the first characteristic, the voice notification includes the audio content representing the event and has a second audio output characteristic which is different from the first output characteristic, selected based on the second characteristic. Electronic devices.

36. An information processing device for use in an electronic device having one or more audio output devices, Means for detecting the occurrence of an event, In response to the detection of the occurrence of the aforementioned event, means for outputting an audio notification including audio content representing the event via one or more audio output devices, Includes, In accordance with the determination that the event is associated with a first characteristic of a first characteristic group, the voice notification includes the audio content representing the event and has a first audio output characteristic selected based on the first characteristic. In accordance with the determination that the event is associated with a second characteristic of the first set of characteristics which is different from the first characteristic, the voice notification includes the audio content representing the event and has a second audio output characteristic which is different from the first output characteristic, selected based on the second characteristic. Information processing device.

37. One or more audio output devices, One or more processors, An electronic device comprising a memory storing one or more programs, wherein the one or more programs are configured to be executed by one or more processors, and the one or more programs perform the method according to any one of claims 23 to 32. Includes instructions for, Electronic devices.

38. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device having one or more audio output devices, the instructions cause the electronic device to perform the method according to any one of claims 23 to 32.

39. A graphical user interface on an electronic device having one or more audio output devices, memory, and one or more processors for executing one or more programs stored in the memory, wherein the graphical user interface includes a user interface displayed according to the method of any one of claims 23 to 32.

40. One or more audio output devices, Means for carrying out the method described in any one of claims 23 to 32 Electronic devices, including those mentioned above.

41. An information processing device for use in an electronic device having one or more audio output devices, An information processing apparatus comprising means for performing the method described in any one of claims 23 to 32.

42. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, To detect the occurrence of the first event, After detecting the occurrence of the first event, output one or more voice notifications corresponding to the first event via one or more wearable audio output devices, After the output of one or more voice notifications has begun, the system detects input directed to one or more wearable audio output devices. Includes, In response to the detection of the input directed to one or more wearable audio output devices, The first operation is performed in accordance with the determination that the input has been detected within a predetermined time period with respect to one or more audio notifications corresponding to the first event, wherein the first operation is associated with one or more audio notifications corresponding to the first event. A second action is performed in accordance with the determination that the input is detected after a predetermined time period has elapsed with respect to one or more voice notifications corresponding to the first event, wherein the second action is not associated with the one or more voice notifications corresponding to the first event. A method that includes this.

43. Detecting a second input directed to one or more wearable audio output devices, In response to the detection of the second input directed to one or more wearable audio output devices, The second operation is performed according to the determination that the second input was detected before the output of the one or more audio notifications corresponding to the first event began, and according to the determination that a predetermined time period has elapsed with respect to one or more audio notifications corresponding to the second event and output before the one or more audio notifications corresponding to the first event, wherein the second operation is not associated with the one or more audio notifications corresponding to the second event. The method according to claim 42, including the method described in claim 42.

44. After detecting the occurrence of the first event, the occurrence of the third event is detected, After detecting the occurrence of the third event and outputting one or more notifications corresponding to the first event, one or more voice notifications corresponding to the third event are output via one or more wearable audio output devices. In response to the detection of the input directed to one or more wearable audio output devices, The first operation is performed in accordance with the determination that the input is detected after the output of one or more voice notifications corresponding to the third event has begun, and within the predetermined time period with respect to the one or more voice notifications corresponding to the third event, wherein the first operation is associated with the one or more voice notifications corresponding to the third event. The method according to claim 42 or 43, including the method described in claim 42 or 43.

45. Outputting one or more audio notifications corresponding to each event includes outputting a portion of the audio content corresponding to each event. Performing the first action associated with the one or more audio notifications corresponding to each of the aforementioned events includes refraining from outputting the remaining portion of the one or more audio notifications corresponding to each of the aforementioned events. The method according to any one of claims 42 to 44.

46. Outputting one or more audio notifications corresponding to each event includes outputting a first portion of the audio content corresponding to each event. Performing the first action associated with the one or more audio notifications corresponding to each of the aforementioned events includes outputting a second portion of the audio content corresponding to each of the aforementioned events, which is different from the first portion. The method according to any one of claims 42 to 44.

47. The method according to claim 45 or 46, wherein performing the second action not associated with the one or more voice notifications corresponding to each event includes invoking a virtual assistant or controlling media content playback.

48. The method according to any one of claims 42 to 44, wherein the first action associated with the one or more voice notifications corresponding to each event is based on the application associated with each of the events.

49. The method according to any one of claims 42 to 44, wherein detecting each of the aforementioned events includes receiving media content other than text content, and performing the first action associated with the one or more audio notifications corresponding to each of the aforementioned events includes providing a linguistic description of the media content other than text content.

50. In accordance with the determination that the input is detected while the first voice notification of one or more voice notifications is being output, the first operation is performed for the first voice notification. According to the determination that the input is detected while a second voice notification of one or more voice notifications different from the first voice notification is being output, the first operation is performed for the second voice notification. The method according to any one of claims 42 to 49.

51. According to the determination that the input is of type first, the first operation is of type first, In accordance with the determination that the input is a second type of input different from the first type of input, the first operation is a second type of operation different from the first type of operation. The method according to any one of claims 42 to 50.

52. According to the determination that the input is of type 1, the second operation is of type 3. In accordance with the determination that the input is a second type of input different from the first type of input, the second operation is a fourth type of operation different from the third type of operation. The method according to any one of claims 42 to 51.

53. The method according to any one of claims 42 to 52, wherein detecting the input directed to the one or more wearable audio output devices includes detecting a tap input on the one or more wearable audio output devices.

54. The method according to any one of claims 42 to 52, wherein the one or more wearable audio output devices include one or more accelerometers, and detecting the input directed to the one or more wearable audio output devices includes detecting the movement of the one or more wearable audio output devices via the one or more accelerometers.

55. Outputting one or more audio notifications corresponding to each event includes outputting a first portion of the audio content corresponding to each event. Performing the first action in accordance with detecting the movement of the one or more wearable audio output devices in a first direction via the one or more accelerometers includes ceasing to output the second portion of the audio content corresponding to each of the events. Performing the first action by detecting the movement of the one or more wearable audio output devices in a second direction opposite to a first direction via the one or more accelerometers includes outputting a second portion of audio content corresponding to each of the events. The method according to claim 54.

56. The method according to any one of claims 42 to 55, further comprising, after detecting the occurrence of each event, ceasing to output one or more notifications corresponding to each event other than the one or more voice notifications output via the one or more wearable audio output devices.

57. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is One or more processors, A memory storing one or more programs is provided, and the one or more programs are configured to be executed by the one or more processors, The occurrence of the first event is detected, After detecting the occurrence of the first event, one or more voice notifications corresponding to the first event are output via one or more wearable audio output devices. After starting to output one or more of the aforementioned voice notifications, the system detects input directed to one or more of the aforementioned wearable audio output devices. In response to the detection of the input directed to one or more wearable audio output devices, A first operation is performed in accordance with the determination that the input has been detected within a predetermined time period with respect to one or more voice notifications corresponding to the first event, and the first operation is associated with one or more voice notifications corresponding to the first event. A second action is performed in accordance with the determination that the input is detected after a predetermined time period has elapsed with respect to one or more voice notifications corresponding to the first event, and the second action is not associated with the one or more voice notifications corresponding to the first event. An electronic device that includes instructions for use.

58. A computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and the instructions are executed by an electronic device that communicates with one or more wearable audio output devices and one or more sensors for detecting the placement of the one or more wearable audio output devices, To detect the occurrence of the first event, After detecting the occurrence of the first event, one or more voice notifications corresponding to the first event are output via one or more wearable audio output devices. After the output of one or more voice notifications has begun, the device detects an input directed to one or more wearable audio output devices. In response to the detection of the input directed to one or more wearable audio output devices, A first operation is performed according to the determination that the input has been detected within a predetermined time period with respect to one or more voice notifications corresponding to the first event, and the first operation is associated with one or more voice notifications corresponding to the first event. A second action is performed in accordance with the determination that the input is detected after a predetermined time period has elapsed with respect to one or more voice notifications corresponding to the first event, and the second action is not associated with the one or more voice notifications corresponding to the first event. Computer-readable storage medium.

59. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is A means for detecting the occurrence of the first event, Means for outputting one or more voice notifications corresponding to the first event via one or more wearable audio output devices, which are activated after the occurrence of the first event is detected, Means for detecting input directed to one or more wearable audio output devices, which are activated after the output of one or more voice notifications begins, Means that are activated in response to the detection of the input directed to one or more wearable audio output devices, Means for performing a first operation, which is activated according to a determination that the input has been detected within a predetermined time period with respect to one or more audio notifications corresponding to the first event, wherein the first operation is associated with one or more audio notifications corresponding to the first event. Means for performing a second operation, which is activated in accordance with a determination that the input has been detected after a predetermined time period has elapsed with respect to one or more voice notifications corresponding to the first event, wherein the second operation is not associated with the one or more voice notifications corresponding to the first event. Electronic devices, including those mentioned above.

60. An information processing device for use in an electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the arrangement of the one or more wearable audio output devices, wherein the information processing device is A means for detecting the occurrence of the first event, Means for outputting one or more voice notifications corresponding to the first event via one or more wearable audio output devices, which are activated after the occurrence of the first event is detected, Means for detecting input directed to one or more wearable audio output devices, which are activated after the output of one or more voice notifications begins, Means that are activated in response to the detection of the input directed to one or more wearable audio output devices, Means for performing a first operation, which is activated according to a determination that the input has been detected within a predetermined time period with respect to one or more audio notifications corresponding to the first event, wherein the first operation is associated with one or more audio notifications corresponding to the first event. Means for performing a second operation, which is activated in accordance with a determination that the input has been detected after a predetermined time period has elapsed with respect to one or more voice notifications corresponding to the first event, wherein the second operation is not associated with the one or more voice notifications corresponding to the first event. Information processing device, including

61. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is One or more processors, A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 42 to 56. Electronic devices.

62. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device communicating with one or more wearable audio output devices and one or more sensors for detecting the placement of the one or more wearable audio output devices, the computer-readable storage medium causes the electronic device to perform the method according to any one of claims 42 to 56.

63. A graphical user interface on an electronic device communicating with one or more sensors for detecting the placement of one or more wearable audio output devices, wherein the electronic device includes a memory and one or more processors for executing one or more programs stored in the memory, and the graphical user interface includes a user interface displayed according to the method of any one of claims 42 to 56.

64. An electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the placement of the one or more wearable audio output devices, wherein the electronic device is An electronic device comprising means for performing the method described in any one of claims 42 to 56.

65. An information processing device for use in an electronic device that communicates with one or more wearable audio output devices and communicates with one or more sensors for detecting the arrangement of the one or more wearable audio output devices, wherein the information processing device is An information processing apparatus comprising means for performing the method described in any one of claims 42 to 56.

66. In an electronic device communicating with one or more wearable audio output devices, To detect the occurrence of the first event, After detecting the occurrence of the first event, output one or more voice notifications corresponding to the first event via one or more wearable audio output devices, After outputting one or more voice notifications corresponding to the first event, the electronic device outputs separate audio outputs from the one or more voice notifications indicating that it is ready to receive a response to the first event, While each of the aforementioned audio outputs is output, an input corresponding to the response to the first event is detected, and a response to the first event is generated based on the input. A method that includes this.

67. The process includes outputting one or more voice notifications corresponding to the first event, and then detecting a second input corresponding to a request to respond to the first event via the electronic device or one or more wearable audio output devices, In response to the detection of the second input, the respective audio outputs are output to indicate that the electronic device is ready to receive a response to the first event. The method according to claim 66.

68. The method according to claim 66 or 67, wherein each of the aforementioned audio outputs includes white noise.

69. The method according to any one of claims 66 to 68, wherein the electronic device communicates with one or more audio input devices, and detecting an input corresponding to a response to the first event includes detecting an audio input via the one or more audio input devices.

70. The electronic device includes a touch-sensing surface, and the method is While each of the aforementioned audio outputs is being output, and before detecting the input corresponding to the first event, The first touch input is detected via the touch sensing surface, In response to detecting the first touch input via the touch-sensing surface, one or more response options for the first event are output via one or more wearable audio output devices. It further includes, Detecting the input corresponding to the response to the first event includes detecting a second touch input via the touch-sensing surface that selects each of the one or more response options. The response generated for the first event corresponds to each of the selected response options, The method according to any one of claims 66 to 68.

71. The electronic device includes a rotatable input mechanism, and the method is While each of the aforementioned audio outputs is being output, and before detecting the input corresponding to the first event, The first input is detected via the rotatable input mechanism, In response to detecting the first input via the rotatable input mechanism, one or more response options for the first event are output via one or more wearable audio output devices. Includes, Detecting the input corresponding to the response to the first event includes detecting a second input via the rotatable input mechanism for selecting each of the one or more response options. The response generated for the first event corresponds to each of the selected response options, The method according to any one of claims 66 to 68.

72. The electronic device includes a display and one or more attitude sensors, and the method is Before detecting the input corresponding to the response to the first event, the change in the attitude of the electronic device is detected via one or more attitude sensors, In response to the detection of the change in the orientation of the electronic device, In accordance with the determination that a change in the orientation of the electronic device is detected within a predetermined time period with respect to one or more audio notifications corresponding to the first event, one or more response options for the first event are displayed on the display. Includes, Detecting the input corresponding to the response to the first event includes detecting the input that selects each of the one or more response options. The method according to any one of claims 66 to 68.

73. After outputting one or more voice notifications corresponding to the first event, and before outputting each audio output separate from the one or more voice notifications indicating that the electronic device is ready to receive a response to the first event, This includes outputting a plurality of response options via one or more wearable audio output devices that prompt the user of the electronic device to provide a response to the first event, The method according to any one of claims 66 to 68.

74. The method according to claim 73, wherein the first event is a calendar event invitation, and the plurality of response options include an option to accept the calendar event invitation and an option to decline the calendar event invitation.

75. The electronic device communicates with one or more audio input devices, and the method is To detect the volume of ambient noise to one or more of the aforementioned audio input devices, Based on the volume of the ambient noise, the output volume of one or more voice notifications is adjusted. The method according to any one of claims 66 to 74, including the method described in any one of claims 66 to 74.

76. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is One or more processors, A memory storing one or more programs is provided, and the one or more programs are configured to be executed by the one or more processors, The occurrence of the first event is detected, After detecting the occurrence of the first event, one or more voice notifications corresponding to the first event are output via one or more wearable audio output devices. After outputting one or more voice notifications corresponding to the first event, the electronic device outputs separate audio outputs from the one or more voice notifications to indicate that it is ready to receive a response to the first event. While each of the aforementioned audio outputs is being output, an input corresponding to the response to the first event is detected, and a response to the first event is generated based on the input. An electronic device that includes instructions for use.

77. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device communicating with one or more wearable audio output devices, the electronic device... To detect the occurrence of the first event, After detecting the occurrence of the first event, one or more voice notifications corresponding to the first event are output via one or more wearable audio output devices. After outputting one or more voice notifications corresponding to the first event, the electronic device outputs separate audio outputs from the one or more voice notifications to indicate that it is ready to receive a response to the first event. While each of the aforementioned audio outputs is being output, an input corresponding to the response to the first event is detected, and a response to the first event is generated based on the input. Computer-readable storage medium.

78. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is A means for detecting the occurrence of the first event, Means for outputting one or more voice notifications corresponding to the first event via one or more wearable audio output devices, which are activated after the occurrence of the first event is detected, Means for outputting separate audio outputs from the one or more audio notifications, which are activated after outputting the one or more audio notifications corresponding to the first event, indicating that the electronic device is ready to receive a response to the first event, Means for detecting an input corresponding to the response to the first event and generating a response to the first event based on the input, which are activated while each of the aforementioned audio outputs is outputting Electronic devices, including those mentioned above.

79. An information processing device for use in an electronic device communicating with one or more wearable audio output devices, wherein the information processing device is A means for detecting the occurrence of the first event, Means for outputting one or more voice notifications corresponding to the first event via one or more wearable audio output devices, which are activated after the occurrence of the first event is detected, Means for outputting separate audio outputs from the one or more audio notifications, which are activated after outputting the one or more audio notifications corresponding to the first event, indicating that the electronic device is ready to receive a response to the first event, Means for detecting an input corresponding to the response to the first event and generating a response to the first event based on the input, which are activated while each of the aforementioned audio outputs is outputting Information processing device, including

80. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is One or more processors, An electronic device comprising a memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 66 to 75.

81. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device communicating with one or more wearable audio output devices, the computer-readable storage medium causes the electronic device to perform the method according to any one of claims 66 to 75.

82. A graphical user interface on an electronic device communicating with one or more wearable audio output devices, wherein the electronic device includes a memory and one or more processors for executing one or more programs stored in the memory, and the graphical user interface includes a user interface displayed according to the method of any one of claims 66 to 75.

83. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is An electronic device comprising means for performing the method described in any one of claims 66 to 75.

84. An information processing device for use in an electronic device communicating with one or more wearable audio output devices, wherein the information processing device is An information processing apparatus comprising means for performing the method described in any one of claims 66 to 75.

85. In an electronic device communicating with one or more wearable audio output devices, The detection of the occurrence of a first event while one or more wearable audio output devices are being worn by the user, In response to the detection of the occurrence of the first event, output one or more voice notifications corresponding to the first event via one or more wearable audio output devices, In accordance with the determination that the user of the electronic device is currently engrossed in a conversation, the output of one or more voice notifications corresponding to the first event is delayed until the conversation ends. In accordance with the determination that the user of the electronic device is not currently engrossed in conversation, the system outputs one or more voice notifications corresponding to the first event without delaying the output. Methods that include...

86. The method according to claim 85, wherein the electronic device communicates with one or more audio input devices, and determining that the user of the electronic device is currently engrossed in a conversation includes detecting speech via the one or more audio input devices.

87. The method according to claim 85 or 86, wherein outputting one or more voice notifications includes outputting audio content corresponding to the first event using pseudo-speech.

88. The method according to any one of claims 85 to 87, wherein outputting one or more voice notifications includes outputting one or more audio tones.

89. The method according to any one of claims 85 to 88, wherein the electronic device includes or communicates with one or more second audio output devices separate from the one or more wearable audio output devices, and the method includes, in addition to outputting verbalized information corresponding to a second event via the one or more second audio output devices, ceasing to output an audio notification corresponding to the second event via the one or more wearable audio output devices.

90. The method according to claim 85, wherein outputting one or more audio notifications corresponding to the first event is performed in accordance with the determination that the metadata of the first event associates the first event with an audio notification.

91. The method according to claim 85, wherein outputting one or more voice notifications corresponding to the first event is performed according to a determination that the electronic device is in each mode that enables voice notifications.

92. The method according to claim 85, wherein outputting one or more voice notifications corresponding to the first event is performed based on previous user behavior monitored by a virtual assistant running on the electronic device.

93. The electronic device communicates with one or more audio input devices, and the method is The process includes detecting language input via one or more audio input devices that indicates the respective event properties of interest to the user, before outputting one or more audio notifications corresponding to the first event, Outputting one or more audio notifications corresponding to the first event is performed according to the determination that the first event is associated with each of the event properties. The method according to claim 85.

94. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is One or more processors, A memory storing one or more programs is provided, and the one or more programs are configured to be executed by the one or more processors, The occurrence of a first event is detected while one or more wearable audio output devices are being worn by the user. The command includes, in response to the detection of the occurrence of the first event, to output one or more voice notifications corresponding to the first event via one or more wearable audio output devices, and outputting the one or more voice notifications is: In accordance with the determination that the user of the electronic device is currently engrossed in a conversation, the output of one or more voice notifications corresponding to the first event is delayed until the conversation ends. In accordance with the determination that the user of the electronic device is not currently engrossed in conversation, the system outputs one or more voice notifications corresponding to the first event without delaying the output. Electronic devices, including those mentioned above.

95. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device communicating with one or more wearable audio output devices, the electronic device... The aforementioned one or more wearable audio output devices are used to detect the occurrence of a first event while being worn by a user. In response to the detection of the occurrence of the first event, one or more voice notifications corresponding to the first event are output via one or more wearable audio output devices, and outputting one or more voice notifications is: In accordance with the determination that the user of the electronic device is currently engrossed in a conversation, the output of one or more voice notifications corresponding to the first event is delayed until the conversation ends. In accordance with the determination that the user of the electronic device is not currently engrossed in conversation, the system outputs one or more voice notifications corresponding to the first event without delaying the output. Computer-readable storage media, including [specific data / information].

96. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is Means for detecting the occurrence of a first event while one or more wearable audio output devices are being worn by a user, Means for outputting one or more voice notifications corresponding to the first event via one or more wearable audio output devices, which are activated in response to the detection of the occurrence of the first event, Means for delaying the output of one or more voice notifications corresponding to the first event until the conversation ends, which are activated according to the determination that the user of the electronic device is currently engrossed in a conversation, Means for outputting one or more voice notifications corresponding to the first event without delay, which are activated according to a determination that the user of the electronic device is not currently engrossed in a conversation, Electronic devices, including those mentioned above.

97. An information processing device for use in an electronic device communicating with one or more wearable audio output devices, wherein the information processing device is Means for detecting the occurrence of a first event while one or more wearable audio output devices are being worn by a user, Means for outputting one or more voice notifications corresponding to the first event via one or more wearable audio output devices, which are activated in response to the detection of the occurrence of the first event, Means for delaying the output of one or more voice notifications corresponding to the first event until the conversation ends, which are activated according to the determination that the user of the electronic device is currently engrossed in a conversation, Means for outputting one or more voice notifications corresponding to the first event without delay, which are activated according to a determination that the user of the electronic device is not currently engrossed in a conversation, Information processing device, including

98. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is One or more processors, An electronic device comprising a memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 85 to 93.

99. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by an electronic device communicating with one or more wearable audio output devices, the computer-readable storage medium causes the electronic device to perform the method according to any one of claims 85 to 93.

100. A graphical user interface on an electronic device communicating with one or more wearable audio output devices, wherein the electronic device includes a memory and one or more processors for executing one or more programs stored in the memory, and the graphical user interface includes a user interface displayed according to the method of any one of claims 85 to 93.

101. An electronic device that communicates with one or more wearable audio output devices, wherein the electronic device is An electronic device comprising means for performing the method described in any one of claims 85 to 93.

102. An information processing device for use in an electronic device communicating with one or more wearable audio output devices, wherein the information processing device is An information processing apparatus comprising means for performing the method described in any one of claims 85 to 93.

Images