Method and interface for home media control

Abstract

This application relates to methods and interfaces for home media control. The present disclosure relates generally to interfaces and techniques for media playback on one or more devices. According to some embodiments, an electronic device includes a display, one or more processors, and memory. The electronic device receives a user input, and in response to receiving the user input, displays, on the display, a multi-device interface that includes one or more indicators associated with a plurality of available playback devices that are connected to the device and available to initiate playback of media from the device and media playback status of the plurality of available playback devices.

Description

[0001] This application is a divisional application of the invention patent application filed on May 16, 2018, with application number 202010728844.3 and entitled "Method and Interface for Home Media Control".

[0002] Cross-references to related applications

[0003] This application claims priority to the following patent applications: U.S. Provisional Patent Application No. 62 / 507,202, filed May 16, 2017, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL"; U.S. Provisional Patent Application No. 62 / 507,208, filed May 16, 2017, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL"; Danish Patent Application No. PA 2017 70392, filed May 29, 2017, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL"; Danish Patent Application No. PA 201770401, filed May 29, 2017, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL"; and others. Danish patent application No. PA 2017 70402 entitled "HOME MEDIA CONTROL"; Danish patent application No. PA 2017 70403 entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL" filed on May 29, 2017; Danish patent application No. PA 2017 70404 entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL" filed on May 29, 2017; Danish patent application No. PA 2017 70406 entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL" filed on May 29, 2017; Danish patent application No. PA 2017 70406 entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL" filed on May 29, 2017. Danish patent application No. PA 2017 70409, filed on May 29, 2017, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL"; Danish patent application No. PA 2017 70409, filed on May 29, 2017, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL".PA2017 70410; U.S. Provisional Patent Application No. 62 / 514,932, filed June 4, 2017, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL"; U.S. Provisional Patent Application No. 62 / 622,122, filed January 25, 2018, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL"; and Danish Patent Application No. PA 201870060, filed January 29, 2018, entitled "METHODS AND INTERFACES FOR HOME MEDIA CONTROL," the contents of each of these patent applications are incorporated herein by reference in their entirety.

[0004] This application also relates to U.S. Patent Application No. 14 / 300,120, filed June 9, 2014, entitled “MULTI-CHANNEL LOUDSPEAKERMATCHING VARIABLE DIRECTIVITY,” the entire contents of which are incorporated herein by reference and are included in the appendix. This application also relates to U.S. Patent Application No. 14 / 732,711, filed June 6, 2015, entitled “MULTI-MICROPHONE SPEECH RECOGNITION SYSTEMS AND RELATED TECHNIQUES,” the entire contents of which are incorporated herein by reference and are included in the appendix. This application also relates to U.S. Patent Application No. 14 / 732,715, filed June 6, 2015, entitled “MULTI-MICROPHONE SPEECH RECOGNITION TECHNIQUES AND RELATED SYSTEMS.” This application also relates to U.S. Patent Application No. 14 / 869,760, filed September 29, 2015, entitled “Modal-based Architecture for Controlling the Directivity of Loudspeare's Arrays,” the entire contents of which are incorporated herein by reference and are included in the appendix. This application also relates to U.S. Patent Application No. 15 / 266,956, filed September 15, 2016, entitled “Intelligent Authenticated Assistant for Media Exposure.” This application also relates to U.S. Patent Application No. 15 / 268,338, filed September 16, 2016, entitled “Intelligent Device Arbitration and Control,” the entire contents of which are incorporated herein by reference and are included in the appendix. This application also relates to U.S. Patent Application No. 15 / 272,214, filed September 21, 2016, entitled “INTELLIGENT LIST READING”. This application also relates to U.S. Patent Application No. 15 / 275,077, filed September 23, 2016, entitled “PRESSURE GRADIENT MICROPHONE FOR MEASURING AN ACOUSTIC CHARACTERISTIC OF A LOUDSPEAKER”.This application also relates to U.S. Patent Application No. 15 / 385,606, filed December 20, 2016, entitled “Intelligent Authenticated Assistant,” the contents of which are incorporated herein by reference in their entirety and are included in the appendix. This application also relates to U.S. Patent Application No. 15 / 504,312, filed August 18, 2014, entitled “A Rotationally Symmetrical Speaker Array.” This application also relates to U.S. Patent Application No. 15 / 514,455, filed September 25, 2015, entitled “Method Todetermine Loudspeaker Change of Placement.” This application also relates to U.S. Patent Application No. 15 / 583,949, filed May 1, 2017, entitled “A Rotationally Symmetrical Speaker Array.” This application also relates to U.S. Patent Application No. 15 / 593,887, filed May 12, 2017, entitled “SPATIAL AUDIORENDERING FOR BEAMFORMING LOUDSPEAKER ARRAY,” the contents of which are incorporated herein by reference in their entirety and are included in the appendix. This application also relates to U.S. Patent Application No. 62 / 004,111, filed May 28, 2014, entitled “MULTI-CHANNEL LOUDSPEAKER MATCHING USING VARIABLE DIRECTIVITY.” This application also relates to U.S. Patent Application No. 62 / 057,989, filed September 30, 2014, entitled “MODALBASED ARCHITECTURE FOR CONTROLLING THE DIRECTIVITY OF LOUDSPEAKER ARRAYS.” This application also relates to U.S. Patent Application No. 62 / 057,999, filed September 30, 2014, entitled “METHOD TODETERMINE LOUDSPEAKER CHANGE OF PLACEMENT”. This application also relates to U.S. Patent Application No. 62 / 346,236, filed June 6, 2016, entitled “INTELLIGENT LIST READING”. This application also relates to U.S. Patent Application No. 62 / 347,480, filed June 8, 2016, entitled “INTELLIGENT AUTOMATED ASSISTANT FOR MEDIA EXPLORATION”.This application also relates to U.S. Patent Application No. 62 / 348,896, filed June 11, 2016, entitled “INTELLIGENT DEVICE ARBITRATION AND CONTROL”. This application also relates to U.S. Patent Application No. 62 / 399,165, filed September 23, 2016, entitled “AUDIO DRIVER AND POWER SUPPLY UNIT ARCHITECTURE”. This application also relates to U.S. Patent Application No. 62 / 399,229, filed September 23, 2016, entitled “MULTI-LOBE MAGNET FOR SUBWOOFER”. This application also relates to U.S. Patent Application No. 62 / 399,232, filed September 23, 2016, entitled “INTELLIGENT AUTOMATED ASSISTANT”. This application also relates to U.S. Patent Application No. 62 / 399,262, filed September 23, 2016, entitled “FORWARD BACK VOLUME OF AIR FOR AUDIO DRIVER”. This application also relates to U.S. Patent Application No. 62 / 399,288, filed September 23, 2016, entitled “UPPER HOUSING COMPONENT ARCHITECTURE”. This application also relates to U.S. Patent Application No. 62 / 399,293, filed September 23, 2016, entitled “AUDIO DRIVER EXIT GEOMETRY”. This application also relates to U.S. Patent Application No. 62 / 402,836, filed September 30, 2016, entitled “SPATIAL AUDIO RENDERING STRATEGIES FOR BEAMFORMING LOUD SPEAKER ARRAY”. This application also relates to U.S. Patent Application No. 62 / 506,912, filed May 16, 2017, entitled “MANAGING MEDIA COLLECTIONS USING DIRECTED ACYCLIC GRAPHS”. This application also relates to U.S. Patent Application No. 62 / 507,007, filed May 16, 2017, entitled “SPEAKER”. This application also relates to U.S. Patent Application No. 62 / 507,100, filed May 16, 2017, entitled “TRANSFERRING PLAYBACK QUEUES BETWEENDEVICES”.This application also relates to U.S. Patent Application No. 62 / 507,151, filed May 16, 2017, entitled “FAR-FIELD EXTENSION FOR DIGITAL ASSISTANT SERVICES,” the contents of which are incorporated herein by reference in their entirety and are included in the appendix. This application also relates to U.S. Patent Application No. 62 / 507,056, filed May 16, 2017, entitled “PROVIDING AN AUDITORY-BASED INTERFACE OF A DIGITAL ASSISTANT.” This application also relates to U.S. Patent Application No. 62 / 506,871, filed May 16, 2017, entitled “METHODS AND SYSTEMS FOR PHONETIC MATCHING IN DIGITAL ASSISTANT SERVICE.” This application also relates to U.S. Patent Application No. 62 / 507,042, filed May 16, 2017, entitled “DETECTING ATRIGGER OF A DIGITAL ASSISTANT”. This application also relates to U.S. Patent Application No. 62 / 506,981, filed May 16, 2017, entitled “INTELLIGENT AUTOMATED ASSISTANT FOR MEDIA EXPLORATION”. This application also relates to a prospective U.S. Patent Application entitled “AUDIO BLUR” (Attorney’s File No. P34400US1). This application also relates to a prospective U.S. Patent Application entitled “ONLINE PROCESSING METHODS FOR MULTI-CHANNEL DEREVERBERATION” (Attorney’s File No. P32630US1). This application also relates to a prospective U.S. Patent Application entitled “AUTOMATIC LOUDSPEAKER ORIENTATION ESTIMATION” (Attorney’s File No. P34635US1). This application also relates to a U.S. patent application (Attorney’s File No. P34719US1) entitled “FOURIER TRANSFORM BASED METHOD FOR CREATING SMOOTH DIRECTIVITY TRANSITIONS IN BEAMFORMING LOUDSPEAKERS”, which is to be filed.This application also relates to the following U.S. patent applications: * **ACOUSTIC ECHO-CANCELLATION USING AN INTERNAL MICROPHONE** (Attorney's File No. P34811US1); * **TECHNIQUESFOR VERIFYING USER INTENT AND SECURELY CONFIGURING COMPUTER DEVICES** (Attorney's File No. P34845USP1); and * **SHARING CONCEPTS** (Attorney's File No. P34762USP1). The contents of each of the above-listed applications are incorporated herein by reference in their entirety. Technical Field

[0005] This disclosure relates in general to computer user interfaces, and more specifically to techniques for managing media playback devices. Background Technology

[0006] The number of electronic devices, especially smart devices, in users' homes continues to increase. These devices are becoming increasingly interconnected, feature-rich, and capable of performing more complex tasks. As a result, users increasingly expect these devices to have well-designed user interfaces. Summary of the Invention

[0007] However, some technologies used to manage media playback devices with electronic devices are often cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may involve multiple keystrokes or button presses. These technologies require more time than necessary, resulting in wasted user time and device power. This latter consideration is particularly important in battery-powered devices.

[0008] Therefore, this technology provides electronic devices with faster and more efficient methods and interfaces for managing media playback devices. Such methods and interfaces optionally complement or replace other methods used to manage media playback devices. These methods and interfaces reduce the cognitive burden on users and result in more efficient human-machine interfaces. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charging. Such methods and interfaces can reduce the amount of unnecessary, irrelevant, or repetitive user input. Furthermore, such methods and interfaces can reduce the power consumption of electronic devices' displays or speakers.

[0009] According to some implementations, a method is performed at a device having a display. The method includes: receiving user input; and in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback status of the plurality of available playback devices connected to the device and usable for initiating playback of media from the device.

[0010] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: receiving user input; and, in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback status of the plurality of available playback devices connected to and usable for initiating playback of media from the device.

[0011] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving user input; and, in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback status of the plurality of available playback devices connected to the device and available for initiating playback of media from the device.

[0012] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for receiving user input; and means for displaying a multi-device interface on the display in response to receiving the user input, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback status of the plurality of available playback devices connected to and usable for initiating playback of media from the device.

[0013] According to some embodiments, a transient computer-readable storage medium is described. This transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: receiving user input; and, in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback status of the plurality of available playback devices connected to and usable for initiating playback of media from the device.

[0014] According to some implementations, a method is performed on a device having a display. The method includes: displaying on the display a first current media interface for the device, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enablement representation; receiving first user input indicating a selection of the playback device selection enablement representation; in response to receiving the first user input, displaying on the display one or more enablement representations associated with a plurality of available playback devices connected to the device; receiving second user input indicating a selection of the one or more enablement representations associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0015] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: displaying on the display a first current media interface for the device, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enablement representation; receiving first user input indicating a selection of the playback device selection enablement representation; in response to receiving the first user input, displaying on the display one or more enablement representations associated with a plurality of available playback devices connected to the device; receiving second user input indicating a selection of the one or more enablement representations associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0016] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a first current media interface for the device on the display, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enablement representation; receiving first user input indicating a selection of the playback device selection enablement representation; in response to receiving the first user input, displaying one or more enablement representations associated with a plurality of available playback devices connected to the device on the display; receiving second user input indicating a selection of the one or more enablement representations associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0017] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for displaying on the display a first current media interface for the device, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enablement representation; means for receiving first user input indicating a selection of the playback device selection enablement representation; means for displaying on the display one or more enablement representations associated with a plurality of available playback devices connected to the device in response to receiving the first user input; means for receiving second user input indicating a selection of the one or more enablement representations associated with the plurality of available playback devices; and means for configuring the device to initiate playback on the plurality of available playback devices in response to receiving the second user input.

[0018] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: displaying on the display a first current media interface for the device, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enablement representation; receiving first user input indicating a selection of the playback device selection enablement representation; in response to receiving the first user input, displaying on the display one or more enablement representations associated with a plurality of available playback devices connected to the device; receiving second user input indicating a selection of the one or more enablement representations associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0019] According to some implementations, a method is performed at a first device. The method includes: detecting an indication that the physical proximity between the first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on the detected indication, wherein the audio tone signal is out-of-band for the communication link; following the output of the audio tone signal, determining whether a response audio tone signal has been received from the second device; proceeding to the next step in the configuration process based on the determination that a response audio tone signal has been received from the second device; and abandoning the progress to the next step in the configuration process based on the determination that no response audio tone signal has been received from the second device.

[0020] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting an indication that the physical proximity between a first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on the detected indication, wherein the audio tone signal is out-of-band for the communication link; after outputting the audio tone signal, determining whether a response audio tone signal has been received from the second device; proceeding to the next step in the configuration process based on the determination that a response audio tone signal has been received from the second device; and abandoning the progress to the next step in the configuration process based on the determination that no response audio tone signal has been received from the second device.

[0021] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting an indication that the physical proximity between a first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on the detected indication, wherein the audio tone signal is out-of-band for the communication link; following the output of the audio tone signal, determining whether a response audio tone signal has been received from the second device; proceeding to the next step in the configuration process based on the determination that a response audio tone signal has been received from the second device; and abandoning the progress to the next step in the configuration process based on the determination that no response audio tone signal has been received from the second device.

[0022] According to some embodiments, an electronic device is described. The electronic device includes: means for detecting an indication that the physical proximity between a first device and a second device meets a proximity condition, wherein the first device is connected to the second device via a communication link; means for audibly outputting an audio tone signal upon detecting the indication, wherein the audio tone signal is out-of-band for the communication link; means for determining, after outputting the audio tone signal, whether a response audio tone signal has been received from the second device; means for proceeding to the next step in the configuration process based on determining that a response audio tone signal has been received from the second device; and means for abandoning proceeding to the next step in the configuration process based on determining that a response audio tone signal has not been received from the second device.

[0023] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting an indication that the physical proximity between a first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on the detected indication, wherein the audio tone signal is out-of-band for the communication link; following the output of the audio tone signal, determining whether a response audio tone signal has been received from the second device; proceeding to the next step in the configuration process based on the determination that a response audio tone signal has been received from the second device; and abandoning the progress to the next step in the configuration process based on the determination that no response audio tone signal has been received from the second device.

[0024] According to some embodiments, a method is performed at a first device having a display. The method includes: detecting, when connected to a second device, an indication that the physical proximity between the first device and the second device meets a proximity condition; and displaying a media information interface on the display based on the detection that the physical proximity meets the proximity condition.

[0025] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: upon connection to a second device, detecting an indication that the physical proximity between the first device and the second device satisfies a proximity condition; and, based on detecting that the physical proximity satisfies the proximity condition, displaying a media information interface on the display.

[0026] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: upon connection to a second device, detecting an indication that the physical proximity between the electronic device and the second device meets a proximity condition; and, based on detecting that the physical proximity meets the proximity condition, displaying a media information interface on the display.

[0027] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for detecting, when connected to a second device, that the physical proximity between the electronic device and the second device meets a proximity condition; and means for displaying a media information interface on the display based on the detection that the physical proximity meets the proximity condition.

[0028] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: upon connection to a second device, detecting an indication that the physical proximity between the first device and the second device satisfies a proximity condition; and, based on detecting that the physical proximity satisfies the proximity condition, displaying a media information interface on the display.

[0029] According to some implementations, a method is performed on a device having a touch-sensitive surface. The method includes: detecting a touch on the touch-sensitive surface; in response to detecting a touch on the touch-sensitive surface, determining whether the device is currently playing media; based on determining that the device is currently playing media: determining whether the location of the touch is within a first area of ​​the touch-sensitive surface or within a second area of ​​the touch-sensitive surface, wherein the first and second areas do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping media playback; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the currently playing media; and based on determining that the device is not currently playing media, initiating media playback.

[0030] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a touch-sensitive surface, the one or more programs including instructions for performing the following operations: detecting a touch on the touch-sensitive surface; in response to detecting a touch on the touch-sensitive surface, determining whether the device is currently playing media; based on determining that the device is currently playing media: determining whether the location of the touch is within a first area or a second area of ​​the touch-sensitive surface, wherein the first and second areas do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping the playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the currently playing media; and based on determining that the device is not currently playing media, initiating playback of the media.

[0031] According to some embodiments, an electronic device is described. The electronic device includes: a touch-sensitive surface; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting a touch on the touch-sensitive surface; in response to detecting a touch on the touch-sensitive surface, determining whether the device is currently playing media; and, based on determining that the device is currently playing media: determining whether the location of the touch is within a first area or a second area of ​​the touch-sensitive surface, wherein the first area and the second area do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the currently playing media; and based on determining that the device is not currently playing media, initiating playback of the media.

[0032] According to some embodiments, an electronic device is described. The electronic device includes: a touch-sensitive surface; means for detecting a touch on the touch-sensitive surface; means for determining whether the device is currently playing media in response to detecting a touch on the touch-sensitive surface; means for performing the following operations: based on determining that the device is currently playing media: determining whether the location of the touch is within a first area or a second area of ​​the touch-sensitive surface, wherein the first and second areas do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the currently playing media; and means for initiating playback of the media based on determining that the device is not currently playing media.

[0033] According to some embodiments, a transient computer-readable storage medium is described. This transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a touch-sensitive surface, the one or more programs including instructions for performing the following operations: detecting a touch on the touch-sensitive surface; in response to detecting a touch on the touch-sensitive surface, determining whether the device is currently playing media; and, based on determining that the device is currently playing media: determining whether the location of the touch is within a first area or a second area of ​​the touch-sensitive surface, wherein the first and second areas do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping the playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the currently playing media; and based on determining that the device is not currently playing media, initiating playback of the media.

[0034] According to some implementations, a method is performed at a first device. The method includes: receiving a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the first device; adding the first media item to a playback queue; receiving a second request to queue a second media item for playback on the set of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the set of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the set of devices.

[0035] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the first device; adding the first media item to the playback queue; receiving a second request to queue a second media item for playback on the set of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the set of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the set of devices.

[0036] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the electronic device; adding the first media item to the playback queue; receiving a second request to queue a second media item for playback on the set of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the set of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the set of devices.

[0037] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the electronic device; means for adding the first media item to the playback queue; means for receiving a second request to queue a second media item for playback on the set of devices; means for adding the second media item to the playback queue; means for accessing the first media item from a first media streaming service; means for initiating playback of the first media item on the set of devices; means for accessing the second media item from a second media streaming service different from the first media streaming service; and means for initiating playback of the second media item on the set of devices.

[0038] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the first device; adding the first media item to the playback queue; receiving a second request to queue a second media item for playback on the set of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the set of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the set of devices.

[0039] According to some implementations, a method is performed at a first device. The method includes: receiving from a second device associated with a first user a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the first device; adding the first media item to the playback queue; receiving from a third device associated with a second user a second request to queue a second media item for playback on the set of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the set of devices; and initiating playback of the second media item on the set of devices.

[0040] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving from a second device associated with a first user a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the first device; adding the first media item to the playback queue; receiving from a third device associated with a second user a second request to queue a second media item for playback on the set of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the set of devices; and initiating playback of the second media item on the set of devices.

[0041] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a first request from a second device associated with a first user to queue a first media item for playback on a set of devices, wherein the set of devices includes the electronic device; adding the first media item to the playback queue; receiving a second request from a third device associated with a second user to queue a second media item for playback on the set of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the set of devices; and initiating playback of the second media item on the set of devices.

[0042] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving from a second device associated with a first user a first request to queue a first media item for playback on a set of devices, wherein the set of devices includes the electronic device; means for adding the first media item to the playback queue; means for receiving from a third device associated with a second user a second request to queue a second media item for playback on the set of devices, wherein the second user is different from the first user; means for adding the second media item to the playback queue; means for initiating playback of the first media item on the set of devices; and means for initiating playback of the second media item on the set of devices.

[0043] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving a first request from a second device associated with a first user to queue a first media item for playback on a set of devices, wherein the set of devices includes the first device; adding the first media item to the playback queue; receiving a second request from a third device associated with a second user to queue a second media item for playback on the set of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the set of devices; and initiating playback of the second media item on the set of devices.

[0044] According to some embodiments, a method is performed at a first device having a display. The method includes: detecting an audio tone signal indicating a pairing handshake with a second device, wherein the first device is connected to the second device via a communication link, and wherein the audio tone signal is out-of-band for the communication link; initiating a configuration process for configuring the second device on the first device based on the detected audio tone signal; displaying a configuration user interface including prompts for selecting a location on the display during the configuration process; receiving user input selection for the location; and configuring the second device to be associated with the location in response to receiving the user input.

[0045] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: detecting an audio tone signal indicating a pairing handshake with a second device, wherein the first device is connected to the second device via a communication link, and wherein the audio tone signal is out-of-band for the communication link; initiating a configuration process for configuring the second device on the first device based on the detected audio tone signal; displaying a configuration user interface including prompts for selecting a location on the display during the configuration process; receiving user input selection of a location; and configuring the second device to be associated with the location in response to receiving the user input.

[0046] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting an audio tone signal indicating a pairing handshake with a second device, wherein the electronic device is connected to the second device via a communication link, and wherein the audio tone signal is out-of-band for the communication link; initiating a configuration process for configuring the second device on the electronic device based on the detected audio tone signal; displaying a configuration user interface including prompts for selecting a location on the display during the configuration process; receiving user input selection of a location; and configuring the second device to be associated with the location in response to receiving the user input.

[0047] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for detecting an audio tone signal indicating a pairing handshake with a second device, wherein the electronic device is connected to the second device via a communication link, and wherein the audio tone signal is out-of-band for the communication link; means for initiating a configuration process for configuring the second device on the electronic device based on the detected audio tone signal; means for displaying a configuration user interface including prompts for selecting a location on the display during the configuration process; means for receiving user input selection of a location; and means for configuring the second device to be associated with the location in response to receiving the user input.

[0048] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: detecting an audio tone signal indicating a pairing handshake with a second device, wherein the first device is connected to the second device via a communication link, and wherein the audio tone signal is out-of-band for the communication link; initiating a configuration process for configuring the second device on the first device based on the detected audio tone signal; displaying a configuration user interface including prompts for selecting a location on the display during the configuration process; receiving user input selection of a location; and configuring the second device to be associated with the location in response to receiving the user input.

[0049] According to some implementations, a method is performed at a device. The method includes: detecting a user-input voice command from a voice command source while outputting an audible signal from the device; determining the spatial position of the voice command source relative to the device in response to detecting the user-input voice command; and adjusting the output of the audible signal based on the spatial position of the voice command source while continuing to output the audible signal.

[0050] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting a user-inputted voice command from a voice command source while an audible signal is output from the device; determining the spatial location of the voice command source relative to the device in response to detecting the user-inputted voice command; and adjusting the output of the audible signal based on the spatial location of the voice command source while the audible signal continues to be output.

[0051] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting a user-input voice command from a voice command source while outputting an audible signal from the device; determining the spatial position of the voice command source relative to the device in response to detecting the user-input voice command; and adjusting the output of the audible signal based on the spatial position of the voice command source while continuing to output the audible signal.

[0052] According to some embodiments, an electronic device is described. The electronic device includes: means for detecting a user-input voice command from a voice command source while outputting an audible signal from the device; means for determining the spatial position of the voice command source relative to the device in response to detecting the user-input voice command; and means for adjusting the output of the audible signal based on the spatial position of the voice command source while continuing to output the audible signal.

[0053] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting a user-input voice command from a voice command source while an audible signal is output from the device; determining the spatial location of the voice command source relative to the device in response to detecting the user-input voice command; and adjusting the output of the audible signal based on the spatial location of the voice command source while the output of the audible signal continues.

[0054] According to some implementations, a method is performed on a first device associated with a first user account. The method includes: detecting that a second device associated with a second user account is connected to the first device, wherein the first user account is associated with a first media preference profile; wherein the second user account is associated with a second media preference profile different from the first media preference profile; and initiating media playback based on the first and second media preference profiles.

[0055] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device associated with a first user account, wherein the first user account is associated with a first media preference profile, and the one or more programs include instructions for performing the following operations: detecting that a second device associated with a second user account is connected to the first device, wherein the second user account is associated with a second media preference profile different from the first media preference profile; and initiating media playback based on the first media preference profile and the second media preference profile.

[0056] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting that a second device associated with a second user account is connected to the electronic device, wherein the electronic device is associated with a first user account, wherein the first user account is associated with a first media preference profile; wherein the second user account is associated with a second media preference profile different from the first media preference profile; and initiating media playback based on the first media preference profile and the second media preference profile.

[0057] According to some embodiments, an electronic device is described. The electronic device includes: means for detecting that a second device associated with a second user account is connected to the electronic device, wherein the electronic device is associated with a first user account, wherein the first user account is associated with a first media preference profile; wherein the second user account is associated with a second media preference profile different from the first media preference profile; and means for initiating media playback based on the first media preference profile and the second media preference profile.

[0058] According to some embodiments, a transient computer-readable storage medium is described. This transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device associated with a first user account, wherein the first user account is associated with a first media preference profile, and the one or more programs include instructions for performing the following operations: detecting that a second device associated with a second user account is connected to the first device, wherein the second user account is associated with a second media preference profile different from the first media preference profile; and initiating media playback based on the first media preference profile and the second media preference profile.

[0059] According to some implementations, a method is performed at a first device. The method includes: receiving from a third device a request to initiate media playback on at least one device in a group of devices, wherein the group of devices includes a first device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the first device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and initiating media playback on the group of devices based on determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to a second device; and initiating media playback on the at least one device in the group of devices in response to receiving permission from the second device.

[0060] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving from a third device a request to initiate media playback on at least one device in a group of devices, wherein the group of devices includes the first device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the first device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and initiating media playback on the group of devices based on determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to a second device; and initiating media playback on the at least one device in the group of devices in response to receiving permission from the second device.

[0061] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving from a third device a request to initiate media playback on at least one device in a group of devices, wherein the group of devices includes the electronic device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the electronic device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and initiating media playback on the group of devices based on determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to a second device; and initiating media playback on the at least one device in the group of devices in response to receiving permission from the second device.

[0062] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving from a third device a request to initiate media playback on at least one device in a group of devices, wherein the group of devices includes the electronic device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the electronic device; and means for performing the following operations: in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and initiating media playback on the group of devices based on determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to a second device; and initiating media playback on the at least one device in the group of devices in response to receiving permission from the second device.

[0063] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving from a third device a request to initiate media playback on at least one device in a group of devices, wherein the group of devices includes the first device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the first device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and initiating media playback on the group of devices based on determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to a second device; and initiating media playback on the at least one device in the group of devices in response to receiving permission from the second device.

[0064] According to some implementation schemes, a method is performed at a first device associated with a user account. The method includes: receiving user input representing a request for access to data from the user account; in response to receiving the user input, determining whether the physical locations of the first device and a second device associated with the user account satisfy the licensing conditions; based on the determination that the physical locations satisfy the licensing conditions, providing the data from the user account through the first device; and based on the determination that the physical locations do not satisfy the licensing conditions, abandoning the provision of the data from the user account through the first device.

[0065] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first electronic device associated with a user account, the one or more programs including instructions for performing the following operations: receiving user input representing a request for access to data from the user account; in response to receiving the user input, determining whether the physical location of the electronic device and a second device associated with the user account satisfies licensing conditions; based on the determination that the physical location satisfies licensing conditions, providing the data from the user account through the electronic device; and based on the determination that the physical location does not satisfy licensing conditions, abandoning the provision of the data from the user account through the electronic device.

[0066] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving user input representing a request for accessing data from a user account, wherein the electronic device is associated with the user account; in response to receiving the user input, determining whether the physical location of the electronic device and a second device associated with the user account satisfies licensing conditions; based on determining that the physical location satisfies licensing conditions, providing the data from the user account through the electronic device; and based on determining that the physical location does not satisfy licensing conditions, abandoning the provision of the data from the user account through the electronic device.

[0067] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving user input representing a request for accessing data from a user account, wherein the electronic device is associated with the user account; means for determining, in response to receiving the user input, whether the physical location of the electronic device and a second device associated with the user account satisfies licensing conditions; means for providing data from the user account through the electronic device based on the determination that the physical location satisfies licensing conditions; and means for abandoning the provision of data from the user account through the electronic device based on the determination that the physical location does not satisfy licensing conditions.

[0068] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first electronic device associated with a user account, the one or more programs including instructions for performing the following operations: receiving user input representing a request for access to data from the user account; in response to receiving the user input, determining whether the physical location of the electronic device and a second device associated with the user account satisfies licensing conditions; based on determining that the physical location satisfies licensing conditions, providing the data from the user account through the electronic device; and based on determining that the physical location does not satisfy licensing conditions, abandoning the provision of the data from the user account through the electronic device.

[0069] According to some implementations, a method is performed at a first device. The method includes: receiving a command for initiating playback of media on a set of devices including the first device and the second device; and initiating synchronous playback of media on the set of devices in response to receiving the command, wherein the second device is designated as a master device configured to coordinate the playback of media on the set of devices, and wherein the first device is configured as a designated successor of the master device.

[0070] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving a command for initiating playback of media on a set of devices including the first device and the second device; and initiating synchronized playback of media on the set of devices in response to receiving the command, wherein the second device is designated as a master device configured to coordinate the playback of media on the set of devices, and wherein the first device is configured as a designated successor of the master device.

[0071] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a command for initiating playback of media on a group of devices including the electronic device and a second device; and initiating synchronized playback of media on the group of devices in response to receiving the command, wherein the second device is designated as a master device configured to coordinate the playback of media on the group of devices, and wherein the electronic device is configured as a designated successor to the master device.

[0072] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving a command for initiating playback of media on a set of devices including the electronic device and a second device; means for initiating synchronous playback of media on the set of devices in response to receiving the command, wherein the second device is designated as a master device configured to coordinate the playback of media on the set of devices, and wherein the electronic device is configured as a designated successor to the master device.

[0073] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving a command for initiating playback of media on a set of devices including the first device and the second device; and initiating synchronized playback of media on the set of devices in response to receiving the command, wherein the second device is designated as a master device configured to coordinate the playback of media on the set of devices, and wherein the first device is configured as a designated successor of the master device.

[0074] According to some implementations, a method is performed at a first device associated with a first user account in a group of devices. The method includes: receiving an audible output trigger event; determining that a first user associated with the first user account is within the audible output range of one of the devices in the group of devices; outputting audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output trigger event; and abandoning the output of audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output trigger event.

[0075] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device associated with a first user account in a group of devices, the one or more programs including instructions for performing the following operations: receiving an audible output trigger event; determining that a first user associated with the first user account is within the audible output range of one of the devices in the group of devices; outputting audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output trigger event; and abandoning the output of audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output trigger event.

[0076] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving an audible output trigger event; determining that a first user associated with a first user account is within the audible output range of a device associated with the first user account in a group of devices, wherein the electronic device is the first device in the group of devices; outputting audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output trigger event; and abandoning the output of audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output trigger event.

[0077] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving an audible output trigger event; means for determining that a first user associated with a first user account is within the audible output range of a device associated with the first user account in a group of devices, wherein the electronic device is a first device in the group of devices; means for outputting audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output trigger event; and means for abandoning the output of audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output trigger event.

[0078] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device associated with a first user account in a group of devices, the one or more programs including instructions for performing the following operations: receiving an audible output trigger event; determining that a first user associated with the first user account is within the audible output range of one of the devices in the group of devices; outputting audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output trigger event; and abandoning the output of audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output trigger event.

[0079] According to some embodiments, a method is performed at a first device. The method includes: receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the first device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving the information about the media currently being played back by the second device; and, based on receiving the information about the media currently being played back by the second device and while the second device continues playback of at least one component of the media, initiating the output of at least one component of the signal representing the media currently being played back by the second device.

[0080] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the first device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving the information about the media currently being played back by the second device; and, based on receiving the information about the media currently being played back by the second device and while the second device continues playback of at least one component of the media, initiating the output of at least one component of the signal representing the media currently being played back by the second device.

[0081] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the electronic device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving the information about the media currently being played back by the second device; and, based on receiving the information about the media currently being played back by the second device and while the second device continues playback of at least one component of the media, initiating the output of at least one component of the signal representing the media currently being played back by the second device.

[0082] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the electronic device; means for transmitting a request for information about the media currently being played back by the second device in response to receiving the user input; means for receiving the information about the media currently being played back by the second device; and means for performing the following operation: based on receiving the information about the media currently being played back by the second device and while the second device continues playback of at least one component of the media, initiating the output of at least one component of the signal representing the media currently being played back by the second device.

[0083] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the first device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving the information about the media currently being played back by the second device; and, based on receiving the information about the media currently being played back by the second device and while the second device continues playback of at least one component of the media, initiating the output of at least one component of the signal representing the media currently being played back by the second device.

[0084] According to some implementations, a method is performed at a first device. The method includes: receiving a request to initiate media playback on a group of devices excluding the first device; in response to receiving the request and while the first device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes a device configured to coordinate media playback, transferring a media playback coordination task from the first device to the device configured to coordinate media playback; and based on determining that the group of devices does not include a device configured to coordinate media playback, abandoning the transmission of the media playback coordination task.

[0085] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices excluding the first device; in response to receiving the request and while the first device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes a device configured to coordinate media playback, transferring a media playback coordination task from the first device to the device configured to coordinate media playback; and based on determining that the group of devices does not include a device configured to coordinate media playback, abandoning the transfer of the media playback coordination task.

[0086] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices excluding the electronic device; in response to receiving the request and while the electronic device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes a device configured to coordinate media playback, transferring a media playback coordination task from the electronic device to the device configured to coordinate media playback; and based on determining that the group of devices does not include a device configured to coordinate media playback, abandoning the transfer of the media playback coordination task.

[0087] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving a request to initiate media playback on a group of devices excluding the electronic device; means for determining, in response to receiving the request and while the electronic device is coordinating media playback, whether the group of devices includes a device configured to coordinate media playback; means for transferring a media playback coordination task from the electronic device to the device configured to coordinate media playback based on the determination that the group of devices includes the device configured to coordinate media playback; and means for abandoning the transmission of the media playback coordination task based on the determination that the group of devices does not include the device configured to coordinate media playback.

[0088] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices excluding the first device; in response to receiving the request and while the first device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes a device configured to coordinate media playback, transferring a media playback coordination task from the first device to the device configured to coordinate media playback; and based on determining that the group of devices does not include a device configured to coordinate media playback, abandoning the transfer of the media playback coordination task.

[0089] According to some implementations, a method is performed at a device. The method includes: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transferring a media playback coordination task from the device to the master device configured to coordinate media playback.

[0090] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transferring a media playback coordination task from the device to the master device configured to coordinate media playback.

[0091] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transferring a media playback coordination task from the device to the master device configured to coordinate media playback.

[0092] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving a request to initiate media playback on a group of devices connected to the device; means for determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices in response to receiving the request and while the device is coordinating media playback; means for selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and means for transmitting a media playback coordination task from the device to the master device configured to coordinate media playback.

[0093] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transferring a media playback coordination task from the device to the master device configured to coordinate media playback.

[0094] According to some embodiments, a method is performed at a first device having a display. The method includes: establishing a communication link with a second device; following the establishment of the communication link with the second device, initiating a configuration process for configuring the second device; during the configuration process: displaying a configuration user interface on the display including prompts for selecting a location; receiving user input selection for the location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicating a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and based on the detection of the audio tone signal, proceeding to the next step in the configuration process for configuring the second device.

[0095] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: establishing a communication link with a second device; subsequently, after establishing the communication link with the second device, initiating a configuration process for configuring the second device; during the configuration process: displaying a configuration user interface on the display including prompts for selecting a location; receiving user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicating a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and based on the detection of the audio tone signal, proceeding to the next step in the configuration process for configuring the second device.

[0096] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: establishing a communication link with a second device; subsequently, initiating a configuration process for configuring the second device after establishing the communication link with the second device; during the configuration process: displaying a configuration user interface on the display including prompts for selecting a location; receiving user input selection for a location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicating a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and proceeding to the next step in the configuration process for configuring the second device based on the detected audio tone signal.

[0097] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for establishing a communication link with a second device; means for initiating a configuration process for configuring the second device after establishing the communication link with the second device; during the configuration process: means for displaying a configuration user interface on the display including prompts for selecting a location; means for receiving user input selection of a location; and means for configuring the second device as associated with the location in response to receiving the user input; means for detecting an audio tone signal indicating a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and means for proceeding to the next step in the configuration process for configuring the second device based on the detected audio tone signal.

[0098] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: establishing a communication link with a second device; subsequently, initiating a configuration process for configuring the second device after establishing the communication link with the second device; during the configuration process: displaying a configuration user interface on the display including prompts for selecting a location; receiving user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicating a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and proceeding to the next step in the configuration process for configuring the second device based on the detected audio tone signal.

[0099] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0100] Therefore, faster and more efficient methods and interfaces are provided for managing media playback devices, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. These methods and interfaces can complement or replace other methods used to manage media playback devices. Attached Figure Description

[0101] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0102] Figure 1AThis is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some embodiments.

[0103] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some implementation schemes.

[0104] Figure 2 A portable multi-functional device with a touchscreen is shown according to some embodiments.

[0105] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some implementation schemes.

[0106] Figure 4A An exemplary user interface for a menu on an application on a portable multifunction device, according to some implementation schemes, is shown.

[0107] Figure 4B An exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0108] Figure 5A A personal electronic device according to some implementation schemes is shown.

[0109] Figure 5B This is a block diagram illustrating a personal electronic device according to some implementation schemes.

[0110] Figures 5C-5D Exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor according to some embodiments are shown.

[0111] Figures 5E-5H Exemplary components and user interfaces of a personal electronic device according to some implementation schemes are shown.

[0112] Figure 5I An electronic device according to some implementation schemes is shown.

[0113] Figure 5J This is a block diagram illustrating an electronic device according to some implementation schemes.

[0114] Figures 6A-6AR Exemplary technologies and user interfaces for playing back media are shown according to some implementation schemes.

[0115] Figures 7A-7D This is a flowchart illustrating a process for playing back media according to some implementation schemes.

[0116] Figures 8A-8P Exemplary technologies and user interfaces for playing back media are shown according to some implementation schemes.

[0117] Figures 9A-9C This is a flowchart illustrating a process for playing back media according to some implementation schemes.

[0118] Figures 10A-10O Exemplary technologies and user interfaces according to some implementation schemes are shown.

[0119] Figures 11A-11D This is a flowchart illustrating a process for displaying content according to some implementation schemes.

[0120] Figures 12A-12AM Exemplary technologies and user interfaces for viewing media information are shown according to some implementation schemes.

[0121] Figures 13A-13F This is a flowchart illustrating a process for viewing media information according to some implementation schemes.

[0122] Figures 14A-14T Exemplary technologies and user interfaces for controlling electronic devices are shown according to some implementation schemes.

[0123] Figures 15A-15C This is a flowchart illustrating a process for controlling an electronic device according to some implementation schemes.

[0124] Figures 16A-16P Exemplary techniques and user interfaces for managing replay queues are shown according to some implementation schemes.

[0125] Figures 17A-17C This is a flowchart illustrating a process for managing a replay queue according to some implementation schemes.

[0126] Figures 18A-18C This is a flowchart illustrating a process for managing a replay queue according to some implementation schemes.

[0127] Figures 19A-19AL Exemplary technologies and user interfaces for configuring electronic devices are shown according to some implementation schemes.

[0128] Figures 20A-20C This is a flowchart illustrating a process for configuring an electronic device according to some implementation schemes.

[0129] Figures 21A-21I Exemplary techniques and user interfaces for adjusting the audible output of an electronic device, according to some implementation schemes, are shown.

[0130] Figures 22A-22B This is a flowchart illustrating a process for adjusting the audible output of an electronic device according to some implementation schemes.

[0131] Figures 23A-23KExemplary techniques and user interfaces for managing replay queues are shown according to some implementation schemes.

[0132] Figures 24A-24B This is a flowchart illustrating a process for managing a replay queue according to some implementation schemes.

[0133] Figures 25A-25I Exemplary technologies and user interfaces for playing back media are shown according to some implementation schemes.

[0134] Figures 26A-26B This is a flowchart illustrating a process for playing back media according to some implementation schemes.

[0135] Figures 27A-27E Exemplary technologies and user interfaces for providing access to electronic devices are shown according to some implementation schemes.

[0136] Figures 28A-28C This is a flowchart illustrating a process for providing access to an electronic device according to some implementation schemes.

[0137] Figures 29A-29I Exemplary technologies and user interfaces for managing media playback are shown according to some implementation schemes.

[0138] Figures 30A-30C This is a flowchart illustrating a process for managing media playback according to some implementation schemes.

[0139] Figures 31A-31N Exemplary technologies and user interfaces for providing output are shown according to some implementation schemes.

[0140] Figures 32A-32C This is a flowchart illustrating a process for providing output according to some implementation schemes.

[0141] Figures 33A-33G Exemplary technologies and user interfaces for controlling media playback are shown according to some implementation schemes.

[0142] Figures 34A-34B This is a flowchart illustrating a process for controlling media playback according to some implementation schemes.

[0143] Figures 35A-35J Exemplary technologies and user interfaces for managing media playback are shown according to some implementation schemes.

[0144] Figures 36A-36C This is a flowchart illustrating a process for managing media playback according to some implementation schemes.

[0145] Figures 37A-37DExemplary technologies and user interfaces for controlling media playback are shown according to some implementation schemes.

[0146] Figures 38A-38B This is a flowchart illustrating a process for managing media playback according to some implementation schemes.

[0147] Figures 39A-39C This is a flowchart illustrating a process for configuring an electronic device according to some implementation schemes. Detailed Implementation

[0148] The following description illustrates exemplary methods, parameters, etc. However, it should be understood that such description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.

[0149] Electronic devices require efficient methods and interfaces for managing media playback devices. For example, methods and techniques for transferring playback between devices are described below. Such techniques can reduce the cognitive burden on users accessing event notifications, thereby improving productivity. Furthermore, such techniques can reduce processor and battery power wasted otherwise on redundant user input.

[0150] under Figure 1A-Figure 1B , Figure 2 , Figure 3 , Figures 4A-4B and Figures 5A-5H A description of an exemplary device for performing management event notifications is provided. Figures 6A-6AR Exemplary technologies and user interfaces for playing back media are shown according to some implementation schemes. Figures 7A-7D This is a flowchart illustrating a process for playing back media according to some implementation schemes. Figures 6A-6AR The user interface in the document is used to illustrate the process described below, including Figures 7A-7D The process in. Figures 8A-8P Exemplary technologies and user interfaces for playing back media are shown according to some implementation schemes. Figures 9A-9C This is a flowchart illustrating a process for playing back media according to some implementation schemes. Figures 8A-8P The user interface in the document is used to illustrate the process described below, including Figures 9A-9C The process in. Figures 10A-10O Exemplary technologies and user interfaces according to some implementation schemes are shown. Figures 11A-11D This is a flowchart illustrating a process for displaying content according to some implementation schemes. Figures 10A-10O The user interface in the document is used to illustrate the process described below, including Figures 11A-11D The process in. Figures 12A-12AM Exemplary technologies and user interfaces for viewing media information are shown according to some implementation schemes. Figures 13A-13FThis is a flowchart illustrating a process for viewing media information according to some implementation schemes. Figures 12A-12AM The user interface in the document is used to illustrate the process described below, including Figures 13A-13F The process in. Figures 14A-14T Exemplary technologies and user interfaces for controlling electronic devices are shown according to some implementation schemes. Figures 15A-15C This is a flowchart illustrating a process for controlling an electronic device according to some implementation schemes. Figures 14A-14T The user interface in the document is used to illustrate the process described below, including Figures 15A-15C The process in. Figures 16A-16P Exemplary techniques and user interfaces for managing replay queues are shown according to some implementation schemes. Figures 17A-17C This is a flowchart illustrating a process for managing a replay queue according to some implementation schemes. Figures 18A-18C This is a flowchart illustrating a process for managing a replay queue according to some implementation schemes. Figures 16A-16P The user interface in the document is used to illustrate the process described below, including Figures 17A-17C and Figures 18A-18C The process in. Figures 19A-19AL Exemplary technologies and user interfaces for configuring electronic devices are shown according to some implementation schemes. Figures 20A-20C This is a flowchart illustrating a process for configuring an electronic device according to some implementation schemes. Figures 19A-19AL The user interface in the document is used to illustrate the process described below, including Figures 20A-20C The process in. Figures 21A-21I Exemplary techniques and user interfaces for adjusting the audible output of an electronic device, according to some implementation schemes, are shown. Figures 22A-22B This is a flowchart illustrating a process for adjusting the audible output of an electronic device according to some implementation schemes. Figures 21A-21I The user interface in the document is used to illustrate the process described below, including Figures 22A-22B The process in. Figures 23A-23K Exemplary techniques and user interfaces for managing replay queues are shown according to some implementation schemes. Figures 24A-24B This is a flowchart illustrating a process for managing a replay queue according to some implementation schemes. Figures 23A-23K The user interface in the document is used to illustrate the process described below, including Figures 24A-24B The process in. Figures 25A-25I Exemplary technologies and user interfaces for playing back media are shown according to some implementation schemes. Figures 26A-26B This is a flowchart illustrating a process for playing back media according to some implementation schemes. Figures 25A-25I The user interface in the document is used to illustrate the process described below, including Figures 26A-26B The process in. Figures 27A-27EExemplary technologies and user interfaces for providing access to electronic devices are shown according to some implementation schemes. Figures 28A-28C This is a flowchart illustrating a process for providing access to an electronic device according to some implementation schemes. Figures 27A-27E The user interface in the document is used to illustrate the process described below, including Figures 28A-28C The process in. Figures 29A-29I Exemplary technologies and user interfaces for managing media playback are shown according to some implementation schemes. Figures 30A-30C This is a flowchart illustrating a process for managing media playback according to some implementation schemes. Figures 29A-29I The user interface in the document is used to illustrate the process described below, including Figures 30A-30C The process in. Figures 31A-31N Exemplary technologies and user interfaces for providing output are shown according to some implementation schemes. Figures 32A-32C This is a flowchart illustrating a process for providing output according to some implementation schemes. Figures 31A-31N The user interface in the document is used to illustrate the process described below, including Figures 32A-32C The process in. Figures 33A-33G Exemplary technologies and user interfaces for controlling media playback are shown according to some implementation schemes. Figures 34A-34B This is a flowchart illustrating a process for controlling media playback according to some implementation schemes. Figures 33A-33G The user interface in the document is used to illustrate the process described below, including Figures 34A-34B The process in. Figures 35A-35J Exemplary technologies and user interfaces for managing media playback are shown according to some implementation schemes. Figures 36A-36C This is a flowchart illustrating a process for managing media playback according to some implementation schemes. Figures 35A-35J The user interface in the document is used to illustrate the process described below, including Figures 36A-36C The process in. Figures 37A-37D Exemplary technologies and user interfaces for managing media playback are shown according to some implementation schemes. Figures 38A-38B This is a flowchart illustrating a process for managing media playback according to some implementation schemes. Figures 37A-37D The user interface in the document is used to illustrate the process described below, including Figures 38A-38B The process in.

[0151] Although the following description uses the terms first, second, etc., to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch may be named a second touch and similarly, a second touch may be named a first touch, without departing from the scope of the various described embodiments. Both a first touch and a second touch are touches, but they are not the same touch.

[0152] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the items listed in connection with the description. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0153] Depending on the context, the term "if" may optionally be interpreted as meaning "when," "at," or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" may optionally be interpreted as meaning "in response to determination..." or "in response to detection [the stated condition or event]."

[0154] This document describes implementations of electronic devices, user interfaces for such devices, and related processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0155] In the following discussion, an electronic device comprising a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0156] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0157] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0158] Now let’s turn our attention to implementation schemes for portable devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and is sometimes referred to as or called a “touch-sensitive display system.” Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface such as the touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more haptic output generators 167 for generating haptic outputs on device 100 (e.g., generating haptic outputs on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0159] As used in this specification and claims, the term "intensity" for contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute for the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface adjacent to the contact, and / or the resistance and / or variation of the touch-sensitive surface adjacent to the contact are optionally used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that the user might not otherwise access on a smaller device with a limited solid area for (e.g., on a touch-sensitive display) displaying an indication and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0160] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the tactile surface remains unchanged, movement of the tactile surface can optionally be interpreted or sensed by the user as “roughness” of the tactile surface. While such interpretations of touch by the user will be limited by the user’s individualized sensory perception, there are many sensory perceptions of touch that are common to most users. Therefore, when a tactile output is described to correspond to a user’s specific sensory perception (e.g., “release click,” “press click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that will generate the sensory perception described by a typical (or ordinary) user.

[0161] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing circuits and / or application-specific integrated circuits.

[0162] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0163] Peripheral interface 118 can be used to couple the device's input and output peripherals to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In other embodiments, they are optionally implemented on separate chips.

[0164] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via near-field communication radio components. Wireless communication may optionally employ any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution, Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE...). 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol that has not been developed as of the date of this document submission.

[0165] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a mono or binaural headphone) and input (e.g., a microphone).

[0166] I / O subsystem 106 couples input / output peripherals on device 100, such as touchscreen 112 and other input control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / send electrical signals to the other input control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click wheels, etc. In some alternative embodiments, input controllers 160 are optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing devices such as a mouse. The one or more buttons (e.g., Figure 2 Optionally, 208) includes volume up / down buttons for volume control of speaker 111 and / or microphone 113. The one or more buttons optionally include push-button buttons (e.g., Figure 2 (206 in the middle).

[0167] A quick press of the push button optionally unlocks the touchscreen 112 or optionally initiates a process of unlocking the device using gestures on the touchscreen, as described in U.S. Patent Application 11 / 322,549 (U.S. Patent No. 7,657,849), filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," the entire contents of which are incorporated herein by reference. A long press of the push button (e.g., 206) optionally powers the device 100 on or off. The function of one or more buttons is optionally user-customizable. The touchscreen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0168] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0169] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In one exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.

[0170] Touchscreen 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. Touchscreen 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In one exemplary embodiment, projected mutual capacitance sensing technology, such as that used in Apple Inc. (Cupertino, California), is employed. and iPod The technology discovered in [the text].

[0171] In some embodiments of the touchscreen 112, the touch-sensitive display optionally resembles the multi-touch-sensitive touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touchscreen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.

[0172] In some embodiments, the touchscreen 112's touch-sensitive display is described in the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, entitled "Multipoint Touch Surface Controller"; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, entitled "Multipoint Touchscreen"; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, entitled "Gestures For Touch Sensitive Input Devices"; (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, entitled "Gestures For Touch Sensitive Input Devices"; and (5) U.S. Patent Application No. 18, 2005, entitled "Mode-Based Graphical User Interfaces For Touch Sensitive". U.S. Patent Application No. 11 / 038,590 entitled “Input Devices”; (6) U.S. Patent Application No. 11 / 228,758 entitled “Virtual Input Device Placement On A Touch Screen User Interface”, filed September 16, 2005; (7) U.S. Patent Application No. 11 / 228,700 entitled “Operation Of A Computer With A Touch Screen Interface”, filed September 16, 2005; (8) U.S. Patent Application No. 11 / 228,737 entitled “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”, filed September 16, 2005; and (9) U.S. Patent Application No. 11 / 367,749 entitled “Multi-Functional Hand-Held Device”, filed March 3, 2006. The entire contents of all these patent applications are incorporated herein by reference.

[0173] Touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. Users optionally use any suitable object or accessory such as a stylus, finger, etc., to interact with touchscreen 112. In some embodiments, the user interface is designed to work primarily with finger-based contact and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0174] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that differs from the touchscreen and does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0175] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0176] The device 100 may optionally also include one or more optical sensors 164. Figure 1AAn optical sensor 164 is shown coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing images of the user to be optionally acquired for video conferencing while the user views other video conferencing participants on the touchscreen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lenses and sensors within the device housing), allowing a single optical sensor 164 to be used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.

[0177] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor is shown coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100, opposite to the touchscreen display 112 located on the front of device 100.

[0178] The device 100 optionally also includes one or more proximity sensors 166. Figure 1AA proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 may optionally be coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 may optionally perform as described in the following U.S. patent applications: 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, the proximity sensor is turned off and the touchscreen 112 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0179] The device 100 may optionally also include one or more tactile output generators 167. Figure 1AA haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.

[0180] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may optionally be coupled to an input controller 160 in an I / O subsystem 106. The accelerometer 168 may optionally perform as described in the following U.S. patent publications: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. In addition to the accelerometer 168, the device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the position and orientation (e.g., longitudinal or lateral) of the device 100.

[0181] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 ( Figure 1A ) or 370 ( Figure 3 Storage device / global internal state 157, such as Figure 1A and Figure 3 As shown in the diagram. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views or other information occupy various areas of the touchscreen display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and positional information about the device's position and / or orientation.

[0182] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0183] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is for communication with… (Trademark of Apple Inc.) The same or similar and / or compatible multi-pin (e.g., 30-pin) connectors used in Apple Inc. devices.

[0184] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disengagement). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a series of contact data. These operations are optionally applied to single-point contact (e.g., single-finger contact) or simultaneous multi-point contact (e.g., "multi-touch" / multiple-finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0185] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen display can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0186] The touch / motion module 130 optionally detects user gesture input. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0187] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, and non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0188] In some implementations, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like to specify the graphic to be displayed, and, if necessary, also receives coordinate data and other graphic attribute data, and then generates screen image data for output to the display controller 156.

[0189] The haptic feedback module 133 includes various software components for generating instructions that are used by the haptic output generator 167 to produce haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0190] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0191] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to phone 138 for use in location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services, such as weather desktop apps, local yellow pages desktop apps, and map / navigation desktop apps).

[0192] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0193] • Contacts module 137 (sometimes called address book or contact list);

[0194] • Telephone module 138;

[0195] • Video conferencing module 139;

[0196] • Email client module 140;

[0197] • Instant Messaging (IM) module 141;

[0198] Fitness support module 142;

[0199] • Camera module 143 for still images and / or video images;

[0200] • Image management module 144;

[0201] • Video player module;

[0202] Music player module;

[0203] • Browser module 147;

[0204] • Calendar module 148;

[0205] • Desktop mini-program module 149, which optionally includes one or more of the following: weather desktop mini-program 149-1, stock desktop mini-program 149-2, calculator desktop mini-program 149-3, alarm clock desktop mini-program 149-4, dictionary desktop mini-program 149-5 and other desktop mini-programs obtained by the user, and user-created desktop mini-program 149-6;

[0206] • Desktop app creator module 150 for creating user-created desktop apps 149-6;

[0207] • Search module 151;

[0208] • Video and music player module 152, which combines the video player module and the music player module;

[0209] • Notepad module 153;

[0210] • Map module 154; and / or

[0211] • Online video module 155.

[0212] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0213] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., in the application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communication via telephone 138, video conferencing module 139, email 140, or instant messaging 141; and so on.

[0214] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 137, modify entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0215] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0216] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0217] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving an instant message, and viewing a received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0218] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes executable instructions for creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (exercise equipment); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing music for fitness; and displaying, storing, and transmitting fitness data.

[0219] In conjunction with the touchscreen 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0220] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or photo album), and storing still images and / or video images.

[0221] Combining RF circuitry 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) according to user instructions.

[0222] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0223] In conjunction with RF circuitry 108, touchscreen 112, display system controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, desktop applet module 149 is optionally a micro-application downloaded and used by a user (e.g., weather desktop applet 149-1, stock market desktop applet 149-2, calculator desktop applet 149-3, alarm clock desktop applet 149-4, and dictionary desktop applet 149-5) or a user-created micro-application (e.g., user-created desktop applet 149-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop applet includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop applet).

[0224] Combining RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop applet creator module 150 can optionally be used by the user to create desktop applets (e.g., to transfer user-specified portions of a webpage into a desktop applet).

[0225] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions to search the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0226] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

[0227] Combining the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0228] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data related to shops and other points of interest at or near a specific location, and other location-based data) according to user instructions.

[0229] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions for performing the following operations: allowing users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Additional descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.

[0230] Each of the modules and applications described above corresponds to a set of executable instructions for performing one or more of the functions described above, as well as the methods described in this patent application (e.g., computer-implemented methods and other information processing methods as described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. For example, a video player module may optionally be combined with a music player module into a single module (e.g., Figure 1A (e.g., video and music player module 152). In some embodiments, memory 102 optionally stores a subset of the above-described modules and data structures. Additionally, memory 102 optionally stores other modules and data structures not described above.

[0231] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on device 100 can be optionally reduced.

[0232] The set of predefined functions, performed exclusively via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main menu, home menu, or root menu. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0233] Figure 1B This is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0234] Event classifier 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0235] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that the application 136-1 is displaying or ready to be displayed by the application, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a repeat / undo queue for previous actions taken by the user.

[0236] Event monitor 171 receives event information from peripheral interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.

[0237] In some implementations, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other implementations, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0238] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0239] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0240] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0241] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure that handles the sub-events. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source to which it is identified as the hit view.

[0242] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.

[0243] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0244] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In other embodiments, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.

[0245] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, wherein each application view includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more event recognizers among the event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0246] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies events from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0247] Event receiver 182 receives event information from event classifier 170. This event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from a longitudinal orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).

[0248] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 112, and lifting off the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0249] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0250] In some implementations, the definition of the corresponding event 187 also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0251] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0252] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, tags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, tags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0253] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and delaying) the sub-events to the corresponding hit view. In some implementations, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag receives the flag and executes a predefined procedure.

[0254] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event series or to an actively participating view. The event handler associated with the sub-event series or the actively participating view receives the event information and performs a predetermined procedure.

[0255] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0256] In some implementations, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.

[0257] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0258] Figure 2A portable multifunction device 100 with a touchscreen 112 is shown according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figures) or using one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of one or more graphics occurs when the user breaks contact with the graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0259] Device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.

[0260] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts verbal input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic outputs for the user of device 100.

[0261] Figure 3This is a block diagram of an exemplary multi-functional device with a display and a touch-sensitive surface according to some embodiments. Device 300 is not necessarily portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the above reference). Figure 1A The tactile output generator 167 and sensor 359 (e.g., optical sensor, accelerometer, proximity sensor, touch sensor and / or contact intensity sensor (similar to the above reference)) are described. Figure 1A The contact strength sensor 165 is described above. The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100. Figure 1A The memory 370 stores programs, modules, and data structures similar to or subsets thereof in the memory 102 of the portable multifunction device 100. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 ( Figure 1A The memory 102 may optionally not store these modules.

[0262] Figure 3Each of the elements described above is optionally stored in one or more memory devices of the previously mentioned memory devices. Each of the modules identified above corresponds to an instruction set for performing the functions described above. The modules or programs (e.g., instruction sets) described above need not be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subgroup of the modules and data structures described above. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0263] Now let’s turn our attention to the implementation of the user interface, which is optionally implemented on, for example, a portable multifunction device 100.

[0264] Figure 4A An exemplary user interface for a menu of an application on a portable multifunction device 100, according to some embodiments, is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof:

[0265] • One or more signal strength indicators 402 for one or more wireless communications (e.g., cellular signals and Wi-Fi signals);

[0266] • Time 404;

[0267] Bluetooth indicator 405;

[0268] • Battery status indicator 406;

[0269] • Tray icon 408 with commonly used application icons, such as:

[0270] ○ The telephone module 138 has an icon 416 labeled "telephone", which optionally includes an indicator 414 indicating the number of missed calls or voicemails;

[0271] ○ An icon 418 labeled "Mail" in the email client module 140, which optionally includes an indicator 410 for the number of unread emails;

[0272] ○ The icon 420 labeled "Browser" in browser module 147; and

[0273] ○ The icon 422 marked "iPod" for the video and music player module 152 (also known as the iPod (a trademark of Apple Inc.) module 152); and

[0274] • Icons of other applications, such as:

[0275] ○Icon 424 labeled "Message" in IM module 141;

[0276] ○The calendar module 148 has an icon 426 labeled "Calendar";

[0277] ○ The icon 428 labeled "Photo" in the image management module 144;

[0278] ○ The icon 430 of camera module 143 is labeled "camera";

[0279] ○ The icon 432 labeled "Online Video" in the online video module 155;

[0280] ○ The icon 434 labeled "Stock Market" in the Stock Market Desktop Mini Program 149-2;

[0281] ○ Map module 154 has icon 436 labeled "Map";

[0282] ○The icon labeled "Weather" in the Weather desktop mini-program 149-1 is 438;

[0283] ○ The icon labeled "clock" in the alarm clock desktop mini-program 149-4;

[0284] ○ The icon 442 labeled "Fitness Support" in the Fitness Support module 142;

[0285] ○ The icon 444 labeled "Notepad" in Notepad module 153; and

[0286] ○ Set an icon 446 labeled "Settings" for an application or module, which provides access to settings for the device 100 and its various applications 136.

[0287] It should be noted that Figure 4A The icon labels shown are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0288] Figure 4B A touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)) is shown. Figure 3 Devices (e.g., tablets or touchpads 355) Figure 3An exemplary user interface on the device 300. The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 359) for detecting the intensity of contact on the tactile surface 451 and / or one or more tactile output generators 357 for generating tactile outputs for the user of the device 300.

[0289] While some examples of input on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given below, in some implementations the device detects input on a touch-sensitive surface separate from the display, such as... Figure 4B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a spindle (e.g., on the display (e.g., 450)). Figure 4B The spindle corresponding to 453 in the middle (e.g., Figure 4B (452 in the example). According to these embodiments, the device detects the position corresponding to the corresponding position on the display (e.g., in the example). Figure 4B In the middle, 460 corresponds to 468 and 462 corresponds to 470) the contact with the touch-sensitive surface 451 (e.g., Figure 4B (460 and 462 in the text). Thus, on touch-sensitive surfaces (e.g., Figure 4B 451) and the display of the multi-functional device ( Figure 4B When 450 is separated from the touch-sensitive surface, user input detected by the device on the touch-sensitive surface (e.g., touches on 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may be optionally used for other user interfaces described herein.

[0290] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger touch, single-finger tap, finger swipe), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of the touch and subsequent cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0291] Figure 5AAn exemplary personal electronic device 500 is illustrated. Device 500 includes a body 502. In some embodiments, device 500 may include components relative to devices 100 and 300 (e.g., Figures 1A-4B Some or all of the features described herein. In some embodiments, device 500 has a touch-sensitive display 504, referred to below as touchscreen 504. Alternatively or as a supplement to touchscreen 504, device 500 also has a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., touch). The one or more intensity sensors of touchscreen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on their intensity, meaning that touches of different intensities can invoke different user interface operations on device 500.

[0292] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, filed May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” published under WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” published under WIPO Patent Publication No. WO / 2014 / 105276, each of which is incorporated herein by reference in its entirety.

[0293] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) may be physical. Examples of physical input mechanisms include push-buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watch straps, bangles, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow device 500 to be worn by a user.

[0294] Figure 5B An exemplary personal electronic device 500 is illustrated. In some embodiments, device 500 may include, relative to... Figure 1A , Figure 1B and Figure 3 Some or all of the components described herein. Device 500 has a bus 512 operatively coupling I / O portion 514 to one or more computer processors 516 and memory 518. I / O portion 514 may be connected to display 504, which may have touch-sensitive component 522 and optionally have intensity sensor 524 (e.g., contact intensity sensor). Furthermore, I / O portion 514 may be connected to communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 may optionally be a rotatable input device or a pressable input device and a rotatable input device. In some examples, input mechanism 508 may optionally be a button.

[0295] In some examples, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to the I / O section 514.

[0296] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, may cause the computer processors to perform, for example, the following techniques: process 700 (Figure 7), process 900 (Figure 9), process 1100 (Figure 11), process 1300 (Figure 13), process 1500 (Figure 15), process 1700 (Figure 17), process 1800 (Figure 18), process 2000 (Figure 20), process 2200 (Figure 22), process 2400 (Figure 24), process 2600 (Figure 26), process 2800 (Figure 28), process 3000 (Figure 30), process 3200 (Figure 32), process 3400 (Figure 34), process 3600 (Figure 36), and process 3800 (Figure 38). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but is not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. Personal electronic devices are not limited to... Figure 5B It is not the components and configurations of a single entity, but rather other components or additional components that may include a variety of configurations.

[0297] As used herein, the term "power indication" refers optionally to the power indication in devices 100, 300, and / or 500 ( Figure 1A , Figure 3 and Figures 5A-5B A user-interactive graphical user interface object displayed on a screen. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) optionally each constitute a functional representation.

[0298] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other positional marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3 The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of enabling direct interaction with user interface elements on the touchscreen display) Figure 1A The touch-sensitive display system 112 or Figure 4A In some embodiments of the touchscreen 112, a detected touch on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by a touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that delivers the user's expected interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0299] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples sampled over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact lift is detected, before or after contact begins to move is detected, before contact ends is detected, before or after contact intensity increases and / or before or after contact intensity decreases). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some implementations, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the feature intensity and one or more thresholds is used to determine whether to perform one or more actions (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.

[0300] Figure 5C This illustrates the use of multiple intensity sensors 524A-524D to detect multiple contacts 552A-552E on a touch-sensitive display 504. Figure 5C It also includes an intensity graph, which shows the current intensity measurements of intensity sensors 524A-524D relative to intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are both 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are both 7 intensity units. In some embodiments, the cumulative intensity is the sum of the intensity measurements of the multiple intensity sensors 524A-524D, which is 32 intensity units in this example. In some implementations, each contact is assigned a corresponding intensity, i.e., a portion of the cumulative intensity. Figure 5DThe diagram illustrates the allocation of cumulative intensity to contacts 552A-552E based on their distance from the center of force 554. In this example, each of contacts 552A, 552B, and 552E is assigned a contact intensity of 8 intensity units of the cumulative intensity, and each of contacts 552C and 552D is assigned a contact intensity of 4 intensity units of the cumulative intensity. More generally, in some specific implementations, each contact j is assigned a corresponding intensity Ij according to a predefined mathematical function Ij = A·(Dj / ΣDi), which is a portion of the cumulative intensity A, where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i = 1 to the last) from the center of force. Reference operations can be performed using electronic devices similar to or equivalent to devices 100, 300, or 500. Figures 5C-5D The operation described above. In some embodiments, the characteristic intensity of the contact is based on one or more intensities of the contact. In some embodiments, an intensity sensor is used to determine the intensity of a single characteristic (e.g., a single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included within... Figures 5C-5D This is to help readers.

[0301] In some implementations, a portion of the gesture is identified to determine the characteristic intensity. For example, the touch-sensitive surface optionally receives a series of swipe contacts that transition from a starting position to an ending position, where the contact intensity increases. In this example, the characteristic intensity of the contact at the ending position is optionally based only on a portion of the series of swipe contacts, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the ending position). In some implementations, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact for determining the characteristic intensity.

[0302] Optionally, the contact intensity on a touch-sensitive surface is characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device performs an operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device performs an operation different from the operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity lower than the light press intensity threshold (e.g., and higher than the nominal contact detection intensity threshold, where contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector based on the movement of the contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.

[0303] The intensity of a contact increasing from below a light press intensity threshold to between a light press intensity threshold and a deep press intensity threshold is sometimes referred to as a "light press" input. The intensity of a contact increasing from below a deep press intensity threshold to above a deep press intensity threshold is sometimes referred to as a "deep press" input. The intensity of a contact increasing from below a contact detection intensity threshold to between a contact detection intensity threshold and a light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. The intensity of a contact decreasing from above a contact detection intensity threshold to below a contact detection intensity threshold is sometimes referred to as detecting a contact being lifted off the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0304] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., a "downward stroke" of the corresponding press input). In some embodiments, the press input includes a corresponding contact intensity increasing to above a press input intensity threshold and the contact intensity subsequently decreasing to below the press input intensity threshold, and the corresponding operation is performed in response to detecting that the corresponding contact intensity subsequently decreases to below the press input threshold (e.g., an "upward stroke" of the corresponding press input).

[0305] Figures 5E-5HThe detection of gestures is shown, including gestures involving contact 562 with an intensity ranging from less than... Figure 5E The light press intensity threshold (e.g., "IT") L The intensity of ”) increases to higher than Figure 5H The deep press intensity threshold (e.g., "IT") D The intensity of the press input corresponds to the strength of the press. On the user interface 570, which includes application icons 572A-572D displayed in the predefined area 574, while the cursor 576 is displayed above the application icon 572B corresponding to application 2, the gesture performed by contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch-sensitive display 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines that the intensity of the contact 562 is within the deep press intensity threshold (e.g., "IT"). D The intensity reaches its peak value above 562. Contact is maintained on the touch-sensitive surface 560. In response to a detected gesture, and based on the intensity increasing to a deep press intensity threshold (e.g., "IT") during the gesture... D The above contact 562 displays a scaled representation 578A-578C (e.g., thumbnail) of the document recently opened for application 2, such as... Figures 5F-5H As shown. In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in... Figures 5E-5H This is to help readers.

[0306] In some implementations, the display of icons 578A-578C includes animation. For example, icon 578A might initially appear near application icon 572B, such as... Figure 5F As shown, as the animation progresses, icon 578A moves upwards, and icon 578B appears near application icon 572B, as shown. Figure 5G As shown in the diagram. Then, representation 578A moves upwards, 578B moves upwards towards representation 578A, and representation 578C is displayed near application icon 572B, as shown. Figure 5H As shown in the diagram. This indicates that 578A-578C form an array above icon 572B. In some implementations, the animation progresses according to the intensity of contact 562, such as... Figures 5F-5G As shown, where 578A-578C appear and increase with the intensity of contact 562 towards the pressure threshold (e.g., "IT"). D The animation progresses upwards as it increases in intensity. In some implementations, the intensity upon which the animation progresses is based is the characteristic intensity of the contact. Reference can be performed using electronic devices similar to or equivalent to devices 100, 300, or 500. Figures 5E-5H The aforementioned operations.

[0307] In some embodiments, the device employs intensity hysteresis to avoid unintended inputs sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units smaller than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes a corresponding contact intensity increasing above the press input intensity threshold and the contact intensity subsequently decreasing below the hysteresis intensity threshold corresponding to the press input intensity threshold, and corresponding operation is performed in response to detecting that the corresponding contact intensity subsequently decreases below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and optionally the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity threshold, and corresponding operation is performed in response to detecting a press input (e.g., depending on the environment, an increase or decrease in contact intensity).

[0308] For ease of explanation, optionally, the description of an operation triggered in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is provided in response to detecting any of the following conditions: the contact strength increases to above the press input strength threshold, the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold, the contact strength decreases to below the press input strength threshold, and / or the contact strength decreases to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example where the operation is described as being performed in response to detecting a decrease in contact strength below the press input strength threshold, the operation is optionally performed in response to detecting a decrease in contact strength below a hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0309] Figure 5I An exemplary electronic device 580 is illustrated. Device 580 includes a body 580A. In some embodiments, device 580 may include components relative to devices 100, 300, and 500 (e.g., Figures 1A-5BThe device 580 may include some or all of the features described herein. In some embodiments, the device 580 has one or more speakers 580B (concealed within a body 580A), one or more microphones 580C, one or more touch-sensitive surfaces 580D, and one or more displays 580E. Alternatively, or as a supplement to the displays and touch-sensitive surfaces 580D, the device may also have a touch-sensitive display (also referred to as a touchscreen). Similar to devices 100, 300, and 500, in some embodiments, the touch-sensitive surface 580D (or touchscreen) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., a touch). The one or more intensity sensors of the touch-sensitive surface 580D (or touchscreen) may provide output data representing the intensity of the touch. The user interface of the device 580 may respond to a touch based on its intensity, meaning that different intensities of touch may invoke different user interface operations on the device 580. In some embodiments, the one or more displays 580E are one or more light-emitting diodes (LEDs). For example, the display may be a single LED, an LED cluster (e.g., red, green, and blue LEDs), multiple discrete LEDs, multiple discrete LED clusters, or other arrangements of one or more LEDs. For example, the display 580E may be an array of nine discrete LED clusters arranged in a circle (e.g., a ring). In some examples, the one or more displays may contain one or more light-emitting elements of another type.

[0310] Figure 5J An exemplary personal electronic device 580 is depicted. In some embodiments, device 580 may include, relative to... Figure 1A , Figure 1B , Figure 3 and Figures 5A-5B Some or all of the components described herein. Device 580 has a bus 592 operatively coupling I / O portion 594 to one or more computer processors 596 and memory 598. I / O portion 594 may be connected to display 582, which may have touch-sensitive component 584 and optionally have intensity sensor 585 (e.g., contact intensity sensor). In some embodiments, touch-sensitive component 584 is a component separate from display 582. Furthermore, I / O portion 594 may be connected to communication unit 590 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 580 may include input mechanism 588. In some examples, input mechanism 588 is optionally a button. In some examples, input mechanism 588 is optionally a microphone. Input mechanism 588 is optionally a plurality of microphones (e.g., microphone array).

[0311] Electronic device 580 includes a speaker 586 for outputting audio. Device 580 may include audio circuitry (e.g., in I / O section 594) that receives audio data, converts the audio data into an electrical signal, and transmits the electrical signal to speaker 586. Speaker 586 converts the electrical signal into sound waves that are audible to humans. The audio circuitry (e.g., in I / O section 594) also receives electrical signals converted from sound waves by a microphone (e.g., input mechanism 588). The audio circuitry (e.g., in I / O section 594) converts the electrical signals into audio data. The audio data may optionally be retrieved by I / O section 594 from and / or transmitted to memory 598 and / or RF circuitry (e.g., in communication unit 590).

[0312] The memory 598 of the personal electronic device 580 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 596, may cause the computer processors to perform, for example, the following techniques: process 700 (Figure 7), process 900 (Figure 9), process 1100 (Figure 11), process 1300 (Figure 13), process 1500 (Figure 15), process 1700 (Figure 17), process 1800 (Figure 18), process 2000 (Figure 20), process 2200 (Figure 22), process 2400 (Figure 24), process 2600 (Figure 26), process 2800 (Figure 28), process 3000 (Figure 30), process 3200 (Figure 32), process 3400 (Figure 34), process 3600 (Figure 36), and process 3800 (Figure 38). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but is not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. Personal electronic devices 580 are not limited to Figure 5J It is not the components and configurations of a single entity, but rather other components or additional components that may include a variety of configurations.

[0313] As used herein, "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched on the device (e.g., become open). In some implementations, the downloaded application becomes an installed application using an installer, which extracts program portions from the downloaded software package and integrates the extracted portions with the computer system's operating system.

[0314] As used herein, the terms "open application" or "running application" refer to a software application that maintains state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or running application is optionally any of the following types of applications:

[0315] • The active application, which is currently displayed on the screen of the device that is using the application;

[0316] • Background applications (or background processes) that are not currently displayed but whose one or more processes are being handled by one or more processors; and

[0317] • A paused or hibernating application that is not running but has state information stored in memory (both volatile and non-volatile) that can be used to resume the execution of the application.

[0318] As used herein, the term "closed application" refers to a software application that does not retain state information (e.g., the state information of a closed application is not stored in the device's memory). Therefore, closing an application includes the application process of stopping and / or removing the application, as well as removing the application's state information from the device's memory. Generally, opening a second application while the first application is in the first application does not close the first application. When the second application is displayed and the first application stops being displayed, the first application becomes a background application.

[0319] This document refers throughout to the functions, operations, and characteristics associated with digital assistants (also referred to as “voice assistants” or “personal digital assistants”). A digital assistant can refer to any information processing system that interprets natural language input in the form of speech and / or text to infer user intent and performs actions based on the inferred user intent. For example, to follow an inferred user intent, a system may perform one or more of the following: identify a task flow with steps and parameters designed to achieve the inferred user intent; input specific requirements from the inferred user intent into the task flow; execute the task flow by invoking programs, methods, services, APIs, etc.; and generate output responses to the user in an audible (e.g., voice) and / or visual form.

[0320] Specifically, a digital assistant may be able to accept user requests, at least in part, in the form of natural language commands, requests, statements, narration, and / or inquiries. Typically, user requests may seek an informative response from the digital assistant or request the digital assistant to perform a task. A satisfactory response to a user request may be providing the requested informative response, performing the requested task, or a combination of both. For example, a user may ask a digital assistant a question such as, “Where am I now?” Based on the user’s current location, the digital assistant may answer, “You are near the west entrance of Central Park.” A user may also request to perform a task, such as, “Please invite my friends to my girlfriend’s birthday party next week.” In response, the digital assistant may confirm the request by saying “Okay, coming right away,” and then send the appropriate calendar invitations on behalf of the user to each of the user’s friends listed in the user’s electronic address book. During the performance of the requested task, the digital assistant may sometimes interact with the user in a continuous dialogue involving multiple exchanges of information over a long period of time. Many other methods exist for interacting with a digital assistant to request information or perform various tasks. In addition to providing verbal responses and taking programmed actions, digital assistants can also provide responses in other visual or audio forms, such as text, alerts, music, video, animation, etc.

[0321] In some examples, a digital assistant may be implemented according to a client-server model (e.g., the exemplary "digital assistant system"). A digital assistant may include a client-side portion (hereinafter referred to as a "DA client") executing on a user device (e.g., device 100, 300, 500, or 580) and a server-side portion (hereinafter referred to as a "DA server") executing on a server system (e.g., remotely from the user device). The DA client may communicate with the DA server over one or more networks. The DA client may provide client-side functionality, such as user-oriented input and output processing, and communication with the DA server. The DA server may provide server-side functionality for any number of DA clients, each located on a respective user device.

[0322] In some examples, a DA server may include a client-facing I / O interface, one or more processing modules, data and models, and I / O interfaces to external services. The client-facing I / O interface facilitates the DA server in performing client-facing input and output processing. The one or more processing modules may utilize the data and models to process voice input and determine the user's intent based on the natural language input. Furthermore, the one or more processing modules perform task execution based on the inferred user intent. In some examples, the DA server may communicate with external services via one or more networks to complete tasks or collect information. The I / O interface to external services facilitates such communication.

[0323] User equipment can be any suitable electronic device. For example, user equipment can be a portable multi-functional device (e.g., see reference above). Figure 1A The described device 100), multi-functional device (e.g., referenced above) Figure 3 The device described is 300), a personal electronic device (e.g., referred to below). Figures 5A-5B The described device 500) or electronic device (e.g., the one mentioned above) Figures 5I-5J The device described is 580. A portable multi-functional device can be, for example, a mobile phone that also includes other functions such as PDA and / or music player functionality. Specific examples of portable multi-functional devices may include those from Apple Inc. (Cupertino, California). Devices, iPod Equipment and Devices. Other examples of portable multifunction devices may include, but are not limited to, laptop computers or tablets. Additionally, in some examples, the user device may be a non-portable multifunction device. Specifically, the user device may be a desktop computer, game console, television, smart speaker, or set-top box. In some examples, the user device may include a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). Furthermore, the user device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick. Various examples of electronic devices such as multifunction devices are described in more detail below.

[0324] Examples of one or more communication networks may include local area networks (LANs) and wide area networks (WANs), such as the Internet. One or more communication networks may be implemented using any known network protocol, including various wired or wireless protocols such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0325] Server systems can be implemented on one or more standalone data processing devices or distributed networks. In some examples, server systems may also utilize various virtual appliances and / or services from third-party service providers (e.g., third-party cloud service providers) to provide the server system's potential computing and / or infrastructure resources.

[0326] In some examples, the user equipment (also referred to as the first user equipment) may communicate with the DA server via a second user equipment. The second user equipment may be similar to or the same as the user equipment described above. For example, the second user equipment may be similar to the referenced above. Figure 1A , Figure 3 , Figures 5A-5B and Figures 5I-5J The described devices are 100, 300, 500, or 580. A first user equipment (UAE) may be communicatively coupled to a second UAE via a direct communication connection such as Bluetooth, NFC, BTLE, etc., or via a wired or wireless network such as a local area network (LAN) Wi-Fi. In some examples, the second UAE may be configured to act as a proxy between the first UAE and a DA server. For example, a DA client of the first UAE may be configured to transmit information (e.g., a user request received at the first UAE) to the DA server via the second UAE. The DA server may process this information and return relevant data (e.g., data content in response to a user request) to the first UAE via the second UAE.

[0327] In some examples, the digital assistant system may be implemented on a standalone computer system. In some examples, the digital assistant system may be distributed across multiple computers. In some examples, as described above, some modules and functions of the digital assistant may be divided into server and client parts, wherein the client part resides on one or more user devices (e.g., devices 100, 300, 500, 580) and communicates with the server part through one or more networks. In some examples, the digital assistant system may be a specific implementation of a server system (and / or a DA server).

[0328] In some examples, the digital assistant system includes memory that may also store digital assistant modules (or the server portion of the digital assistant). In some examples, a digital assistant module may include, or a subset or superset of, the following submodules: an input / output processing module, a speech-to-text (STT) processing module, a natural language processing module, a dialogue flow processing module, a task flow processing module, a service processing module, and a speech synthesis module. Each of these modules may have access to, or a subset or superset of, one or more of the following system or data and models of the digital assistant modules: knowledge ontology, vocabulary index, user data, task flow model, service model, and ASR system.

[0329] In some examples, using processing modules, data, and models implemented in the digital assistant module, the digital assistant can perform at least some of the following: converting voice input into text; recognizing the user's intent expressed in natural language input received from the user; proactively eliciting and obtaining the information needed to fully infer the user's intent (e.g., by de-emphasizing words, games, intent ambiguity, etc.); determining a task flow to satisfy the inferred intent; and executing the task flow to satisfy the inferred intent.

[0330] In some examples, the I / O processing module interacts with the user via an I / O device or with a user device (e.g., device 100, 300, 500, or 580) via a network communication interface to receive user input (e.g., voice input) and provide a response to the user input (e.g., as voice output). The I / O processing module may optionally acquire contextual information associated with the user input from the user device, either along with or shortly after receiving the user input. Contextual information may include user-specific data, vocabulary, and / or preferences associated with the user input. In some examples, this contextual information may also include the software and hardware state of the user device at the time the user request is received, and / or information related to the user's surrounding environment at the time the user request is received. In some examples, the I / O processing module may also send follow-up questions about the user request to the user and receive responses from the user. When a user request is received by the I / O processing module and the user request may include voice input, the I / O processing module may forward the voice input to the STT processing module (or speech recognizer) for speech-to-text conversion.

[0331] The STT processing module may include one or more ASR systems. These ASR systems process speech input received through the I / O processing module to produce recognition results. Each ASR system may include a front-end speech preprocessor. The front-end speech preprocessor extracts representative features from the speech input. For example, the front-end speech preprocessor may perform a Fourier transform on the speech input to extract spectral features characterizing the speech input as a sequence of representative multidimensional vectors. Furthermore, each ASR system may include one or more speech recognition models (e.g., sound models and / or language models) and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines may include engines based on dynamic time warping and engines based on weighted finite-state transducers (WFST). One or more speech recognition models and one or more speech recognition engines can be used to process the extracted representational features from the front-end speech preprocessor to produce intermediate recognition results (e.g., phonemes, phoneme strings, and sub-words) and ultimately produce text recognition results (e.g., words, word strings, or sequences of symbols). In some examples, the speech input may be processed at least partially by a third-party service or on the user's device (e.g., device 100, 300, 500, or 580) to produce recognition results. Once the STT processing module produces a recognition result containing a text string (e.g., words, or sequences of words, or sequences of symbols), the recognition result can be passed to the natural language processing module for intent inference.

[0332] In some examples, the digital assistant's natural language processing module ("natural language processor") receives a sequence of words or symbols ("symbol sequence") generated by the STT processing module and attempts to associate that symbol sequence with one or more "executable intents" recognized by the digital assistant. An "executable intent" can represent a task that can be performed by the digital assistant and has an associated task flow implemented in the task flow model of the digital assistant system. An associated task flow can be a series of programmed actions and steps taken by the digital assistant to perform the task. The capabilities of the digital assistant can depend on the number and types of task flows implemented and stored in the task flow model, or in other words, on the number and types of "executable intents" recognized by the digital assistant. However, the effectiveness of the digital assistant can also depend on its ability to infer the correct "one or more executable intents" from a user request expressed in natural language.

[0333] In some examples, natural language processing can be based on, for example, a knowledge ontology. A knowledge ontology can be a hierarchical structure containing many nodes, each node representing an "executable intent" or an "attribute" associated with one or more of the "executable intent" or other "attributes." As mentioned above, an "executable intent" can represent a task that a digital assistant can perform, i.e., the task is "executable" or can be done. An "attribute" can represent a parameter associated with a sub-aspect of an executable intent or another attribute. The links between executable intent nodes and attribute nodes in a knowledge ontology can define how the parameters represented by the attribute nodes are subordinate to the task represented by the executable intent nodes.

[0334] In some examples, once the NLP module identifies an executable intent (or domain) based on a user request, it can generate a structured query representing the identified executable intent. In some examples, the structured query may include parameters for one or more nodes within the domain of the executable intent, and at least some of these parameters are populated with specific information and requirements specified in the user request. For example, a user might say, “Reserve a table at a sushi restaurant for 7 pm.” In this case, the NLP module might be able to correctly identify the executable intent as “restaurant reservation” based on the user input. According to the knowledge ontology, a structured query for the “restaurant reservation” domain could include parameters such as {cuisine}, {time}, {date}, {number of people in the party}, etc. In some examples, based on voice input and text derived from the voice input using the STT processing module, the NLP module can generate a partially structured query for the restaurant reservation domain, where the partially structured query includes the parameters {cuisine = “sushi”} and {time = “7 pm”}. However, in this example, the user's utterance contains insufficient information to complete a structured query associated with the domain. Therefore, based on the currently available information, other necessary parameters, such as {number of people in the same group} and {date}, may not be specified in the structured query. In some examples, the natural language processing module can use the received context information to populate some parameters of the structured query. For example, in some examples, if a user requests a "nearby" sushi restaurant, the natural language processing module can use GPS coordinates from the user's device to populate the {location} parameter in the structured query.

[0335] In some examples, the natural language processing module may pass the generated structured query (including any completed parameters) to the task flow processing module (“task flow processor”). The task flow processing module may be configured to receive the structured query from the natural language processing module, complete the structured query (if necessary), and perform the actions required to “complete” the user’s final request. In some examples, the various processes necessary to complete these tasks may be provided in the task flow model. In some examples, the task flow model may include processes for obtaining additional information from the user, and task flows for performing actions associated with the executable intent.

[0336] As described above, to complete a structured query, the task flow processing module may need to initiate additional dialogue with the user to obtain additional information and / or clarify potentially ambiguous statements. When such interaction is necessary, the task flow processing module may invoke the dialogue flow processing module to participate in the dialogue with the user. In some examples, the dialogue flow processing module may determine how (and / or when) to request additional information from the user and receive and process user responses. The I / O processing module may present questions to the user and receive answers from the user. In some examples, the dialogue flow processing module may present dialogue output to the user via audio and / or visual output and receive input from the user via verbal or physical (e.g., click) responses. Continuing with the example above, when the task flow processing module invokes the dialogue flow processing module to determine the "number of people in a group" and "date" information for a structured query associated with the domain "restaurant reservation," the dialogue flow processing module may generate questions such as "How many people in a group?" and "Which day to book?" to deliver to the user. Once an answer is received from the user, the dialogue flow processing module may either populate the structured query with missing information or pass information to the task flow processing module to complete the missing information based on the structured query.

[0337] Once the task flow processing module has completed a structured query for the executable intent, it can proceed to execute the final task associated with the executable intent. Therefore, the task flow processing module can execute steps and instructions in the task flow model based on specific parameters contained in the structured query. For example, a task flow model for the executable intent "restaurant reservation" could include steps and instructions for contacting a restaurant and actually requesting a reservation for a specific number of people at a specific time. For example, using a structured query such as: {restaurant reservation, restaurant = ABC Cafe, date = 3 / 12 / 2012, time = 7 pm, number of people = 5}, the task flow processing module could perform the following steps: (1) Log in to ABC Cafe's server or restaurant reservation system such as (2) Enter the date, time and number of people in the party on the website, (3) Submit the form and (4) Create a calendar entry for the reservation on the user's calendar.

[0338] As discussed throughout this document, various techniques are discussed for receiving user input at a first device and inducing an action or output at a second device. For example, a first user device (e.g., 580 or 3300) receives user input (e.g., "Play Ramones songs in my living room"). In some examples, the user input includes user speech, and the first user device uses a microphone of a first electronic device to receive the user input. In some examples, the first electronic device converts the user input into a representation of audio input. The representation of the audio input can be analog or digital. In some examples, the representation is a text representation, and the first electronic device uses speech-to-text conversion to convert the user input into a text representation. Any known speech-to-text conversion process can be used to convert the user speech.

[0339] In some implementations, user input or its representation is provided (e.g., transmitted) by a first electronic device to a virtual assistant server (e.g., a DA server) for processing.

[0340] Based on user input, a virtual assistant server identifies one or more tasks and parameters. For example, the virtual assistant server interprets a text representation of user input to derive an intent and operationalizes that intent into one or more tasks. In the depicted example, based on the user input “Play Ramones songs in my living room,” the virtual assistant server identifies the task of adding one or more media items to the media playback queue at a desired electronic device (e.g., 580, 3304). The task can be identified in any manner. Techniques for identifying tasks can be found, for example, in U.S. Utility Model Application No. 14 / 503,105, filed September 30, 2014, entitled “Intelligent Assistant for Home Automation,” and U.S. Provisional Patent Application Serial No. 62 / 348,015 (Attorney’s File No. 770003000100 (P30331USP1)), filed June 9, 2016, entitled “Intelligent Automated Assistant in a Home Environment,” the entire disclosure of which is incorporated herein by reference.

[0341] In addition, the virtual assistant server identifies the relevant parameters required to complete the task. In some examples, parameters include an indication of the media item to be played, such as the song title and media file name. In some examples, parameters include an indication of the type of media item to be played, such as audio, video, and text. In some examples, parameters include an indication of the expected location of the electronic device, such as a named location and configuration associated with the user (“living room,” “garage,” “upstairs”) or GPS coordinates. In some examples, parameters include an indication of the expected type of electronic device, such as a TV, speaker dock, and smartwatch. In this example, the virtual assistant server recognizes the parameters as “Ramones” and “living room.”

[0342] The virtual assistant server may transmit a request for identifying one or more user devices to a device identification storage device (e.g., on the virtual assistant server). This request includes one or more parameters for identifying the one or more user devices. In some examples, the one or more parameters may specify a name location (e.g., "living room"), a device type (e.g., "Apple TV"), and / or one or more attributes of a media item, such as the type of the media item (e.g., "audio" and "video").

[0343] Device identification storage (e.g., on a virtual assistant server) can store data and / or models for a wide variety of attributes, as well as relevant information associated with various electronic devices. Attributes that can be stored for electronic devices include, but are not limited to: unique identifier, status, type, location, supported media types, and ownership. For example, device identification storage can store information associating a device with the type "Apple TV," the named location "living room," and the supported media types "audio" and "video." As another example, device identification storage can store information associating a device with the type "speaker," the named location "garage," and the supported media type "audio."

[0344] When a user's input specifies a task to be performed (such as, "Hey Siri, find me a TV episode" or "Hey Siri, drive me there"), the device type is particularly relevant to smart device arbitration. In some implementations, if, based on the broadcast value, the electronic device is the only device capable of handling the type of task specified in the audio input, then the electronic device will respond to the audio input.

[0345] In some examples, the electronic device acquires a list of acceptable device types capable of handling a specified task (e.g., by locally analyzing audio input and / or by receiving a list from one or more servers), and determines whether the device will respond to the audio input based on the broadcast "device type" value and the list of acceptable device types. In some examples, one or more servers receive data representing sampled audio input from one or more electronic devices within the electronic device, derive a user intent based on that data, and identify a task with one or more parameters based on that user intent. In some examples, once the one or more servers have determined the task based on the audio input, they transmit the task, parameters (if any), and a list of acceptable device types for handling the task. Further details regarding task identification from natural language input can be found, for example, in U.S. Utility Application 15 / 144,618, filed May 2, 2016, entitled "Intelligent Device Identification," which is incorporated herein by reference in its entirety.

[0346] The device identification storage device (e.g., on a virtual assistant server) may be implemented via hardware, software, or a combination thereof. In some examples, the device identification storage device may reside on one or more of the aforementioned electronic devices, including a first user device (e.g., 580, 3300) and a second user device (e.g., 5380, 3304). The device identification storage device may reside on the same electronic device as either the hosting media identification storage device or the virtual assistant server.

[0347] At the device identification storage device, one or more user devices are identified based on parameters provided in a request from the virtual assistant server. In some examples, the device identification storage device determines whether any of the electronic devices with entries in the database has an attribute that matches the parameters. Further details regarding the identification of electronic devices can be found, for example, in U.S. Utility Application No. 14 / 503,105, filed September 30, 2014, entitled “Intelligent Assistant for Home Automation,” the entire disclosure of which is incorporated herein by reference.

[0348] In response to a request to identify one or more devices, the virtual assistant server receives identification information for the one or more devices. For example, the identification information includes a unique identifier corresponding to the identified user device (e.g., "uid=123456").

[0349] The virtual assistant server transmits a request for identifying one or more media items to a media identification storage device. This request includes one or more parameters, such as the parameter "Ramones," that take into account the identification of the one or more media items. The media identification storage device includes a database that associates (e.g., catalogs) a set of media items through one or more attributes, including title, artist, genre, length, and any other relevant information associated with the media items. By way of example only, one or more media items are identified based on the parameter "Ramones" located at the media identification storage device.

[0350] In response to a request to identify one or more media items, the virtual assistant server receives identification information for the one or more media items identified from a media identification storage device. In some embodiments, the identification information includes a unique identifier corresponding to the identified one or more media items, such as “artist: / / store / 56989”. The unique identifier of the one or more media items can be used, for example, with one or more applications that support media playback to identify the one or more media items and subsequently access them. The media identification storage device may reside on any local or remote electronic device (such as the same electronic device where the virtual assistant server resides).

[0351] In some implementations, the virtual assistant server provides a unified command to a first user device (e.g., 580, 3300). This command includes identification information for the one or more media items and identification information for a second user device (e.g., 3403). For example, the unified command includes the text string "SetMediaPlayerQueue artist: / / store / 56989AirplayRouteUid=12456". In response to receiving the unified command, the user device (e.g., 3300) initiates playback of the identified one or more media items (e.g., media related to Ramones) at the identified user device (e.g., the second user device 3304). In some examples, the first user device (e.g., 3300) transmits relevant information derived from the unified command to the second user device (e.g., 3304).

[0352] Now let’s turn our attention to the implementation of user interfaces (“UIs”) and associated processes on electronic devices (such as portable multifunction devices 100, 300 or 500).

[0353] Figures 6A-6AR Exemplary user interfaces for accessing the media playback status of multiple available playback devices are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 7A-7DThe process described below is the process in the middle.

[0354] Figures 6A-6B An exemplary user interface for managing and controlling a group of devices is shown. In some embodiments, the group of devices includes devices distributed throughout the user's home to perform various functions. Such functions may include controlling smart devices (e.g., light bulbs, thermostats, media devices), configuring home automation, and controlling home entertainment.

[0355] Figure 6A A home summary screen 601 (on a display 602 of device 600) is depicted representing an exemplary interface for a home control application (also referred to as a “device management application”) executed on device 600. In some embodiments, device 600 includes one or more features of device 100, 300, or 500. An example of a home control application is “Home” (developed by Apple Inc. (Cupertino, California, USA)). The home summary screen 601 includes stored scenes (e.g., a pre-configured set of settings for one or more home devices) and indications of devices in the user’s home (also referred to as “home devices”, etc.). Home devices are indicated by indicators 601A to 601C. Indicator 601A corresponds to a device named “Smart Speaker 1” in the room “Living Room”. Indicator 601B corresponds to a device named “Smart Speaker 2” in the room “Kitchen”. Indicator 601C corresponds to a device named “Smart Light Bulb” in the room “Bedroom”. Exemplary interfaces and techniques for home control applications are described in U.S. Patent Application 62 / 349,057, filed June 12, 2016, entitled “User Interface for Managing Controllable External Devices,” which is incorporated herein by reference in its entirety.

[0356] Figure 6AAn exemplary home configuration screen 603 is also depicted, illustrating an exemplary interface for managing configuration settings associated with home devices represented in the home summary screen 601. In some examples, the device 600 displays the home configuration screen 603 in response to a user input selection at the display representation 601D on the home summary screen 601. In the home configuration screen 603, the user can edit the name of the home (“Home” in this example), edit the members of the home (e.g., users 1, 2, and 3 in this example), include notes for other members, and other functions. It can be seen that “Home” in this example includes three members: user 1 (represented by indicator 603A), user 2 (represented by indicator 603B), and user 3 (represented by indicator 603C). In some examples, users 1 through 3 represent users residing at a location that physically includes home devices (collectively referred to as “Home”). Therefore, users 1 through 3 are members of the home. As members of the home (or other location), in some examples, users are granted permission to control devices associated with the home. Users can control home devices using one or more personal devices associated with at least one of the users. In some examples, a device is associated with a user when it is logged into an account associated with the user (e.g., owned by the user). For example, a device logged into one or more user accounts associated with the exemplary users 1 to 3 described above is associated with at least one user, and more specifically, with at least one user of Home (e.g., a home device).

[0357] Figure 6B An exemplary user interface of a portable electronic device 600 is depicted. A device home screen 605 displayed on a display 602 is an exemplary default home screen of the device 600. The device home screen 605 includes a power indicator associated with an application stored on the device. For example, a power indicator 605A corresponds to a music application. In some examples, selecting a power indicator 605A causes the device to display the user interface of the corresponding music application. Figure 6B An exemplary user input 604 received by device 600 when displaying screen 603 is also depicted. In this example, user input 604 is a swipe gesture (e.g., followed by a drag touch input) on the touch-sensitive display 602 of device 600. Figure 6BIn the depicted example, user input 604 is a swipe gesture originating from the edge of the touch-sensitive display 602. In some embodiments, user input 604 is an arc-shaped swipe gesture. For example, an arc-shaped swipe gesture is a gesture that includes both a horizontal component (e.g., x-direction) and a vertical component (e.g., y-direction) that conforms to an arc criterion. For example, user input 604 could be a touch and drag along a quarter circle on a touch-sensitive surface. In response to input 604, device 600 displays a control center interface (e.g., Figure 6C and Figure 6D ).

[0358] Figure 6C An exemplary control center interface is illustrated. The control center interface 606 includes various capability representations for controlling various functions and / or components of device 600. In the depicted example, the control center interface 606 includes exemplary illustrations of the following: media control interface 606A, volume control interface 606C, screen brightness control interface 606D, camera capability representation 606E, calculator capability representation 606F, timer capability representation 606G, flashlight capability representation 606H, wireless media streaming capability representation 606I, screen rotation lock capability representation 606J, do not disturb capability representation 606K, and wireless antenna control interface 606L. The exemplary media control interface 606A includes media playback control 606M (e.g., previous track, pause / play, next track) and device selection capability representation 606B. The exemplary wireless antenna control capability representation 606L includes capability representation 606N for toggling (e.g., turning on and off) the Wi-Fi functionality of device 600.

[0359] Figure 6D Another exemplary control center interface 606 is shown when device 600 is playing media through an external device. Figure 6D As shown, the device selection indicator 606B includes an indication that the media currently being played (e.g., the song track "Same Thing" by artist "Jambug") is being played through an external device (or multiple devices) called "AirPods". Therefore, the media control interface 606A indicates the media currently playing (also referred to as playing, currently in playback, etc.) and the device routing the audio output thereon. In this case, the audio output is generated by wireless personal earbuds ("AirPods" developed by Apple Inc. (Cupertino, California, USA)). Figure 6D Also shown is a volume control indicator 606C including an indicator 606M representing an "AirPods" device, indicating that the volume control indicator 606C can be used to adjust the volume of the audio output of the external device "AirPods".

[0360] Figure 6E As shown Figure 6C The exemplary control center interface shown is illustrated, but user input 608A is provided on the device selection indicator 606B. In some embodiments, in response to receiving user input 608A, device 600 displays a multi-device interface (e.g., such as...). Figure 6J (As shown). For example, user input 608A is a touch on the device select indicator (e.g., also referred to as a "tap" or "press").

[0361] Figure 6F As shown Figure 6C The exemplary control center interface shown is provided, but user input 608B is located at the position on the media control interface 606A. In some embodiments, in response to receiving user input 608B, device 600 displays a multi-device interface (e.g., as shown in the image). Figure 6J (As shown). In some examples, user input 608B is a deep press (e.g., a touch with a characteristic strength that meets a strength-dependent criterion). In some examples, the strength-dependent criterion is whether the characteristic strength of the touch exceeds a threshold strength. In some examples, user input 608B is a tap and hold (e.g., continuous detection of touch inputs reaching a duration greater than a threshold on the touch-sensitive surface of device 600). In some examples, user input 608B (e.g., whether a deep press or a tap and hold input) can be received at any location on the media control interface 606A (e.g., entirely or partially within the media control interface). For example, in response to receiving user input 606B on a playback control enable display (e.g., 606M), device 600 performs an operation different from what it would have done if the user input were a tap (e.g., device 600 performs a second operation instead of a playback control operation such as pausing playback).

[0362] Figure 6G An exemplary expanded media control interface of the device is shown. In some embodiments, in response to receiving user input 608B, device 600 displays... Figure 6GThe expanded media control interface 609 includes an indicator 609A representing the media currently being played on device 600. In this example, device 600 is named "iPhone" and is currently playing the media track "Same Thing" by artist "Jambug". Indicator 609A includes a graphic to the left of the track name representing the album art associated with the media track. In some examples, the graphic included in indicator 609A can be any graphic associated with the media track or that can otherwise provide a visual representation of the currently playing media. Other suitable graphic indicators include, but are not limited to: artist photos, album art, movie covers, movie posters, stylized text depictions, etc. The exemplary expanded media control interface 609 includes a media progress indicator 609B (e.g., depicting the time progress of the media track (elapsed time) and the amount of time remaining until the current media ends). The exemplary expanded media control interface 609 also includes a playback control indicator 609C (e.g., functioning similarly or identically to those described above), a volume control indicator 609D (e.g., functioning similarly or identically to those described above), and a device selection indicator 609E.

[0363] Figures 6H-6N An exemplary multi-device interface according to some implementation schemes is shown. Figure 6H An exemplary multi-device interface 610 is depicted, including power indicators 610B to 610D for the current device 600. In some embodiments, the device 600 displays the multi-device interface 610 in response to receiving user input 608A. In some embodiments, the device 600 displays the multi-device interface 610 in response to receiving user input 608B.

[0364] like Figure 6H As shown above regarding the expanded media control interface 609, the multi-device interface 610 includes an indicator 610A representing media (the media currently being played on the current device 600 (named "iPhone")), a media progress indicator 610B (e.g., depicting the time progress of the media track (elapsed time) and the amount of time remaining until the current media ends), and an exemplary multi-device interface 610 also includes a playback control indicator 610C (e.g., functioning similarly or identically to those described above), a volume control indicator 610D (e.g., functioning similarly or identically to those described above), and a device selection indicator 610E. Figure 6G Compared to the expanded media control interface 609, the multi-device interface 610 provides the user with information about the media playback status of multiple devices and the ability to influence the media playback status of the multiple devices.

[0365] like Figure 6H As shown, the multi-device interface 610 also includes indicators associated with other playback devices. Indicator 610F corresponds to one or more devices named "Living Room" (or within a group of devices named "Living Room"). In some embodiments, a group of devices may include one or more devices. The name "Living Room" may be assigned to the group of devices based on their location in the user's home. For example, when configuring one or more devices in a group named "Living Room" (e.g., before first use) using device 600 and a home control application, the name "Living Room" is assigned to said one or more devices in that group. Thus, when selecting or viewing the playback status of devices throughout the home, the user is given a visual identification of the group of devices by location. In some embodiments, the name of the device or group of devices is a default name. For example, if the user does not configure a device to have a specific name, the device may simply be named "Device 1" or "Device 1 - Living Room" (if location is associated) (e.g., by the user or automatically). In some embodiments, the name of the device or group of devices is a user-configured name. For example, the user selects a name from a predefined group of names. In some examples, the name displayed for a device (e.g., in a multi-device interface) is the device's location. For example, a device might be named "Tim's Speaker" (e.g., in configuration settings or a home control app) but would appear as "Tim's Room" based on the location associated with the device. In some implementations, a group of devices has a custom name. For example, a user could assign a personalized name to a group of devices, such as "Tim's Room." In some implementations, names other than location names are used. For example, a user could give a group of devices an arbitrary name.

[0366] In this example, a group of devices named "Living Room," represented by indicator 610F, is currently playing a media item titled "Heart Wave" by artist "MKTS," with the artist and title indicated within indicator 610F. In some implementations, the media item is one or more of the following: song, video, podcast, audio stream, video stream, audio file, video file, ebook (e.g., also referred to as an ebook), text document, webpage, telephone call, video call, etc. Indicator 610G corresponds to a group of devices named "Bedroom," or included in a group named "Bedroom," which is currently playing a song titled "Vibes" by artist "John Appleseed." In some implementations, indicators (e.g., indicators 610F and 610G) are displayed for the corresponding group of devices even when a group of devices is not currently playing media (e.g., in a multi-device interface). In some implementations, even when not all devices in a group are currently playing media, an indicator for the corresponding group of devices (e.g., indicator 610F and indicator 610G) is displayed (e.g., in a multi-device interface). For example, if a living room device group includes three devices (e.g., three smart speaker devices, such as device 580), and only one of these devices is currently playing a media item (e.g., "Heart Wave" by "MKTS"), the corresponding indicator in the multi-device interface still includes the identification of the currently playing media (e.g., "Heart Wave" by "MKTS" is displayed inside indicator 610F for the device group "living room").

[0367] In some implementations, if individual devices in the group are simultaneously playing different media items, the multi-device interface displays indications that multiple media items are being played on devices in the group. For example, the multi-device interface may include a single indicator for the group (e.g., 610F) and identifications of multiple media items being played on those devices (e.g., "1 movie, 1 song"). Alternatively, the multi-device interface may include two indicators for the group (e.g., 610F) and identifications of each corresponding media item being played (e.g., for each media item, the indicator such as 610F is both identified as "living room," or as "living room - device 1" and "living room - device 2").

[0368] A set of devices may also be referred to as a set of "available playback devices". In some implementations, available playback devices include one or more (e.g., remote, external) devices on which the current device (e.g., device 600) can control or influence playback (e.g., using a multi-device interface such as 610). For example, as Figure 6HAs shown, if a set of devices called the living room includes three external devices (e.g., Bluetooth, Wi-Fi, etc.) connected to device 600 via a communication link (e.g., Bluetooth, Wi-Fi, etc.) Figure 5I Devices in a group called the living room (580) and which can be controlled by instructions issued by device 600 (e.g., indicating user input) are considered "available playback devices". In some embodiments, influencing or controlling playback on available playback devices is subject to conditions or limitations. For example, device 600 may see the devices in the living room indicated by indicator 610F, but modifying the playback status of these devices may be subject to permission (e.g., if the media being played back on the living room device is initiated by another device or user). In some embodiments, available playback devices include the current device. For example, in an example where the living room includes three devices, multiple available devices include device 600 and the three living room devices, totaling at least four devices. Therefore, the multi-device interface 610 provides a convenient interface that indicates what is playing on the current device and other devices.

[0369] In some implementations, one or more devices automatically detect another device. In some implementations, the devices then form a network between them. For example, two devices (e.g., 580) may detect each other after initial startup (e.g., out of the box, not configured by the user), establish a connection link between them (e.g., and between device 600), and appear in a multi-device interface (e.g., with a name such as "Speaker 1" and / or a location such as "No Location"). In some implementations, a device (e.g., 580) is configured as part of a user's home (e.g., a home appliance). Those skilled in the art will understand that a wide variety of technologies can be used to interconnect and / or transmit information between devices (e.g., 580, 600), all of which are contemplated within the scope of this disclosure and therefore additional details are not included herein.

[0370] exist Figure 6I In this context, device 600 receives user input 612 indicating a selection of device selection display 610E. For example, user input 612 may be a tap or touch at a location associated with the displayed device selection display 610E.

[0371] Figure 6J Exemplary multi-device user interfaces are illustrated according to some embodiments. In some embodiments, device 600 displays multi-device interface 610 in response to receiving user input 608B. In some embodiments, device 600 displays multi-device interface 610 in response to receiving user input 612. In some examples, in response to selecting a device enablement representation (e.g., in any media application)... Figure 6I The 610E) option displays a multi-device interface.

[0372] like Figure 6J As shown, the multi-device interface 610 includes individual indicators 610H to 610O for each of the available playback devices. In the depicted example, the individual indicators are visually associated with indicator 610A (which identifies the media track "Same Thing" by "Jambug" and the device currently playing that media (in this case, only the device named "iPhone")). Indicator 610H identifies the current device (named "iPhone") and includes a selection indicator 610I. In this example, selection indicator 610I includes a checkmark indicating that device 610H is currently playing the media of indicator 610A. Indicator 610J identifies a group of devices named "Bedroom" and includes indicator 610K which does not include a checkmark (e.g., the devices in the Bedroom group are not currently playing the media track "Same Thing" by "Jambug"). Indicator 610J also includes an indication that the devices in the Bedroom group are currently playing other media. For example, in Figure 6J In the diagram, indicator 610J includes the title of the track "Vibes". The visual indicator conveys to the user that the bedroom device group is currently busy. If the user wishes to add the bedroom device group to the current playback of "Same Thing" by "Jambug" (e.g., by selecting indicator 610K), doing so interrupts the media currently playing in the bedroom. Indicator 610L identifies a group of devices named "Kitchen" and includes indicator 610M without a checkmark (e.g., the kitchen group devices are not currently playing the media track "Same Thing" by "Jambug"). Indicator 610N identifies a group of devices named "Living Room" and includes indicator 610O without a checkmark (e.g., the living room group devices are not currently playing the media track "Same Thing" by "Jambug"). Indicator 610N also includes an indication that the living room group devices are currently playing other media. For example, in... Figure 6J In the middle, indicator 610J includes the title of the track "Vibes".

[0373] like Figure 6J As shown, individual indicators are visually associated with indicator 610A. It can be seen that individual indicators 610H to 610O are included within the box-like outline of indicator 610A. In some examples, this provides a visual indication that the multi-device interface is currently "pointing" or "focusing" on the "media session" identified by indicator 610A. In some embodiments, a media session includes devices and media items. Figure 6JIn the example shown, indicator 610A represents a media session that includes the media item "Same Thing" created by "Jambug" and the device "iPhone" (e.g., device 600). In some embodiments, a media session includes multiple devices. In some embodiments, a media session includes multiple media items. For example, a media session may include playback of a playlist or another set of multiple media items. Visually associating indicators for individual devices with indicators for media sessions visually indicates that selecting an individual device adds that device to the media session. This visual indication is useful, for example, when multiple concurrent media sessions exist in a user's home. For example, indicators 610F and 610G respectively represent other media sessions (e.g., playback of different media on different devices).

[0374] like Figure 6J As shown, the indicator for a single device has replaced the media playback control enable indicator (e.g., 610B to 610D). In some embodiments, both the indicator for a single device and the media playback control enable indicator are displayed simultaneously.

[0375] In some implementations, the indicator associated with a group of devices or a single device includes identification of the user who initiates playback of media on that group of devices or the single device. For example, refer to Figure 6J When a media session in the bedroom is initiated by a user other than the user currently associated with device 600 (e.g., Tim), one or more of indicators 610G or 610J can identify the user who initiated the media playback in the bedroom—for example, “Steve is playing Vibes by John Appleseed.”

[0376] Figure 6J The multi-device interface 610 may include a power indicator 610R. For example, selecting the power indicator 610R may cause the display of indicators for individual devices to stop (and, for example, the media playback control power indicator to reappear). In some examples, if device 600 receives a selection for one or more individual indicators 610K, 610M, or 610O (e.g., causing device 600 to display a checkmark within the indicator), the selection of the power indicator 610R causes device 600 to initiate media playback on the corresponding selected device in response. In some embodiments, device 600 initiates playback in response to a selection for one or more individual indicators 610K, 610M, or 610O.

[0377] Figure 6KThe illustration shows device 600 receiving a selection of a separate indicator for a device among the available playback devices. User input 614 is received on indicator 610 associated with the kitchen group device (represented by 610L). In some examples, user input 614 is a tap or touch at a location associated with indicator 610M. In some examples, user input 614 is a deep press at a location associated with indicator 610M.

[0378] Figure 6L An exemplary multi-device interface 610 is shown after receiving a selection (e.g., user input 614) for a device among the available playback devices. Figure 6L As can be seen, indicator 610A now includes an indication that the current media session includes the kitchen appliance group and a device named iPhone (e.g., device 600) (“Kitchen + iPhone”). Furthermore, indicator 610L has been updated to include a separate volume control indication, and indicator 610M now includes a checkmark. In some examples, the kitchen appliance group is added to the current media session of indicator 610A in response to a user input selection of its associated indicator 610M. In some examples, additional user input on indication 610R (e.g., user input 615) causes the kitchen appliances to be added to the current media session of indicator 610A.

[0379] Figures 6M-6N This illustrates switching between media sessions when using an exemplary multi-device interface. Figure 6M The image shows a multi-device interface 610 with a first media session associated with indicator 610A (e.g., playback of "Same Thing" by artist "Jambug" on device "iPhone"). Figure 6M As shown, device 600 receives user input 616 associated with indicator 610F, which is associated with a second media session (e.g., playback of "Heart Wave" by artist "MKTS" on a "living room" group device). In some examples, user input 616 is a tap or touch at the location associated with indicator 610F. In some examples, user input 616 is a deep press at the location associated with indicator 610F.

[0380] In some implementations, in response to user input 616, device 600 updates the display of the multi-device interface to "focus" (also referred to as "pointing") the second media session. For example, Figure 6NThe indicator 610A associated with the first media session is depicted as minimal (e.g., no visually associated indicator for a single available playback device or media playback control). However, indicator 610F has been expanded to include a separate indicator for each of the plurality of available playback devices. Alternatively or in addition, expanded indicators for the media session (e.g., Figure 6N The indicator 610F may include a media playback control function display. For example... Figure 6N As shown, indicators 610H, 610J, 610L, and 610N are now visually associated with indicator 610F for a second media session on the living room group equipment. In particular, indicators 610M and 610O now include checkmarks indicating that they are part of the currently selected media session (e.g., associated with expand indicator 610F).

[0381] Switching between media sessions can be referred to as "redirection," "direction," "refocusing," "focusing," etc. Therefore, selecting indicator 610F can be referred to as "focusing" the current device 600 or making the multi-device interface 610 "focus" on a group of devices or media sessions.

[0382] As mentioned above, in contrast to Figures 6H-6N As shown, the multi-device interface can be used to view the media playback status of multiple available playback devices. Additionally, the multi-device interface can be used to view the content currently being played by each device in a group of available playback devices. Furthermore, the multi-device interface can be used to add or remove individual devices from a media session. Besides... Figures 6H-6N In addition to the example shown, other exemplary techniques for invoking and interacting with a multi-device interface can be used, as described below.

[0383] Figure 6O-Figure 6Y Exemplary techniques for interacting with multi-device user interfaces are shown according to some implementation schemes. Figures 6O-6R Exemplary techniques for directly initiating playback to one or a dynamically selected external device are shown. Figure 6O An exemplary browsing interface is shown. Browsing interface 620 depicts an interface for browsing media items. For example, a user can use the browsing interface to find and select one or more media items for playback. Exemplary browsing interfaces include interfaces for browsing music libraries, video libraries, media content available for streaming, etc. In this example, browsing interface 620 is a music browsing interface. Indicator 620A indicates that the device is displaying a list of media items titled "New Music" (e.g., a playlist). Media item 620B (also referred to as song 620B) is a song titled "Monday Morning" by the artist "The Fomoers".

[0384] exist Figure 6P In this example, device 600 receives user input 622 on song 620B. In some examples, user input indicating a selection of a media item (e.g., a tap on a song) causes playback of that song to be initiated on the current device. For example, if device 600 is currently only “pointed” to itself, a tap on a media item causes playback to be initiated on device 600. As another example, if device 600 is currently pointed to itself and the device group “Living Room,” a tap on a media item causes playback to be initiated on device 600 and the devices in that group named “Living Room.” In some examples, a deep press of user input on a media item or a tap and hold of user input causes device 600 to perform a secondary operation (e.g., immediately initiating playback of the corresponding media item on one or more of the pointed-to devices). In some examples, the secondary operation allows the user to select one or more devices on which they want to initiate media playback before initiating playback of the corresponding media. In this way, users can directly begin playback on the desired set of devices without first initiating playback on the current device (e.g., device 600) and then transferring the playback to one or more external devices (e.g., via a cumbersome interface setup). One disadvantage of initiating playback on the current device (e.g., a personal device) and then transferring it to other devices (e.g., a home theater system) is that the media item has already started playing on the device before playback is initiated on the desired set of devices—which is undesirable if playback is being performed in a room full of listeners. Even if playback is restarted in response to transferring it to the desired set of devices or afterward (manually or automatically), the original device that initiated the playback will waste device resources by starting the song's playback, an operation that is undesirable to the user. In some examples, a tap of the input invokes a primary action, and a deep press or tap and hold of the input invokes a secondary action. Those skilled in the art will recognize that any two user inputs that can be recognized by the device as different types can be used to invoke the corresponding primary and secondary actions.

[0385] Figure 6Q An exemplary media item menu is shown. For example, in response to user input 622, a media item menu 624 is displayed, whereby the user input is a deep press or tap and hold input (e.g., invoking a secondary action related to the media item). Menu 624 includes a list of possible actions selectable for media item 620B (a song titled "Monday Morning" by "The Fomoers"). Media item menu 624 includes a switch to speaker playback indication 624A. Figure 6RIn this context, device 600 receives user input 626 on the speaker playback enable indicator 624A. For example, user input 626 is a tap. In some examples, in response to receiving user input 626 indicating a selection to switch to speaker playback enable indicator 624A, device 600 displays a multi-device interface (e.g., similar to...) for selecting one or more devices on which playback should be initiated. Figure 6J A multi-device interface 610 may be displayed. For example, a multi-device interface may include indicators for a media session (e.g., 610A) that include the media item "Monday Morning" by "The Fomoers" and that no device is selected (e.g., no device with an indicator including a checkmark). In this example, after selecting the one or more desired devices (e.g., adding them to the media session), playback is then initiated on the desired device (e.g., device 600 initiates playback on itself and / or transmits instructions to other devices in the group of desired devices). In this example, user input associated with the media item (e.g., 622) is used to invoke the multi-device interface (e.g., 610) so that one or more external devices can be selected before device 600 initiates playback of the media. Thus, device 600 initiates the media item from the beginning (e.g., from the start of the song) on ​​the selected one or more external devices (e.g., as shown in the image). Figure 6P The playback of “Monday Morning” shown is performed instead of first initiating playback on device 600 and then transmitting the playback (e.g., during playback).

[0386] Figure 6S-Figure 6Y An exemplary technique for providing a device focus indication is shown. Figure 6S An exemplary current media interface is shown. The current media interface 630 (also referred to as the "now playing interface") identifies the media item currently being played. The current media interface 630 includes album art 630A, media identifier 630B (song title "Same Thing"), media identifier 630C (artist "Jambug"), media identifier 630D (album "Fly Away"), and device selection indication 630E. Figure 6S As shown, the device selection indication 630E does not include an identifier for another device. Therefore, in this example, the media item identified in the current media interface 630A is currently being played back on the current device (device 600). In some examples, the media identified by the current media interface is currently being played back on another device (e.g., in a group other than the current device or excluding the current device).

[0387] exist Figure 6TIn the process, device 600 receives user input 632 on device selection indicator 630E. In response to receiving user input 632, in... Figure 6U In the middle, the device displays a multi-device interface 610. For example... Figure 6U As can be seen, the first area 611A of the display (shown, for example, with the first dashed line pattern as its boundary for illustrative purposes) includes the multi-device interface 610, and the second area 611B of the display (shown, for example, with the second dashed line pattern for illustrative purposes) includes a visible indication of the content displayed before the multi-device interface is displayed. In this case, the content displayed before the multi-device interface is the current media interface 630. For example, the album cover 630A is visible through the multi-device interface. In this example, the first area visually overlays the second area.

[0388] exist Figure 6V In this process, device 600 receives user input 634 on indicator 610F. In some examples, user input 634 is a tap or touch at a location associated with indicator 610F. In some examples, user input 634 is a deep press at a location associated with indicator 610F.

[0389] like Figure 6W As shown, in some examples, in response to user input 634, device 600 displays a multi-device interface 610, where indicator 610F is expanded to show individual indicators for the devices in the group of available playback devices that are visually associated with indicator 610F. Therefore, device 600 now focuses on the media session encompassing the living room of the device group. It can be seen that in Figure 6W In this context, the current media interface 636 (described below) is visible (via the multi-device interface 610). Specifically, as... Figure 6W As seen, in response to input 634, device 600 stops displaying the visual indication associated with the current media interface 630 and displays the visual indication associated with the current media interface 636. This is a replacement of the current media interface, instructing the device to change the media session it is pointing to from the session associated with indicator 610A to the session associated with indicator 610F. Figure 6U ).

[0390] like Figure 6X As shown, in some examples, in response to user input 634, device 600 stops displaying the multi-device interface 610 and displays the current media interface 636. Therefore, Figure 6XThe diagram illustrates an alternative response from device 600 to received user input 634. Similar to interface 630, the current media interface 636 includes album art 636A, media identifier 636B (song title “Heart Wave”), media identifier 636C (artist “MKTS”), media identifier 636D (album “MKTS”), and device selection enablement 636E. However, each of these indicators has been updated to reflect changes in the media items and devices of the selected media session (e.g., because device 600 now points to a media session including a different set of devices). Specifically, device selection enablement 636E includes an identifier 636F identifying the currently pointed-to device. Figure 6X As shown, identifier 636F includes the text "living room", which indicates that the current media interface 636 is the current media interface for media currently being played on a set of devices called living room.

[0391] Even when viewing the interface of a non-media-related application on their device, users may expect to check the media status of multiple devices in their home at any time. Therefore, this article describes a technique for easily and quickly accessing a multi-device interface from any application running on the device. Figure 6Y-Figure 6AA Exemplary techniques for accessing multi-device interfaces are shown according to some implementation schemes. Figure 6Y An exemplary interface for an application is shown. Web browsing interface 640 depicts a currently displayed web browsing session (e.g., the homepage of the website "Apple.com"). Web browsing interface 640 includes web page content 640A. Figure 6Z In this context, device 600 receives user input 642. In some examples, user input 642 is a swipe gesture originating from the edge of a touch-sensitive surface of device 600. In some examples, in response to user input 642, device 600 displays a playback control interface. For example, in response to user input 642, device 600 displays... Figure 6C The playback control interface 606A. In some embodiments, in response to receiving user input 608A on the device selection enablement display 606B at the playback control interface 606A, the device 600 displays a multi-device interface (e.g., Figure 6AA (610).

[0392] Figure 6AA An exemplary multi-device interface is shown, visually overlaid on the application interface. For example... Figure 6AA As shown, the multi-device interface 610 includes separate indicators for available playback devices. Figure 6AAIn the illustrated example, only one media session is active on the available devices in the user's home, and this session includes playback of "Same Thing" performed by "Jambug" on the current device 600. In some examples, the multi-device interface is displayed simultaneously with previously displayed content. For example, as... Figure 6AA As shown, the multi-device interface 610 (e.g., displayed in the first area) visually overlays the web browsing interface 640 (e.g., displayed in the second area).

[0393] Figures 6AB-6AC An exemplary technique for adjusting the volume of an available playback device is shown. Figure 6AB A multi-device interface 610 is depicted with a single media session, which includes playback of the media item "Same Thing" made by "Jambug" on three devices: "iPhone" (e.g., device 600), kitchen group device, and living room group device. Figure 6AB Individual volume control indicator 610S (corresponding to device iPhone), individual volume control indicator 610T (corresponding to device group kitchen), and individual volume control indicator 610U (corresponding to device group living room) are depicted. For example, the individual volume control indicator is selectable and manipulate (e.g., left and right) to adjust the volume of its respective device. In some examples, individual volume control indicator is not displayed for devices not included in the current media session (e.g., ...). Figure 6AB The device in the bedroom does not correspond to the individual volume control indicator shown. Figure 6AB A group volume control indicator 610D is also depicted, which is selectable and manipulatory (e.g., left and right) to adjust the volume of multiple devices included in a media session, as indicated by indicator 610A. For example, in Figure 6AB In this context, device 600 receives user input 644 on group volume control indicator 610D, which represents a tap and swipe gesture in the left direction (e.g., indicating a request to adjust the volume). In response to user input 644, the device lowers the group volume indicated by the position of the swipe indicator 610D, as shown below. Figure 6AC As shown. Figure 6AC It also shows that the individual volume control indications for each device in the group have been turned down accordingly.

[0394] Figure 6AD-Figure 6AGAn exemplary interface for initiating playback on a preferred device is shown. In some cases, a user may expect media playback to be automatically initiated on one or more preferred devices. For example, because media content may include visual aspects (e.g., if the media is video), it is desirable to automatically initiate playback of such content on a device with a display or on a device with a large display (e.g., larger than the device that receives the user's input request to initiate playback). For example, a user may browse media content using their personal device with a relatively small screen (e.g., device 600) but expect to begin playback of the selected media item (e.g., a feature film) on their large-screen TV. In other examples, a user may request media playback via voice command to a device that does not have video playback capabilities (e.g., device 580). In this latter example, the device without video playback capabilities is able to receive the user's request but delegates the responsibility of video playback to another device.

[0395] Switch to the attached image. Figure 6AD An exemplary browsing interface is shown. Figure 6AD As shown, the media browsing interface 650 includes media items with video content. In this example, media item 650A represents an episode of a television program called "Super Heroes". Figure 6AE In this process, device 600 receives user input 652 associated with media item 650A. In some examples, user input 652 is a tap or touch on media item 650A. In some examples, user input 652 is a deep press on media item 650A.

[0396] In response to receiving user input 652, device 600 displays... Figure 6AFThe message interface 654 includes an instruction (“Start playback on Apple TV”) that playback of video content has been initiated or has been initiated on a different device than device 600. In this example, the device named “Apple TV” is different from device 600 and represents the preferred device for playback of video media. For example, the Apple TV device may be connected to the user’s large-screen TV. In some implementations, in response to user input (e.g., user input 652), the device (e.g., 600) determines whether it is currently connected to a home network. Based on the determination that the device is currently connected to a home network, the device initiates playback on the preferred device (e.g., transmits an instruction to initiate playback on the preferred device). For example, if the device determines that the user is at home, it abandons initiating playback on itself and initiates playback on an external preferred device. Based on the determination that the device is not currently connected to a home network, the device initiates playback on itself. For example, if the device determines that the user is not at home, it initiates playback on itself. In some examples, the preferred device is a set of preferred devices (e.g., one or more). In some examples, the preferred device (or set of devices) is predetermined. For example, a user can select a preferred device and save it to their device or user account settings. In another example, the preferred device is selected automatically. For instance, if a user's home includes only one recognized external device capable of playing back video (e.g., Apple TV), it can be designated as the preferred device. In some implementations, the device prompts the user whether to initiate playback on the preferred device. For example, regardless of whether the user has pre-configured the preferred device, in response to user input 652, device 600 may prompt (e.g., visually or audibly) and ask, "Would you like to initiate playback of 'Super Heroes' on your Apple TV?".

[0397] For example, each device (e.g., a group of devices such as each of a set of preferred devices, such as device 600 and device Apple TV) generates and broadcasts a set of one or more values ​​(e.g., in response to user input such as 652). In this example, each set of one or more values ​​includes a "Device Type" value. The device (e.g., 600) analyzes the broadcast "Device Type" value based on the task specified in the user request (e.g., 652). The device (e.g., 600) makes the determination based on a list of device types broadcast by other electronic devices (e.g., Apple TV) and acceptable device types. In some examples, if the device (e.g., 600) determines that none of the other electronic devices has an acceptable device type for processing the task, the device outputs an error message to the user or prompts the user for additional input (e.g., "Would you like to play this video back on your iPhone?").

[0398] Figure 6AGAn exemplary playback control interface for video content is shown. Playback control interface 656 includes an indication 656A of the currently playing media content (“Super Heroes”), playback control enablement 656B (e.g., rewind 10 seconds, pause, and skip 10 seconds), and an indication 656C of the device currently playing the media content (“Apple TV”). In some embodiments, interface 656 is displayed after message interface 654.

[0399] Figure 6AH-Figure 6AQ An exemplary multi-device interface is illustrated. As described herein, a multi-device interface can be used to view and control the media playback status of multiple available playback devices in a set of interconnected devices (e.g., devices in a user's home associated with a user's account). Therefore, multiple different types of devices can be controlled, and each device can have a different interface. For example, a set of devices in a home may include a first smart speaker (e.g., 580) manufactured by a first manufacturer, a second smart speaker (e.g., 580) manufactured by a second manufacturer different from the first manufacturer, and a home media hub (e.g., Apple TV developed by Apple Inc. (Cupertino, California, USA)). An exemplary home media hub may include or otherwise connect to a large display (e.g., a display larger than personal device 600), and therefore may have the same, similar, or different interfaces as those described above with respect to device 600. Further exemplary interfaces are described in detail below.

[0400] Figure 6AH An exemplary current media interface 664 displayed on a display 660A of device 660 is shown. In some embodiments, device 660 includes one or more features of device 100, 300, or 500. Device 660 itself is a device included in a set of devices associated with a user's home (or otherwise configured to act as a display of a device included in a set of devices associated with a user's home). The current media interface 664 includes a device selection enablement display 664A (e.g., which is associated with...). Figure 6SThe device selection indicator 630E functions similarly or identically. The device selection indicator 664A is associated with the text "Living Room Apple TV," which indicates that device 660 is currently pointing to itself (i.e., device 660). The current media interface 664 also includes an indication of the currently playing media (e.g., indicator 664C), such as a song titled "Heart Wave" by artist "MKTS" (e.g., indicator 664E) titled "Heart Wave" (e.g., indicator 664D). In some examples, indicator 664C of the currently playing interface 664 is an animated indicator, further conveying that the associated media item is currently playing back. Indicators 664F and 664G correspond to one or more other media items that can be played back.

[0401] Figure 6AH An exemplary remote control for controlling device 660 is also shown. In some examples, remote control 662 is connected to device 660 (e.g., paired) and used by a user to interact with an interface displayed on device 660. Because the remote control is essentially an extension of the device for remotely controlling it, for the purposes of this disclosure, user input received at device 662 is considered to be the same as user input received at device 660, unless otherwise stated herein. For example, a tap input received on device 662 indicating a request to select a power indication is considered to be the same as a tap on device 660 indicating a request to select the same power indication (e.g., if device 660 includes a touch-sensitive surface), unless otherwise stated. However, those skilled in the art will understand that user input can be received at any device and communicated to another device (e.g., for processing and performing actions in response), and the specific device that physically receives the user input is not important unless specifically stated. Similarly, in the example where device 660 is an external display device (e.g., a television) used by another device (e.g., an Apple TV, a "home device" that receives and processes user-input playback commands), device 660 as mentioned herein is interchangeable with said other device because device 660 is, in effect, an extension of the other device, unless otherwise stated. For example, references to "focus" or "pointing" on device 660, relative to control media, refer to the focus or pointing of the other device (e.g., Apple TV), unless otherwise stated.

[0402] exist Figure 6AHIn this embodiment, device 662 receives user input 668 on device 662. User input 668 indicates a selection of device selection indication 664A. In some examples, user input 668 is a button press on device 662, and device selection indication 664A is highlighted or otherwise selected at interface 664. In some examples, user input 662 is a tap, a click, and a hold or deep press on a touch-sensitive surface. In some implementations, in response to receiving user input 668, device 660 displays a multi-device interface (e.g., ...). Figure 6AI (670).

[0403] Figure 6AI An exemplary multi-device interface is illustrated according to some embodiments. Multi-device interface 670 includes identification of the current focus of device 660 (e.g., the currently selected media session). Header 670A indicates that the current focus of the device is a living room Apple TV (e.g., device 660). Multi-device interface 670 also includes a first indicator 670B that indicates the currently pointed-to media session. For example, the media session indicated by indicator 670B includes playback of the song titled "Heart Wave" by artist "MKTS" from the album "MKTS". Header 670A and indicator 670B (which may be collectively referred to as indicators) together indicate the currently pointed-to media session of interface 670 (e.g., the media being played and the device performing the playback). Multi-device interface 670 also includes a second indicator 670C, which corresponds to a media session different from the currently selected media session. Figure 6AI As shown, indicator 670C is displa...

Claims

1. An electronic device, comprising: a display; one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: initiating an initial configuration process for configuring a second device, the initial configuration process including sequential display of a plurality of different user interfaces corresponding to a plurality of different operations in the initial configuration process, the sequential display of the plurality of different user interfaces being predefined by the electronic device, wherein at least one user interface of the plurality of different user interfaces requests user confirmation that a connection should be established with the second device; and during the initial configuration process and the sequential display of the plurality of different user interfaces, and prior to providing an indication that the initial configuration process is complete: displaying, on the display, a first configuration user interface, the first configuration user interface being included in the plurality of different user interfaces and including a prompt to select a location, the prompt indicating that a selected location is to be assigned to the second device; receiving user input indicating a selection of a respective location; in response to receiving the user input indicating the selection of the respective location, configuring the second device to be assigned to the respective location; displaying, on the display, a second configuration user interface, the second configuration user interface being included in the plurality of different user interfaces and indicating that the second device is outputting an audio tone signal indicating a pairing handshake with the electronic device; while displaying the second configuration user interface on the display, detecting the audio tone signal indicating the pairing handshake; and in response to detecting the audio tone signal, proceeding to a next step in the initial configuration process for configuring the second device.

2. The electronic device of claim 1, wherein the audio tone signal is generated by the second device.

3. The electronic device of claim 1, wherein the audio tone signal includes an audio passcode. The audio tone signal is detected when the electronic device is physically located within a threshold physical proximity to the second device. 4.The electronic device of claim 1, wherein, 5. The electronic device of claim 1, the one or more programs further including instructions for: after displaying a user interface indicating completion of the initial configuration process: displaying, on the display, a configuration user interface, the configuration user interface including a stereo pair affordance; receiving input corresponding to a selection of the stereo pair affordance; and in response to receiving the input corresponding to the selection of the stereo pair affordance, initiating a process to pair a third device with the second device.

6. The electronic device of claim 1, the one or more programs further including instructions for: receiving user input representing a positive request to create a group of devices; in response to receiving the user input representing the positive request to create the group of devices, displaying, on the display, a configuration user interface, the configuration user interface including a prompt to select an audio output channel associated with the second device; ​ receiving user input indicative of a selection of a first audio output channel; and in response to receiving the user input indicative of the selection of the first audio output channel, associating the second device with the first audio output channel.

7. The electronic device of claim 1, wherein, The prompt to select a location includes a plurality of predetermined location identifiers.

8. The electronic device of claim 7, wherein, The plurality of predetermined location identifiers includes a user-defined location identifier.

9. The electronic device of claim 7, wherein, The prompt to select a location includes a field for creating a new location identifier, the one or more programs further including instructions for: receiving user input associated with the field for creating the new location identifier, wherein the user input includes a first location identifier; creating the first location identifier based on the received user input associated with the field for creating the new location identifier; and associating the second device with the first location identifier.

10. The electronic device of claim 1, the one or more programs further including instructions for: after initiating the initial configuration process, receiving data regarding output of an operation being performed by the second device; and in response to receiving data regarding output of an operation being performed by the second device, displaying a respective user interface.

11. The electronic device of claim 1, the one or more programs further including instructions for: after the initial configuration process, displaying a configuration user interface associated with a voice assistant, wherein the configuration user interface associated with the voice assistant includes a prompt to provide a user voice input that invokes one or more functions.

12. A computer-implemented method, comprising: on an electronic device with a display: initiating an initial configuration process for configuring a second device, the initial configuration process including a sequential display of a plurality of different user interfaces corresponding to a plurality of different operations in the initial configuration process, an order of the sequential display of the plurality of different user interfaces being predefined by the electronic device, wherein at least one user interface of the plurality of different user interfaces requests a user to confirm that a connection should be established with a second device; and during the initial configuration process and the sequential display of the plurality of different user interfaces, and prior to providing an indication that the initial configuration process is complete: displaying, on the display, a first configuration user interface included in the plurality of different user interfaces and including a prompt to select a location, the prompt indicating that a selected location is to be assigned to the second device; receiving user input indicative of a selection of a respective location; in response to receiving the user input indicative of the selection of the respective location, configuring the second device to be assigned to the respective location; displaying, on the display, a second configuration user interface included in the plurality of different user interfaces and indicating that the second device is outputting an audio tone signal indicating a pairing handshake with the electronic device; while displaying the second configuration user interface on the display, detecting the audio tone signal indicative of the pairing handshake; and in response to detecting the audio tone signal indicative of the pairing handshake, displaying, on the display, a third configuration user interface included in the plurality of different user interfaces and indicating that the second device is paired with the electronic device. continuing with a next step in the initial configuration process for configuring the second device in response to detecting the audio tone signal.

13. The computer-implemented method of claim 12, wherein the audio tone signal is generated by the second device.

14. The computer-implemented method of claim 12, wherein the audio tone signal comprises an audio password.

15. The computer-implemented method of claim 12, wherein, detecting the audio tone signal when the electronic device is physically located within a threshold physical proximity to the second device.

16. The computer-implemented method of claim 12, further comprising: after displaying a user interface indicating completion of the initial configuration process: displaying a configuration user interface on the display, the configuration user interface including a stereo pair affordance; receiving input corresponding to a selection of the stereo pair affordance; and in response to receiving input corresponding to a selection of the stereo pair affordance, initiating a process to pair a third device with the second device.

17. The computer-implemented method of claim 12, further comprising: receiving user input representing a positive request to create a device group; in response to receiving the user input representing the positive request to create the device group, displaying a configuration user interface on the display, the configuration user interface including a prompt to select an audio output channel associated with the second device; receiving user input representing a selection of a first audio output channel; and in response to receiving the user input representing a selection of the first audio output channel, associating the second device with the first audio output channel.

18. The computer-implemented method of claim 12, wherein, the prompt to select a location includes a plurality of predetermined location identifiers.

19. The computer-implemented method of claim 18, wherein, the plurality of predetermined location identifiers includes a user-defined location identifier.

20. The computer-implemented method of claim 18, wherein, the prompt to select a location includes a field for creating a new location identifier, the computer-implemented method further comprising: receiving user input associated with the field for creating the new location identifier, wherein the user input includes a first location identifier; creating the first location identifier based on the received user input associated with the field for creating the new location identifier; and associating the second device with the first location identifier.

21. The computer-implemented method of claim 12, further comprising: after initiating the initial configuration process, receiving data regarding output of an operation being performed by the second device; and in response to receiving data regarding output of an operation being performed by the second device, displaying a corresponding user interface.

22. The computer-implemented method of claim 12, further comprising: after the initial configuration process, displaying a configuration user interface associated with a voice assistant, wherein the configuration user interface associated with the voice assistant includes a prompt to provide user voice input invoking one or more functions.

23. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs comprising instructions for: initiating an initial configuration process for configuring a second device, the initial configuration process including sequential display of a plurality of different user interfaces corresponding to a plurality of different operations in the initial configuration process, the sequential display of the plurality of different user interfaces being predefined by the electronic device, wherein, requesting, from a user, confirmation that a connection should be established with a second device; and during the initial configuration process and the sequential display of the plurality of different user interfaces, and prior to providing an indication that the initial configuration process is complete: displaying, on the display, a first configuration user interface, the first configuration user interface included in the plurality of different user interfaces and including a prompt to select a location, the prompt indicating that a selected location is to be assigned to the second device; receiving user input indicative of a selection of a respective location; in response to receiving the user input indicative of the selection of the respective location, configuring the second device to be assigned to the respective location; displaying, on the display, a second configuration user interface, the second configuration user interface included in the plurality of different user interfaces and indicating that the second device is outputting an audio tone signal indicative of a pairing handshake with the electronic device; while displaying the second configuration user interface on the display, detecting the audio tone signal indicative of the pairing handshake; and in response to detecting the audio tone signal, proceeding to a next step in the initial configuration process for configuring the second device.

24. The non-transitory computer-readable storage medium of claim 23, wherein the audio tone signal is generated by the second device.

25. The non-transitory computer-readable storage medium of claim 23, wherein the audio tone signal comprises an audio passcode. the audio tone signal is detected when the electronic device is physically located within a threshold physical proximity of the second device.

26. The non-transitory computer-readable storage medium of claim 23, wherein, 27. The non-transitory computer-readable storage medium of claim 23, wherein the one or more programs further include instructions for: after displaying a user interface indicating that the initial configuration process is complete: displaying, on the display, a configuration user interface, the configuration user interface including a stereo pair affordance; receiving input corresponding to a selection of the stereo pair affordance; and in response to receiving input corresponding to a selection of the stereo pair affordance, initiating a process to pair a third device with the second device.

28. The non-transitory computer-readable storage medium of claim 23, wherein the one or more programs further include instructions for: receiving user input representing a positive request to create a device group; in response to receiving the user input representing the positive request to create the device group, displaying, on the display, a configuration user interface, the configuration user interface including a prompt to select an audio output channel to associate with the second device; receiving user input representing a selection of a first audio output channel; and ​ ​ in response to receiving the user input indicative of selection of the first audio output channel, associating the second device with the first audio output channel.

29. The non-transitory computer-readable storage medium of claim 23, wherein, The prompt to select a location includes a plurality of predetermined location identifiers.

30. The non-transitory computer-readable storage medium of claim 29, wherein, The plurality of predetermined location identifiers includes a user-defined location identifier.

31. The non-transitory computer-readable storage medium of claim 29, wherein, The prompt to select a location includes a field for creating a new location identifier, and wherein the one or more programs further include instructions for: receiving user input associated with the field for creating the new location identifier, wherein the user input includes a first location identifier; creating the first location identifier based on the received user input associated with the field for creating the new location identifier; and associating the second device with the first location identifier.

32. The non-transitory computer-readable storage medium of claim 23, wherein the one or more programs further include instructions for: after initiating the initial configuration process, receiving data regarding output of an operation being performed by the second device; and in response to receiving the data regarding output of an operation being performed by the second device, displaying a respective user interface.

33. The non-transitory computer-readable storage medium of claim 23, wherein the one or more programs further include instructions for: after the initial configuration process, displaying a configuration user interface associated with a voice assistant, wherein the configuration user interface associated with the voice assistant includes a prompt to provide user voice input that invokes one or more functions. ​

Images