Method and interface for home media control

Abstract

The invention relates to a method and interface for home media control. The present disclosure relates generally to interfaces and techniques for media playback on one or more devices. According to some embodiments, an electronic device includes a display, one or more processors, and a memory. The electronic device receives a user input and, in response to receiving the user input, displays a multi-device interface on the display, the multi-device interface comprising: one or more indicators associated with a plurality of available playback devices and media playback conditions of the plurality of available playback devices, the plurality of available playback devices are connected to the device and are available to initiate playback of media from the device.

Description

[0001] This application is a divisional application of the invention patent application with the application date of May 16, 2018, application number 202010728844.3, and name “Method and interface for home media control”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to the following patent applications: U.S. Provisional Patent Application No. 62 / 507,202, filed May 16, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; U.S. Provisional Patent Application No. 62 / 507,208, filed May 16, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No. PA 2017 70392, filed May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No. PA 201770401, filed May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No. PA 2017 70402, filed on May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No. PA 2017 70403, filed on May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No. PA 2017 70404, filed on May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No. PA 2017 70406, filed on May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No. PA 2017 70408, filed on May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL” 201770408; Danish Patent Application No. PA 2017 70409, filed on May 29, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; Danish Patent Application No.PA2017 70410; U.S. Provisional Patent Application No. 62 / 514,932, filed on June 4, 2017, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; U.S. Provisional Patent Application No. 62 / 622,122, filed on January 25, 2018, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”; and Danish Patent Application No. PA 201870060, filed on January 29, 2018, entitled “METHODS AND INTERFACES FOR HOME MEDIA CONTROL”, the contents of each of which are hereby incorporated by reference in their entirety.

[0004] This application is also related to U.S. patent application Ser. No. 14 / 300,120, filed on June 9, 2014, entitled “MULTI-CHANNEL LOUDSPEAKERMATCHING VARIABLE DIRECTIVITY,” the contents of which are hereby incorporated by reference in their entirety and are included in the Appendix. This application is also related to U.S. patent application Ser. No. 14 / 732,711, filed on June 6, 2015, entitled “MULTI-MICROPHONE SPEECH RECOGNITION SYSTEMS AND RELATED TECHNIQUES,” the contents of which are hereby incorporated by reference in their entirety and are included in the Appendix. This application is also related to U.S. patent application Ser. No. 14 / 732,715, filed on June 6, 2015, entitled “MULTI-MICROPHONE SPEECH RECOGNITION TECHNIQUES AND RELATED SYSTEMS.” This application is also related to U.S. patent application No. 14 / 869,760, filed on September 29, 2015, entitled “MODAL BASED ARCHITECTURE FOR CONTROLLING THE DIRECTIVITY OF LOUDSPEAKER ARRAYS,” the contents of which are hereby incorporated by reference in their entirety and are included in the Appendix. This application is also related to U.S. patent application No. 15 / 266,956, filed on September 15, 2016, entitled “INTELLIGENT AUTOMATED ASSISTANT FOR MEDIA EXPLORATION.” This application is also related to U.S. patent application No. 15 / 268,338, filed on September 16, 2016, entitled “INTELLIGENT DEVICE ARBITRATION AND CONTROL,” the contents of which are hereby incorporated by reference in their entirety and are included in the Appendix. This application is also related to U.S. patent application No. 15 / 272,214, filed on September 21, 2016, entitled “INTELLIGENT LIST READING.” This application is also related to U.S. patent application No. 15 / 275,077, filed on September 23, 2016, entitled “PRESSURE GRADIENT MICROPHONE FOR MEASURING AN ACOUSTIC CHARACTERISTIC OF A LOUDSPEAKER.”This application is also related to U.S. patent application No. 15 / 385,606, filed on December 20, 2016, entitled “Intelligent Automated Assistant,” the contents of which are hereby incorporated by reference in their entirety and are included in the Appendix. This application is also related to U.S. patent application No. 15 / 504,312, filed on August 18, 2014, entitled “A Rotationally Symmetric Speaker Array.” This application is also related to U.S. patent application No. 15 / 514,455, filed on September 25, 2015, entitled “Method to Determine Terminal Loudspeaker Change of Placement.” This application is also related to U.S. patent application No. 15 / 583,949, filed on May 1, 2017, entitled “A Rotationally Symmetric Speaker Array.” This application is also related to U.S. patent application No. 15 / 593,887, filed on May 12, 2017, entitled “SPATIAL AUDIORENDERING FOR BEAMFORMING LOUDSPEAKER ARRAY,” the contents of which are hereby incorporated by reference in their entirety and are included in the Appendix. This application is also related to U.S. patent application No. 62 / 004,111, filed on May 28, 2014, entitled “MULTI-CHANNEL LOUDSPEAKER MATCHING USING VARIABLE DIRECTIVITY.” This application is also related to U.S. patent application No. 62 / 057,989, filed on September 30, 2014, entitled “MODALBASED ARCHITECTURE FOR CONTROLLING THE DIRECTIVITY OF LOUDSPEAKER ARRAYS.” This application is also related to U.S. Patent Application No. 62 / 057,999, entitled “METHOD TO DETERMINE LOUDSPEAKER CHANGE OF PLACEMENT,” filed on September 30, 2014. This application is also related to U.S. Patent Application No. 62 / 346,236, entitled “INTELLIGENT LIST READING,” filed on June 6, 2016. This application is also related to U.S. Patent Application No. 62 / 347,480, entitled “INTELLIGENT AUTOMATED ASSISTANT FOR MEDIA EXPLORATION,” filed on June 8, 2016.This application is also related to U.S. patent application No. 62 / 348,896, entitled “Intelligent Device Arbitration and Control,” filed on June 11, 2016. This application is also related to U.S. patent application No. 62 / 399,165, entitled “Audio Driver and Power Supply Unit Architecture,” filed on September 23, 2016. This application is also related to U.S. patent application No. 62 / 399,229, entitled “Multi-Lobe Magnet for Subwoofer,” filed on September 23, 2016. This application is also related to U.S. patent application No. 62 / 399,232, entitled “Intelligent Automatic Assistant,” filed on September 23, 2016. This application is also related to U.S. patent application No. 62 / 399,262, filed on September 23, 2016, entitled “FORWARD BACK VOLUME OF AIR FOR AUDIO DRIVER.” This application is also related to U.S. patent application No. 62 / 399,288, filed on September 23, 2016, entitled “UPPER HOUSING COMPONENT ARCHITECTURE.” This application is also related to U.S. patent application No. 62 / 399,293, filed on September 23, 2016, entitled “AUDIO DRIVER EXIT GEOMETRY.” This application is also related to U.S. patent application No. 62 / 402,836, filed on September 30, 2016, entitled “SPATIAL AUDIO RENDERING STRATEGIES FOR BEAMFORMING LOUD SPEAKER ARRAY.” This application is also related to U.S. patent application No. 62 / 506,912, filed May 16, 2017, entitled “MANAGING MEDIA COLLECTIONS USING DIRECTED ACYCLIC GRAPHS.” This application is also related to U.S. patent application No. 62 / 507,007, filed May 16, 2017, entitled “SPEAKER.” This application is also related to U.S. patent application No. 62 / 507,100, filed May 16, 2017, entitled “TRANSFERRING PLAYBACK QUEUES BETWEEN EVICES.”This application is also related to U.S. patent application No. 62 / 507,151, filed May 16, 2017, entitled “FAR-FIELD EXTENSION FOR DIGITAL ASSISTANT SERVICES,” the contents of which are hereby incorporated by reference in their entirety and included in the Appendix. This application is also related to U.S. patent application No. 62 / 507,056, filed May 16, 2017, entitled “PROVIDING AN AUDITORY-BASED INTERFACE OF A DIGITAL ASSISTANT.” This application is also related to U.S. patent application No. 62 / 506,871, filed May 16, 2017, entitled “METHODS AND SYSTEMS FOR PHONETIC MATCHING IN DIGITAL ASSISTANT SERVICE.” This application also relates to U.S. patent application No. 62 / 507,042, filed on May 16, 2017, entitled “DETECTING ATRIGGER OF A DIGITAL ASSISTANT.” This application also relates to U.S. patent application No. 62 / 506,981, filed on May 16, 2017, entitled “INTELLIGENT AUTOMATED ASSISTANT FOR MEDIA EXPLORATION.” This application also relates to a U.S. patent application entitled “AUDIO BLUR” (Agent Docket No. P34400US1), filed in preparation for filing. This application also relates to a U.S. patent application entitled “ONLINE PROCESSING METHODS FOR MULTI-CHANNEL DEREVERBERATION” (Agent Docket No. P32630US1), filed in preparation for filing. This application also relates to a U.S. patent application entitled “AUTOMATIC LOUDSPEAKER ORIENTATION ESTIMATION” (Agent Docket No. P34635US1), filed in preparation for filing. This application is also related to a filed U.S. patent application entitled "FOURIER TRANSFORM BASED METHOD FOR CREATING SMOOTH DIRECTIVITY TRANSITIONS IN BEAMFORMING LOUDSPEAKERS" (Attorney Docket No. P34719US1).This application is also related to a U.S. patent application entitled “ACOUSTIC ECHO-CANCELLATION USING AN INTERNAL MICROPHONE” (Agent Docket No. P34811US1), which is being filed. This application is also related to a U.S. patent application entitled “TECHNIQUESFOR VERIFYING USER INTENT AND SECURELY CONFIGURING COMPUTER DEVICES” (Agent Docket No. P34845USP1), which is being filed. This application is also related to a U.S. patent application entitled “SHARING CONCEPTS” (Agent Docket No. P34762USP1), which is being filed. The contents of each of the above-listed related applications are hereby incorporated by reference in their entirety. Technical Field

[0005] The present disclosure relates generally to computer user interfaces and, more particularly, to techniques for managing media playback devices. Background Art

[0006] The number of electronic devices, particularly smart devices, in users' homes continues to grow. These devices are becoming increasingly interconnected, feature-rich, and capable of performing more complex tasks. As a result, users increasingly expect these devices to have well-designed user interfaces. Summary of the Invention

[0007] However, some techniques for managing media playback devices using electronic devices are often cumbersome and inefficient. For example, some existing techniques use complex and time-consuming user interfaces that may include multiple button presses or keystrokes. These techniques require more time than necessary, resulting in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0008] Therefore, the present technology provides electronic devices with faster and more efficient methods and interfaces for managing media playback devices. Such methods and interfaces optionally supplement or replace other methods for managing media playback devices. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges. Such methods and interfaces can reduce the number of unnecessary, irrelevant, or repetitive user inputs. In addition, such methods and interfaces can reduce the power usage of the electronic device's display or speakers.

[0009] According to some embodiments, a method is performed at a device having a display. The method includes: receiving user input; and in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback statuses of the plurality of available playback devices, the plurality of available playback devices being connected to the device and being usable to initiate playback of media from the device.

[0010] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: receiving user input; and in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback statuses of the plurality of available playback devices, the plurality of available playback devices being connected to the device and being usable to initiate playback of media from the device.

[0011] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving user input; and in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with multiple available playback devices and media playback status of the multiple available playback devices, the multiple available playback devices being connected to the device and usable to initiate playback of media from the device.

[0012] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for receiving user input; and means for displaying a multi-device interface on the display in response to receiving the user input, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback statuses of the plurality of available playback devices, the plurality of available playback devices being connected to the device and being usable to initiate playback of media from the device.

[0013] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: receiving user input; and in response to receiving the user input, displaying a multi-device interface on the display, the multi-device interface including: one or more indicators associated with a plurality of available playback devices and media playback statuses of the plurality of available playback devices, the plurality of available playback devices being connected to the device and being usable to initiate playback of media from the device.

[0014] According to some embodiments, a method is performed at a device having a display. The method includes: displaying a first current media interface for the device on the display, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection affordance; receiving a first user input indicating a selection of the playback device selection affordance; in response to receiving the first user input, displaying one or more affordances associated with a plurality of available playback devices connected to the device on the display; receiving a second user input indicating a selection of the one or more affordances associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0015] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: displaying a first current media interface for the device on the display, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enable representation; receiving a first user input indicating a selection of the playback device selection enable representation; in response to receiving the first user input, displaying one or more enable representations associated with a plurality of available playback devices connected to the device on the display; receiving a second user input indicating a selection of the one or more enable representations associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0016] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a first current media interface for the device on the display, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enable representation; receiving a first user input indicating a selection of the playback device selection enable representation; in response to receiving the first user input, displaying one or more enable representations associated with a plurality of available playback devices connected to the device on the display; receiving a second user input indicating a selection of the one or more enable representations associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0017] According to some embodiments, an electronic device is described. The electronic device includes: a display; a device for displaying a first current media interface for the device on the display, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enable representation; a device for receiving a first user input representing a selection of the playback device selection enable representation; a device for displaying one or more enable representations associated with a plurality of available playback devices connected to the device on the display in response to receiving the first user input; a device for receiving a second user input representing a selection of the one or more enable representations associated with the plurality of available playback devices; and a device for configuring the device to initiate playback on the plurality of available playback devices in response to receiving the second user input.

[0018] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: displaying a first current media interface for the device on the display, wherein the first current media interface includes: a representation of a first media item available for playback on the device and a playback device selection enable representation; receiving a first user input indicating a selection of the playback device selection enable representation; in response to receiving the first user input, displaying one or more enable representations associated with a plurality of available playback devices connected to the device on the display; receiving a second user input indicating a selection of the one or more enable representations associated with the plurality of available playback devices; and in response to receiving the second user input, configuring the device to initiate playback on the plurality of available playback devices.

[0019] According to some embodiments, a method is performed at a first device. The method includes: detecting an indication that physical proximity between the first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on detecting the indication, wherein the audio tone signal is out-of-band with respect to the communication link; determining whether a response audio tone signal has been received from the second device after outputting the audio tone signal; proceeding to a next step in a configuration process based on determining that a response audio tone signal has been received from the second device; and forgoing proceeding to the next step in the configuration process based on determining that a response audio tone signal has not been received from the second device.

[0020] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting an indication that physical proximity between a first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on detecting the indication, wherein the audio tone signal is out-of-band for the communication link; determining whether a response audio tone signal has been received from the second device after outputting the audio tone signal; proceeding to a next step in a configuration process based on determining that a response audio tone signal has been received from the second device; and abstaining from proceeding to the next step in the configuration process based on determining that a response audio tone signal has not been received from the second device.

[0021] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting an indication that physical proximity between a first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on detecting the indication, wherein the audio tone signal is out-of-band for the communication link; determining whether a response audio tone signal has been received from the second device after outputting the audio tone signal; proceeding to the next step in a configuration process based on determining that a response audio tone signal has been received from the second device; and abstaining from proceeding to the next step in the configuration process based on determining that a response audio tone signal has not been received from the second device.

[0022] According to some embodiments, an electronic device is described. The electronic device includes: a device for detecting an indication that physical proximity between a first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; a device for audibly outputting an audio tone signal based on detecting the indication, wherein the audio tone signal is out-of-band for the communication link; a device for determining whether a response audio tone signal has been received from the second device after outputting the audio tone signal; a device for proceeding to a next step in a configuration process based on determining that a response audio tone signal has been received from the second device; and a device for abstaining from proceeding to the next step in the configuration process based on determining that a response audio tone signal has not been received from the second device.

[0023] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting an indication that physical proximity between a first device and a second device satisfies a proximity condition, wherein the first device is connected to the second device via a communication link; audibly outputting an audio tone signal based on detecting the indication, wherein the audio tone signal is out-of-band for the communication link; determining whether a response audio tone signal has been received from the second device after outputting the audio tone signal; proceeding to a next step in a configuration process based on determining that a response audio tone signal has been received from the second device; and abstaining from proceeding to the next step in the configuration process based on determining that a response audio tone signal has not been received from the second device.

[0024] According to some embodiments, a method is performed at a first device having a display. The method includes: detecting an indication that physical proximity between the first device and the second device satisfies a proximity condition when connected to a second device; and displaying a media information interface on the display based on detecting that the physical proximity satisfies the proximity condition.

[0025] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: when connected to a second device, detecting an indication that physical proximity between the first device and the second device satisfies a proximity condition; and based on detecting that the physical proximity satisfies the proximity condition, displaying a media information interface on the display.

[0026] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: when connected to a second device, detecting an indication that physical proximity between the electronic device and the second device satisfies a proximity condition; and based on detecting that the physical proximity satisfies the proximity condition, displaying a media information interface on the display.

[0027] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for detecting an indication that physical proximity between the electronic device and the second device satisfies a proximity condition when connected to the second device; and means for displaying a media information interface on the display based on detecting that the physical proximity satisfies the proximity condition.

[0028] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: when connected to a second device, detecting an indication that physical proximity between the first device and the second device satisfies a proximity condition; and based on detecting that the physical proximity satisfies the proximity condition, displaying a media information interface on the display.

[0029] According to some embodiments, a method is performed at a device having a touch-sensitive surface. The method includes: detecting a touch on the touch-sensitive surface; in response to detecting the touch on the touch-sensitive surface, determining whether the device is currently playing media; based on determining that the device is currently playing media: determining whether the touch is within a first area of ​​the touch-sensitive surface or within a second area of ​​the touch-sensitive surface, wherein the first area and the second area do not overlap; based on determining that the touch is within the first area of ​​the touch-sensitive surface, stopping playback of the media; and based on determining that the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the media currently playing; and based on determining that the device is not currently playing media, initiating playback of the media.

[0030] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a touch-sensitive surface, the one or more programs including instructions for performing the following operations: detecting a touch on the touch-sensitive surface; in response to detecting the touch on the touch-sensitive surface, determining whether the device is currently playing media; based on determining that the device is currently playing media: determining whether the location of the touch is within a first area of ​​the touch-sensitive surface or within a second area of ​​the touch-sensitive surface, wherein the first area and the second area do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the media currently playing; and based on determining that the device is not currently playing media, initiating playback of the media.

[0031] According to some embodiments, an electronic device is described. The electronic device includes: a touch-sensitive surface; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting a touch on the touch-sensitive surface; in response to detecting the touch on the touch-sensitive surface, determining whether the device is currently playing media; based on determining that the device is currently playing media: determining whether the location of the touch is within a first area of ​​the touch-sensitive surface or within a second area of ​​the touch-sensitive surface, wherein the first area and the second area do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the media currently playing; and based on determining that the device is not currently playing media, initiating playback of the media.

[0032] According to some embodiments, an electronic device is described. The electronic device includes: a touch-sensitive surface; a device for detecting a touch on the touch-sensitive surface; a device for determining whether the device is currently playing media in response to detecting the touch on the touch-sensitive surface; a device for performing the following operations: based on determining that the device is currently playing media: determining whether the location of the touch is within a first area of ​​the touch-sensitive surface or within a second area of ​​the touch-sensitive surface, wherein the first area and the second area do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the media currently playing; and a device for initiating playback of the media based on determining that the device is not currently playing media.

[0033] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a touch-sensitive surface, the one or more programs including instructions for performing the following operations: detecting a touch on the touch-sensitive surface; in response to detecting the touch on the touch-sensitive surface, determining whether the device is currently playing media; based on determining that the device is currently playing media: determining whether the location of the touch is within a first area of ​​the touch-sensitive surface or within a second area of ​​the touch-sensitive surface, wherein the first area and the second area do not overlap; based on determining that the location of the touch is within the first area of ​​the touch-sensitive surface, stopping playback of the media; and based on determining that the location of the touch is within the second area of ​​the touch-sensitive surface, adjusting the volume of the media currently playing; and based on determining that the device is not currently playing media, initiating playback of the media.

[0034] According to some embodiments, a method is performed at a first device. The method includes: receiving a first request to queue a first media item for playback on a group of devices, wherein the group of devices includes the first device; adding the first media item to a playback queue; receiving a second request to queue a second media item for playback on the group of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the group of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the group of devices.

[0035] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a first request to queue a first media item for playback on a group of devices, wherein the group of devices includes the first device; adding the first media item to a playback queue; receiving a second request to queue a second media item for playback on the group of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the group of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the group of devices.

[0036] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a first request to queue a first media item for playback on a group of devices, wherein the group of devices includes the electronic device; adding the first media item to a playback queue; receiving a second request to queue a second media item for playback on the group of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the group of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the group of devices.

[0037] According to some embodiments, an electronic device is described. The electronic device includes: a device for receiving a first request to queue a first media item for playback on a group of devices, wherein the group of devices includes the electronic device; a device for adding the first media item to a playback queue; a device for receiving a second request to queue a second media item for playback on the group of devices; a device for adding the second media item to the playback queue; a device for accessing the first media item from a first media streaming service; a device for initiating playback of the first media item on the group of devices; a device for accessing the second media item from a second media streaming service different from the first media streaming service; and a device for initiating playback of the second media item on the group of devices.

[0038] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a first request to queue a first media item for playback on a group of devices, wherein the group of devices includes the first device; adding the first media item to a playback queue; receiving a second request to queue a second media item for playback on the group of devices; adding the second media item to the playback queue; accessing the first media item from a first media streaming service; initiating playback of the first media item on the group of devices; accessing the second media item from a second media streaming service different from the first media streaming service; and initiating playback of the second media item on the group of devices.

[0039] According to some embodiments, a method is performed at a first device. The method includes: receiving a first request from a second device associated with a first user to queue a first media item for playback on a group of devices, wherein the group of devices includes the first device; adding the first media item to a playback queue; receiving a second request from a third device associated with a second user to queue a second media item for playback on the group of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the group of devices; and initiating playback of the second media item on the group of devices.

[0040] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a first request from a second device associated with a first user to queue a first media item for playback on a group of devices, wherein the group of devices includes the first device; adding the first media item to a playback queue; receiving a second request from a third device associated with a second user to queue a second media item for playback on the group of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the group of devices; and initiating playback of the second media item on the group of devices.

[0041] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a first request from a second device associated with a first user to queue a first media item for playback on a group of devices, wherein the group of devices includes the electronic device; adding the first media item to a playback queue; receiving a second request from a third device associated with a second user to queue a second media item for playback on the group of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the group of devices; and initiating playback of the second media item on the group of devices.

[0042] According to some embodiments, an electronic device is described. The electronic device includes: a device for receiving a first request from a second device associated with a first user to queue a first media item for playback on a group of devices, wherein the group of devices includes the electronic device; a device for adding the first media item to a playback queue; a device for receiving a second request from a third device associated with a second user to queue a second media item for playback on the group of devices, wherein the second user is different from the first user; a device for adding the second media item to the playback queue; a device for initiating playback of the first media item on the group of devices; and a device for initiating playback of the second media item on the group of devices.

[0043] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a first request from a second device associated with a first user to queue a first media item for playback on a group of devices, wherein the group of devices includes the first device; adding the first media item to a playback queue; receiving a second request from a third device associated with a second user to queue a second media item for playback on the group of devices, wherein the second user is different from the first user; adding the second media item to the playback queue; initiating playback of the first media item on the group of devices; and initiating playback of the second media item on the group of devices.

[0044] According to some embodiments, a method is performed at a first device having a display. The method includes: detecting an audio tone signal indicative of a pairing handshake with a second device, wherein the first device is connected to the second device via a communication link and wherein the audio tone signal is out-of-band with respect to the communication link; initiating a configuration process on the first device for configuring the second device based on detecting the audio tone signal; during the configuration process, displaying a configuration user interface on the display including a prompt for selecting a location; receiving a user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location.

[0045] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: detecting an audio tone signal indicating a pairing handshake with a second device, wherein the first device is connected to the second device via a communication link and wherein the audio tone signal is out-of-band for the communication link; initiating a configuration process on the first device for configuring the second device based on detecting the audio tone signal; during the configuration process, displaying a configuration user interface on the display including a prompt for selecting a location; receiving a user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location.

[0046] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting an audio tone signal indicating a pairing handshake with a second device, wherein the electronic device is connected to the second device via a communication link and wherein the audio tone signal is out-of-band for the communication link; initiating a configuration process on the electronic device for configuring the second device based on detecting the audio tone signal; during the configuration process, displaying a configuration user interface on the display including a prompt for selecting a location; receiving a user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location.

[0047] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for detecting an audio tone signal indicative of a pairing handshake with a second device, wherein the electronic device is connected to the second device via a communication link and wherein the audio tone signal is out-of-band for the communication link; means for initiating a configuration process on the electronic device for configuring the second device based on detecting the audio tone signal; means for displaying a configuration user interface on the display including a prompt for selecting a location during the configuration process; means for receiving a user input selection of a location; and means for configuring the second device to be associated with the location in response to receiving the user input.

[0048] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: detecting an audio tone signal indicating a pairing handshake with a second device, wherein the first device is connected to the second device via a communication link and wherein the audio tone signal is out-of-band with respect to the communication link; initiating a configuration process on the first device for configuring the second device based on detecting the audio tone signal; during the configuration process, displaying a configuration user interface on the display including a prompt for selecting a location; receiving a user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location.

[0049] According to some embodiments, a method is performed at a device. The method includes: detecting a user-input voice command from a voice command source while outputting an audible signal from the device; determining a spatial location of the voice command source relative to the device in response to detecting the user-input voice command; and adjusting output of the audible signal based on the spatial location of the voice command source while continuing to output the audible signal.

[0050] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting a user-input voice command from a voice command source while outputting an audible signal from the device; determining a spatial location of the voice command source relative to the device in response to detecting the user-input voice command; and adjusting output of the audible signal based on the spatial location of the voice command source while continuing to output the audible signal.

[0051] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting a user-input voice command from a voice command source while outputting an audible signal from the device; determining a spatial position of the voice command source relative to the device in response to detecting the user-input voice command; and adjusting the output of the audible signal based on the spatial position of the voice command source while continuing to output the audible signal.

[0052] According to some embodiments, an electronic device is described. The electronic device includes: means for detecting a user-input voice command from a voice command source while outputting an audible signal from the device; means for determining a spatial location of the voice command source relative to the device in response to detecting the user-input voice command; and means for adjusting output of the audible signal based on the spatial location of the voice command source while continuing to output the audible signal.

[0053] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: detecting a user-input voice command from a voice command source while outputting an audible signal from the device; determining a spatial location of the voice command source relative to the device in response to detecting the user-input voice command; and adjusting output of the audible signal based on the spatial location of the voice command source while continuing to output the audible signal.

[0054] According to some embodiments, a method is performed at a first device associated with a first user account. The method includes: detecting that a second device associated with a second user account is connected to the first device, wherein the first user account is associated with a first media preference profile; wherein the second user account is associated with a second media preference profile that is different from the first media preference profile; and initiating playback of media based on the first media preference profile and the second media preference profile.

[0055] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device associated with a first user account, wherein the first user account is associated with a first media preference profile, the one or more programs including instructions for performing the following operations: detecting that a second device associated with a second user account is connected to the first device, wherein the second user account is associated with a second media preference profile that is different from the first media preference profile; and initiating playback of media based on the first media preference profile and the second media preference profile.

[0056] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting that a second device associated with a second user account is connected to the electronic device, wherein the electronic device is associated with a first user account, wherein the first user account is associated with a first media preference profile; wherein the second user account is associated with a second media preference profile that is different from the first media preference profile; and initiating playback of media based on the first media preference profile and the second media preference profile.

[0057] According to some embodiments, an electronic device is described. The electronic device includes: means for detecting that a second device associated with a second user account is connected to the electronic device, wherein the electronic device is associated with a first user account, wherein the first user account is associated with a first media preference profile; wherein the second user account is associated with a second media preference profile that is different from the first media preference profile; and means for initiating playback of media based on the first media preference profile and the second media preference profile.

[0058] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device associated with a first user account, wherein the first user account is associated with a first media preference profile, the one or more programs including instructions for performing the following operations: detecting that a second device associated with a second user account is connected to the first device, wherein the second user account is associated with a second media preference profile that is different from the first media preference profile; and initiating playback of media based on the first media preference profile and the second media preference profile.

[0059] According to some embodiments, a method is performed at a first device. The method includes: receiving a request from a third device to initiate media playback on at least one device in a group of devices, wherein the group of devices includes the first device and the second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the first device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and in response to determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to a second device; and in response to receiving the permission from the second device, initiating media playback on the at least one device in the group of devices.

[0060] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving a request from a third device to initiate media playback on at least one device in a group of devices, wherein the group of devices includes a first device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the first device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and in response to determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to the second device; and in response to receiving the permission from the second device, initiating media playback on the at least one device in the group of devices.

[0061] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a request from a third device to initiate media playback on at least one device in a group of devices, wherein the group of devices includes the electronic device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the electronic device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and in response to determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to the second device; and in response to receiving the permission from the second device, initiating media playback on the at least one device in the group of devices.

[0062] According to some embodiments, an electronic device is described. The electronic device includes: a device for receiving a request from a third device to initiate media playback on at least one device in a group of devices, wherein the group of devices includes the electronic device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the electronic device; a device for performing the following operations: in response to receiving the request: in response to determining that the third device has permission to initiate media playback on the at least one device in the group of devices, initiating media playback on the group of devices; and in response to determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to the second device; and in response to receiving the permission from the second device, initiating media playback on the at least one device in the group of devices.

[0063] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving a request from a third device to initiate media playback on at least one device in a group of devices, wherein the group of devices includes a first device and a second device, wherein the third device is not a member of the group of devices, and wherein the third device is connected to the first device; in response to receiving the request: initiating media playback on the group of devices based on determining that the third device has permission to initiate media playback on the at least one device in the group of devices; and in response to determining that the third device does not have permission to initiate media playback on the at least one device in the group of devices: transmitting a request for permission to the second device; and in response to receiving the permission from the second device, initiating media playback on the at least one device in the group of devices.

[0064] According to some embodiments, a method is performed at a first device associated with a user account. The method includes: receiving user input representing a request to access data from the user account; in response to receiving the user input, determining whether the physical location of the first device and a second device associated with the user account satisfies a permission condition; based on determining that the physical location satisfies the permission condition, providing the data from the user account through the first device; and based on determining that the physical location does not satisfy the permission condition, forgoing providing the data from the user account through the first device.

[0065] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first electronic device associated with a user account, and the one or more programs include instructions for performing the following operations: receiving user input representing a request for accessing data from the user account; in response to receiving the user input, determining whether the physical location of the electronic device and a second device associated with the user account satisfies a permission condition; based on determining that the physical location satisfies the permission condition, providing the data from the user account through the electronic device; and based on determining that the physical location does not satisfy the permission condition, forgoing providing the data from the user account through the electronic device.

[0066] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving user input representing a request for accessing data from a user account, wherein the electronic device is associated with the user account; in response to receiving the user input, determining whether the physical location of the electronic device and a second device associated with the user account satisfies a permission condition; based on determining that the physical location satisfies the permission condition, providing the data from the user account through the electronic device; and based on determining that the physical location does not satisfy the permission condition, forgoing providing the data from the user account through the electronic device.

[0067] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving user input representing a request to access data from a user account, wherein the electronic device is associated with the user account; means for determining whether the physical location of the electronic device and a second device associated with the user account satisfies a permission condition in response to receiving the user input; means for providing the data from the user account via the electronic device based on determining that the physical location satisfies the permission condition; and means for forgoing providing the data from the user account via the electronic device based on determining that the physical location does not satisfy the permission condition.

[0068] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first electronic device associated with a user account, and the one or more programs include instructions for performing the following operations: receiving user input representing a request for accessing data from the user account; in response to receiving the user input, determining whether the physical location of the electronic device and a second device associated with the user account satisfies a permission condition; based on determining that the physical location satisfies the permission condition, providing the data from the user account through the electronic device; and based on determining that the physical location does not satisfy the permission condition, forgoing providing the data from the user account through the electronic device.

[0069] According to some embodiments, a method is performed at a first device. The method includes: receiving a command to initiate playback of media on a group of devices including a first device and a second device; in response to receiving the command, initiating synchronized playback of the media on the group of devices, wherein the second device is designated as a master device configured to coordinate playback of the media on the group of devices, and wherein the first device is configured as a designated successor to the master device.

[0070] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a command to initiate playback of media on a group of devices including a first device and a second device; in response to receiving the command, initiating synchronized playback of the media on the group of devices, wherein the second device is designated as a master device configured to coordinate playback of the media on the group of devices, and wherein the first device is configured as a designated successor to the master device.

[0071] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a command to initiate playback of media on a group of devices including the electronic device and a second device; in response to receiving the command, initiating synchronized playback of the media on the group of devices, wherein the second device is designated as a master device configured to coordinate playback of the media on the group of devices, and wherein the electronic device is configured as a designated successor to the master device.

[0072] According to some embodiments, an electronic device is described. The electronic device includes: means for receiving a command to initiate playback of media on a group of devices including the electronic device and a second device; means for initiating synchronized playback of the media on the group of devices in response to receiving the command, wherein the second device is designated as a master device configured to coordinate playback of the media on the group of devices, and wherein the electronic device is configured as a designated successor to the master device.

[0073] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a command to initiate playback of media on a group of devices including a first device and a second device; in response to receiving the command, initiating synchronized playback of the media on the group of devices, wherein the second device is designated as a master device configured to coordinate playback of the media on the group of devices, and wherein the first device is configured as a designated successor to the master device.

[0074] According to some embodiments, a method is performed at a first device associated with a first user account in a group of devices. The method includes: receiving an audible output triggering event; determining that a first user associated with the first user account is within an audible output range of a device in the group of devices; outputting an audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output triggering event; and forgoing outputting the audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output triggering event.

[0075] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device in a group of devices associated with a first user account, the one or more programs including instructions for performing the following operations: receiving an audible output triggering event; determining that a first user associated with the first user account is within an audible output range of a device in the group of devices; outputting an audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output triggering event; and forgoing outputting the audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output triggering event.

[0076] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving an audible output triggering event; determining that a first user associated with a first user account is within an audible output range of a device associated with the first user account in a group of devices, wherein the electronic device is the first device in the group of devices; outputting an audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output triggering event; and forgoing outputting the audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output triggering event.

[0077] According to some embodiments, an electronic device is described. The electronic device includes: a device for receiving an audible output triggering event; a device for determining that a first user associated with a first user account is within an audible output range of a device associated with the first user account in a group of devices, wherein the electronic device is the first device in the group of devices; a device for outputting an audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output triggering event; and a device for forgoing outputting an audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output triggering event.

[0078] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device in a group of devices associated with a first user account, the one or more programs including instructions for performing the following operations: receiving an audible output triggering event; determining that a first user associated with the first user account is within an audible output range of a device in the group of devices; outputting an audible output based on determining that the first user is within the audible output range of the first device and based on the occurrence of the audible output triggering event; and forgoing outputting the audible output based on determining that the first user is within the audible output range of a second device and based on the occurrence of the audible output triggering event.

[0079] According to some embodiments, a method is performed at a first device. The method includes: receiving a user input voice command representing a request to output a signal representing media currently being played back by a second device connected to the first device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving information about the media currently being played back by the second device; and initiating output of at least one component of the signal representing the media currently being played back by the second device based on receiving the information about the media currently being played back by the second device and when the second device resumes playback of at least one component of the media.

[0080] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the first device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving information about the media currently being played back by the second device; and initiating output of at least one component of the signal representing the media currently being played back by the second device based on receiving the information about the media currently being played back by the second device and when the second device resumes playback of at least one component of the media.

[0081] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the electronic device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving information about the media currently being played back by the second device; and initiating output of at least one component of the signal representing the media currently being played back by the second device based on receiving the information about the media currently being played back by the second device and when the second device resumes playback of at least one component of the media.

[0082] According to some embodiments, an electronic device is described. The electronic device includes: a device for receiving a user input voice command representing a request to output a signal representing media currently being played back by a second device connected to the electronic device; a device for transmitting a request for information about the media currently being played back by the second device in response to receiving the user input; a device for receiving information about the media currently being played back by the second device; and a device for performing the following operations: based on receiving the information about the media currently being played back by the second device and when the second device resumes playback of at least one component of the media, initiating output of at least one component of the signal representing the media currently being played back by the second device.

[0083] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, and the one or more programs include instructions for performing the following operations: receiving a user input voice command representing a request for outputting a signal representing media currently being played back by a second device connected to the first device; in response to receiving the user input, transmitting a request for information about the media currently being played back by the second device; receiving information about the media currently being played back by the second device; and initiating output of at least one component of the signal representing the media currently being played back by the second device based on receiving the information about the media currently being played back by the second device and when the second device resumes playback of at least one component of the media.

[0084] According to some embodiments, a method is performed at a first device. The method includes: receiving a request to initiate media playback on a group of devices that does not include the first device; in response to receiving the request and while the first device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes the device configured to coordinate media playback, transferring a media playback coordination task from the first device to the device configured to coordinate media playback; and based on determining that the group of devices does not include the device configured to coordinate media playback, forgoing transferring the media playback coordination task.

[0085] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices that does not include the first device; in response to receiving the request and while the first device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes the device configured to coordinate media playback, transferring a media playback coordination task from the first device to the device configured to coordinate media playback; and based on determining that the group of devices does not include the device configured to coordinate media playback, abandoning the transfer of the media playback coordination task.

[0086] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a request for initiating media playback on a group of devices that does not include the electronic device; in response to receiving the request and while the electronic device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes the device configured to coordinate media playback, transferring a media playback coordination task from the electronic device to the device configured to coordinate media playback; and based on determining that the group of devices does not include the device configured to coordinate media playback, abandoning the transfer of the media playback coordination task.

[0087] According to some embodiments, an electronic device is described. The electronic device includes: a device for receiving a request to initiate media playback on a group of devices that does not include the electronic device; a device for determining, in response to receiving the request and while the electronic device is coordinating media playback, whether the group of devices includes a device configured to coordinate media playback; a device for transferring a media playback coordination task from the electronic device to the device configured to coordinate media playback based on determining that the group of devices includes the device configured to coordinate media playback; and a device for abandoning the transfer of the media playback coordination task based on determining that the group of devices does not include the device configured to coordinate media playback.

[0088] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices that does not include the first device; in response to receiving the request and while the first device is coordinating media playback, determining whether the group of devices includes a device configured to coordinate media playback; based on determining that the group of devices includes the device configured to coordinate media playback, transferring a media playback coordination task from the first device to the device configured to coordinate media playback; and based on determining that the group of devices does not include the device configured to coordinate media playback, abandoning the transfer of the media playback coordination task.

[0089] According to some embodiments, a method is performed at a device. The method includes: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transmitting a media playback coordination task from the device to the master device configured to coordinate media playback.

[0090] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transmitting a media playback coordination task from the device to the master device configured to coordinate media playback.

[0091] According to some embodiments, an electronic device is described. The electronic device includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transmitting a media playback coordination task from the device to the master device configured to coordinate media playback.

[0092] According to some embodiments, an electronic device is described. The electronic device includes: a device for receiving a request to initiate media playback on a group of devices connected to the device; a device for determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices in response to receiving the request and while the device is coordinating media playback; a device for selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and a device for transferring a media playback coordination task from the device to the master device configured to coordinate media playback.

[0093] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: receiving a request to initiate media playback on a group of devices connected to the device; in response to receiving the request and while the device is coordinating media playback, determining one or more device characteristics for each of a plurality of devices configured to coordinate media playback and connected to the group of devices; selecting a master device from the plurality of devices configured to coordinate media playback based at least in part on the one or more characteristics of each of the plurality of devices configured to coordinate media playback; and transmitting a media playback coordination task from the device to the master device configured to coordinate media playback.

[0094] According to some embodiments, a method is performed at a first device having a display. The method includes: establishing a communication link with a second device; subsequent to establishing the communication link with the second device, initiating a configuration process for configuring the second device; during the configuration process: displaying a configuration user interface on the display including a prompt for selecting a location; receiving user input selecting a location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicative of a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and based on detecting the audio tone signal, proceeding to a next step in the configuration process for configuring the second device.

[0095] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, wherein the electronic device is a first device, the one or more programs including instructions for performing the following operations: establishing a communication link with a second device; subsequent to establishing the communication link with the second device, initiating a configuration process for configuring the second device; during the configuration process: displaying a configuration user interface on the display including a prompt for selecting a location; receiving a user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicative of a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and based on detecting the audio tone signal, proceeding to a next step in the configuration process for configuring the second device.

[0096] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: establishing a communication link with a second device; subsequent to establishing the communication link with the second device, initiating a configuration process for configuring the second device; during the configuration process: displaying a configuration user interface on the display including a prompt for selecting a location; receiving a user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicative of a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and based on detecting the audio tone signal, proceeding to a next step in the configuration process for configuring the second device.

[0097] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for establishing a communication link with a second device; means for initiating a configuration process for configuring the second device after establishing the communication link with the second device; during the configuration process: means for displaying on the display a configuration user interface including a prompt for selecting a location; means for receiving a user input selection of a location; and means for configuring the second device to be associated with the location in response to receiving the user input; means for detecting an audio tone signal indicative of a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and means for proceeding to a next step in the configuration process for configuring the second device based on detecting the audio tone signal.

[0098] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: establishing a communication link with a second device; subsequent to establishing the communication link with the second device, initiating a configuration process for configuring the second device; during the configuration process: displaying a configuration user interface on the display including a prompt for selecting a location; receiving a user input selection of a location; and in response to receiving the user input, configuring the second device to be associated with the location; detecting an audio tone signal indicative of a pairing handshake with the second device, wherein the audio tone signal is out-of-band for the communication link; and based on detecting the audio tone signal, proceeding to a next step in the configuration process for configuring the second device.

[0099] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0100] Thus, a faster, more efficient method and interface is provided for devices to manage media playback devices, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can supplement or replace other methods for managing media playback devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0102] Figure 1Ais a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0103] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0104] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0105] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0106] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to some embodiments.

[0107] Figure 4B An exemplary user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0108] Figure 5A A personal electronic device according to some embodiments is shown.

[0109] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0110] Figure 5C-5D Example components of a personal electronic device with a touch-sensitive display and an intensity sensor are shown according to some embodiments.

[0111] Figure 5E-5H Exemplary components and user interfaces of a personal electronic device according to some embodiments are shown.

[0112] Figure 5I An electronic device according to some embodiments is shown.

[0113] Figure 5J is a block diagram illustrating an electronic device according to some embodiments.

[0114] Figures 6A-6AR Exemplary techniques and user interfaces for playing back media according to some embodiments are shown.

[0115] Figures 7A-7D is a flowchart illustrating a process for playing back media according to some embodiments.

[0116] Figures 8A-8P Exemplary techniques and user interfaces for playing back media according to some embodiments are shown.

[0117] Figures 9A-9C is a flowchart illustrating a process for playing back media according to some embodiments.

[0118] Figures 10A-10O Exemplary techniques and user interfaces according to some embodiments are shown.

[0119] Figures 11A-11D is a flowchart illustrating a process for displaying content according to some embodiments.

[0120] Figures 12A-12AM Exemplary techniques and user interfaces for viewing media information according to some embodiments are shown.

[0121] Figures 13A-13F is a flowchart illustrating a process for viewing media information according to some embodiments.

[0122] Figures 14A-14T Exemplary techniques and user interfaces for controlling electronic devices according to some embodiments are shown.

[0123] Figures 15A-15C is a flowchart illustrating a process for controlling an electronic device according to some embodiments.

[0124] Figures 16A-16P Exemplary techniques and user interfaces for managing a playback queue are shown according to some embodiments.

[0125] Figures 17A-17C is a flowchart illustrating a process for managing a playback queue according to some embodiments.

[0126] Figures 18A-18C is a flowchart illustrating a process for managing a playback queue according to some embodiments.

[0127] Figures 19A-19AL Exemplary techniques and user interfaces for configuring electronic devices according to some embodiments are shown.

[0128] Figures 20A-20C is a flowchart illustrating a process for configuring an electronic device according to some embodiments.

[0129] Figures 21A-21I Example techniques and user interfaces for adjusting the audible output of an electronic device are shown according to some embodiments.

[0130] Figures 22A-22B is a flowchart illustrating a process for adjusting the audible output of an electronic device according to some embodiments.

[0131] Figures 23A-23KExemplary techniques and user interfaces for managing a playback queue are shown according to some embodiments.

[0132] Figures 24A-24B is a flowchart illustrating a process for managing a playback queue according to some embodiments.

[0133] Figures 25A-25I Exemplary techniques and user interfaces for playing back media according to some embodiments are shown.

[0134] Figures 26A-26B is a flowchart illustrating a process for playing back media according to some embodiments.

[0135] Figures 27A-27E Exemplary techniques and user interfaces for providing access to an electronic device according to some embodiments are shown.

[0136] Figures 28A-28C is a flowchart illustrating a process for providing access to an electronic device according to some embodiments.

[0137] Figures 29A-29I Described are exemplary techniques and user interfaces for managing media playback according to some embodiments.

[0138] Figures 30A-30C is a flowchart illustrating a process for managing media playback according to some embodiments.

[0139] Figures 31A-31N Exemplary techniques and user interfaces for providing output according to some embodiments are shown.

[0140] Figures 32A-32C is a flow chart illustrating a process for providing output according to some embodiments.

[0141] Figures 33A-33G Depicted are exemplary techniques and user interfaces for controlling media playback according to some embodiments.

[0142] Figure 34A-Figure 34B is a flowchart illustrating a process for controlling media playback according to some embodiments.

[0143] Figures 35A-35J Described are exemplary techniques and user interfaces for managing media playback according to some embodiments.

[0144] Figures 36A-36C is a flowchart illustrating a process for managing media playback according to some embodiments.

[0145] Figures 37A-37DDepicted are exemplary techniques and user interfaces for controlling media playback according to some embodiments.

[0146] Figures 38A-38B is a flowchart illustrating a process for managing media playback according to some embodiments.

[0147] Figures 39A-39C is a flowchart illustrating a process for configuring an electronic device according to some embodiments. DETAILED DESCRIPTION

[0148] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0149] Electronic devices need to provide efficient methods and interfaces for managing media playback devices. For example, the following describes methods and techniques for transferring playback between devices. Such techniques can reduce the cognitive burden on users accessing event notifications, thereby improving productivity. Furthermore, such techniques can reduce processor and battery power that would otherwise be wasted on redundant user input.

[0150] under Figure 1A-1B 、 Figure 2 、 Figure 3 、 Figure 4A-4B and Figures 5A-5H A description of an exemplary device for performing techniques for management event notification is provided. Figures 6A-6AR Exemplary techniques and user interfaces for playing back media according to some embodiments are shown. Figures 7A-7D is a flowchart illustrating a process for playing back media according to some embodiments. Figures 6A-6AR The user interface in is used to illustrate the process described below, including Figures 7A-7D in the process. Figures 8A-8P Exemplary techniques and user interfaces for playing back media according to some embodiments are shown. Figures 9A-9C is a flowchart illustrating a process for playing back media according to some embodiments. Figures 8A-8P The user interface in is used to illustrate the process described below, including Figures 9A-9C in the process. Figures 10A-10O Exemplary techniques and user interfaces according to some embodiments are shown. Figures 11A-11D is a flowchart illustrating a process for displaying content according to some embodiments. Figures 10A-10O The user interface in is used to illustrate the process described below, including Figures 11A-11D in the process. Figures 12A-12AM Exemplary techniques and user interfaces for viewing media information according to some embodiments are shown. Figures 13A-13Fis a flowchart illustrating a process for viewing media information according to some embodiments. Figures 12A-12AM The user interface in is used to illustrate the process described below, including Figures 13A-13F in the process. Figures 14A-14T Exemplary techniques and user interfaces for controlling electronic devices according to some embodiments are shown. Figures 15A-15C is a flowchart illustrating a process for controlling an electronic device according to some embodiments. Figures 14A-14T The user interface in is used to illustrate the process described below, including Figures 15A-15C in the process. Figures 16A-16P Exemplary techniques and user interfaces for managing a playback queue are shown according to some embodiments. Figures 17A-17C is a flowchart illustrating a process for managing a playback queue according to some embodiments. Figures 18A-18C is a flowchart illustrating a process for managing a playback queue according to some embodiments. Figures 16A-16P The user interface in is used to illustrate the process described below, including Figures 17A-17C and Figures 18A-18C in the process. Figures 19A-19AL Exemplary techniques and user interfaces for configuring electronic devices according to some embodiments are shown. Figures 20A-20C is a flowchart illustrating a process for configuring an electronic device according to some embodiments. Figures 19A-19AL The user interface in is used to illustrate the process described below, including Figures 20A-20C in the process. Figures 21A-21I Example techniques and user interfaces for adjusting the audible output of an electronic device are shown according to some embodiments. Figures 22A-22B is a flowchart illustrating a process for adjusting the audible output of an electronic device according to some embodiments. Figures 21A-21I The user interface in is used to illustrate the process described below, including Figures 22A-22B in the process. Figures 23A-23K Exemplary techniques and user interfaces for managing a playback queue are shown according to some embodiments. Figures 24A-24B is a flowchart illustrating a process for managing a playback queue according to some embodiments. Figures 23A-23K The user interface in is used to illustrate the process described below, including Figures 24A-24B in the process. Figures 25A-25I Exemplary techniques and user interfaces for playing back media according to some embodiments are shown. Figures 26A-26B is a flowchart illustrating a process for playing back media according to some embodiments. Figures 25A-25I The user interface in is used to illustrate the process described below, including Figures 26A-26B in the process. Figures 27A-27EExemplary techniques and user interfaces for providing access to an electronic device according to some embodiments are shown. Figures 28A-28C is a flowchart illustrating a process for providing access to an electronic device according to some embodiments. Figures 27A-27E The user interface in is used to illustrate the process described below, including Figures 28A-28C in the process. Figures 29A-29I Described are exemplary techniques and user interfaces for managing media playback according to some embodiments. Figures 30A-30C is a flowchart illustrating a process for managing media playback according to some embodiments. Figures 29A-29I The user interface in is used to illustrate the process described below, including Figures 30A-30C in the process. Figures 31A-31N Exemplary techniques and user interfaces for providing output according to some embodiments are shown. Figures 32A-32C is a flow chart illustrating a process for providing output according to some embodiments. Figures 31A-31N The user interface in is used to illustrate the process described below, including Figures 32A-32C in the process. Figures 33A-33G Depicted are exemplary techniques and user interfaces for controlling media playback according to some embodiments. Figure 34A-Figure 34B is a flowchart illustrating a process for controlling media playback according to some embodiments. Figures 33A-33G The user interface in is used to illustrate the process described below, including Figure 34A-Figure 34B in the process. Figures 35A-35J Described are exemplary techniques and user interfaces for managing media playback according to some embodiments. Figures 36A-36C is a flowchart illustrating a process for managing media playback according to some embodiments. Figures 35A-35J The user interface in is used to illustrate the process described below, including Figures 36A-36C in the process. Figures 37A-37D Described are exemplary techniques and user interfaces for managing media playback according to some embodiments. Figures 38A-38B is a flowchart illustrating a process for managing media playback according to some embodiments. Figures 37A-37D The user interface in is used to illustrate the process described below, including Figures 38A-38B in the process.

[0151] Although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch can be named a second touch and similarly, a second touch can be named a first touch without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0152] The terms used in the description of the various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the items listed in association. It will also be understood that the terms "includes," "including," "comprises," and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, and / or parts when used in this specification, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0153] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that," or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining," or "in response to determining," or "upon detecting [stated condition or event]," or "in response to detecting [stated condition or event]," depending on the context.

[0154] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod equipment, and Device. Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0155] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0156] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a rendering application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0157] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0158] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, peripheral device interfaces 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and external ports 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contacts on device 100 (e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0159] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, the force measurements from multiple force sensors are combined (e.g., a weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface adjacent to the contact and / or its change, and / or the resistance of the touch-sensitive surface adjacent to the contact and / or its change are optionally used as a surrogate for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the surrogate measure of contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measure). In some embodiments, the surrogate measure of contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functionality that the user may not otherwise be able to access on a smaller device that has limited real estate for displaying indications (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0160] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such interpretation of touch by the user will be limited by the user's individualized sensory perceptions, there are many sensory perceptions of touch that are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "up click," "down click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate the described sensory perception for a typical (or average) user.

[0161] It should be understood that device 100 is only one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in FIG. 5 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0162] Memory 102 optionally includes high-speed random access memory and optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0163] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0164] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). RF circuitry 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via a short-range communication radio. The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11d), IEEE 802.11e, IEEE 802.11f, IEEE 802.11g, IEEE 802.11g), IEEE 802.11f, IEEE 802.11g ... 11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0165] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0166] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to a peripherals interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 is optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. The one or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in the ).

[0167] A quick press of the push button optionally unlocks the touch screen 112 or optionally begins the process of unlocking the device using gestures on the touch screen, as described in U.S. patent application Ser. No. 11 / 322,549, filed Dec. 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," (U.S. Pat. No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0168] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0169] The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to a finger of the user.

[0170] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In one exemplary embodiment, projected mutual capacitance sensing technology is used, such as that used in the Apple Inc. (Cupertino, California) and iPod Technology found in.

[0171] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive trackpad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, whereas a touch-sensitive trackpad does not provide visual output.

[0172] In some embodiments, the touch-sensitive display of the touch screen 112 is described in the following applications: (1) U.S. patent application No. 11 / 381,313, entitled “Multipoint Touch Surface Controller,” filed on May 2, 2006; (2) U.S. patent application No. 10 / 840,862, entitled “Multipoint Touchscreen,” filed on May 6, 2004; (3) U.S. patent application No. 10 / 903,964, entitled “Gestures For Touch Sensitive Input Devices,” filed on July 30, 2004; (4) U.S. patent application No. 11 / 048,264, entitled “Gestures For Touch Sensitive Input Devices,” filed on January 31, 2005; and (5) U.S. patent application No. 11 / 086,964, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices,” filed on January 18, 2005. and (9) U.S. patent application Ser. No. 11 / 367,749, filed on March 3, 2006, entitled “Multi-Functional Hand-Held Device.” All of these patent applications are hereby incorporated by reference in their entirety.

[0173] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or appendage, such as a stylus, finger, etc., to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device translates rough finger-based input into precise pointer / cursor positions or commands for performing the action desired by the user.

[0174] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0175] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0176] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor is shown coupled to the optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, facing away from the touch screen display 112 on the front of the device, enabling the touch screen display to be used as a viewfinder for still and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing the user to optionally capture an image of the user while viewing other video conference participants on the touch screen display for video conferencing. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), allowing a single optical sensor 164 to be used with the touch screen display for both video conferencing and still and / or video image acquisition.

[0177] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite touch screen display 112 located on the front of device 100.

[0178] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent applications: 11 / 241,839, entitled “Proximity Detector In Handheld Device”; 11 / 240,788, entitled “Proximity Detector In Handheld Device”; 11 / 620,702, entitled “Using Ambient Light Sensor To Augment Proximity Sensor Output”; 11 / 586,862, entitled “Automated Response To And Sensing Of User Activity In Portable Devices”; and 11 / 638,251, entitled “Methods And Systems For Automatic Configuration Of Peripherals,” which are hereby incorporated by reference in their entireties. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0179] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile outputs on the device 100 that can be felt by the user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0180] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripherals interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally implements as described in the following U.S. Patent Publication: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. Device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of device 100.

[0181] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications, if any, are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; a sensor state, which includes information obtained from the device's various sensors and input control devices 116; and position information regarding the device's position and / or posture.

[0182] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware components and software components.

[0183] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are suitable for coupling directly to other devices or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external ports are connected to (trademark of Apple Inc.) devices.

[0184] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a trackpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, with the movement of the contact point represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single-finger contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multiple-finger contact). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touchpad.

[0185] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0186] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0187] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0188] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program or the like, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0189] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0190] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0191] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services, such as the weather widget, the local yellow pages widget, and the map / navigation widget).

[0192] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0193] Contacts module 137 (sometimes called address book or contact list);

[0194] Telephone module 138;

[0195] Video conferencing module 139;

[0196] Email client module 140;

[0197] Instant messaging (IM) module 141;

[0198] Fitness support module 142;

[0199] A camera module 143 for still and / or video images;

[0200] Image management module 144;

[0201] Video player module;

[0202] Music player module;

[0203] Browser module 147;

[0204] Calendar module 148;

[0205] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, and user-created widgets 149-6;

[0206] A widget creator module 150 for forming user-created widgets 149-6;

[0207] Search module 151;

[0208] Video and music player module 152, which combines the video player module and music player

[0209] Device module;

[0210] Notepad module 153;

[0211] Map module 154; and / or

[0212] Online video module 155.

[0213] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0214] In combination with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contacts module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating telephone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or instant messaging 141; and so on.

[0215] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contacts module 137, modify an entered phone number, dial the corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0216] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the touch / motion module 130, the graphics module 132, the text input module 134, the contacts module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0217] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0218] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messaging, or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0219] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing, and transmitting fitness data.

[0220] In combination with the touch screen 112, the display controller 156, one or more optical sensors 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, and the image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0221] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0222] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0223] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0224] In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and, JavaScript files (e.g., the Yahoo! desktop widget).

[0225] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop widget creator module 150 is optionally used by a user to create a desktop widget (e.g., to transfer a user-specified portion of a web page into a desktop widget).

[0226] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0227] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0228] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0229] In combination with the RF circuitry 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0230] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send links to specific online videos. Additional description of online video applications can be found in U.S. provisional patent application 60 / 936,562, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed on June 20, 2007, and U.S. patent application 11 / 968,067, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed on December 31, 2007, the contents of which are hereby incorporated by reference in their entirety.

[0231] Each of the above modules and applications corresponds to a set of executable instructions for performing one or more of the above functions and methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs, processes, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0232] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touchpad. By using a touch screen and / or a touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0233] The set of predefined functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigating between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, the touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0234] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0235] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0236] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed by the application 136-1 or is ready for display by the application, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user.

[0237] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0238] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0239] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0240] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0241] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0242] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies a hit view as the lowest view in the hierarchy where sub-events should be processed. In most cases, the hit view is the lowest-level view in which an initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0243] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0244] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver 182.

[0245] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In other embodiments, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0246] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of the data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0247] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0248] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0249] The event comparator 184 compares the event information with a predefined event or sub-event definition and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, the sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, a drag includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0250] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0251] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0252] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event not possible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0253] In some embodiments, corresponding event recognizers 180 include metadata 183 having configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0254] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag receives the flag and performs a predefined process.

[0255] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event series or to an actively participating view. The event handler associated with the sub-event series or with the actively participating view receives the event information and performs a predetermined process.

[0256] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0257] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0258] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touchpad, such as taps, drags, scrolls, etc.; stylus input; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0259] Figure 2A portable multifunction device 100 with a touch screen 112 is shown according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, and in other embodiments described below, a user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figures) or using one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0260] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0261] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding it in the depressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts speech input for activating or deactivating certain functions via the microphone 113. The device 100 also optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0262] Figure 33 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes called chipsets) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above referenced devices). Figure 1A The tactile output generator 167 described above), sensor 359 (e.g., optical sensor, acceleration sensor, proximity sensor, touch sensor and / or contact intensity sensor (similar to the above reference Figure 1A The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores data related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0263] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the memory device mentioned previously.Each module in the above-mentioned identified module corresponds to the instruction set for performing the above-mentioned functions.Above-mentioned modules or programs (for example, instruction set) need not be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the subset of above-mentioned modules and data structure.In addition, memory 370 optionally stores other modules and data structure not described above.

[0264] Attention is now turned to an embodiment of a user interface, optionally implemented on, for example, portable multifunction device 100 .

[0265] Figure 4A An exemplary user interface is shown for a menu of applications on portable multifunction device 100 according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0266] One or more of one or more wireless communications (e.g., cellular signals and Wi-Fi signals)

[0267] a signal strength indicator 402;

[0268] Time 404;

[0269] Bluetooth indicator 405;

[0270] Battery status indicator 406;

[0271] A tray 408 with common application icons, such as:

[0272] o An icon 416 labeled "Phone" for the phone module 138, the icon 416 optionally including an indicator 414 of the number of missed calls or voice messages;

[0273] o An icon 418 labeled "Mail" of the email client module 140, the icon 418 optionally including an indicator 410 of the number of unread emails;

[0274] o An icon 420 labeled "Browser" for the browser module 147; and

[0275] o Video and music player module 152 (also known as iPod (trademark of Apple Inc.) module

[0276] Block 152) labeled "iPod" icon 422; and

[0277] Icons for other applications, such as:

[0278] o Icon 424 labeled "Message" of IM module 141;

[0279] o An icon 426 labeled “Calendar” of the calendar module 148;

[0280] o Icon 428 labeled “Photos” of the image management module 144;

[0281] o An icon 430 labeled “Camera” of the camera module 143;

[0282] ○ An icon 432 labeled “Online Video” of the online video module 155;

[0283] ○ Icon 434 labeled “Stock Market” of the Stock Market desktop applet 149-2;

[0284] o An icon 436 labeled “Map” for the map module 154;

[0285] ○ Icon 438 labeled “Weather” of the weather widget 149-1;

[0286] ○ Icon 440 labeled “Clock” of the alarm clock widget 149-4;

[0287] o An icon 442 labeled “Fitness Support” of the fitness support module 142;

[0288] o An icon 444 labeled "Notepad" of the Notepad module 153; and

[0289] o An icon 446 labeled “Settings” for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0290] Please note that Figure 4A The icon labels shown in FIG are exemplary only. For example, icon 422 for video and music player module 152 is labeled "Music" or "Music Player." Other labels are optionally used for various application icons. In some embodiments, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0291] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, touch screen display 112) that is separate from a display 450 (eg, touch screen display 112). Figure 3 tablet or trackpad 355) of the device (e.g., Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0292] Although some of the examples below will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B , 460 corresponds to 468 and 462 corresponds to 470 ) at contact with touch-sensitive surface 451 (eg, Figure 4B 460 and 462 in FIG. 4 ). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and the display of the multi-function device ( Figure 4B When the user interface 450 in FIG. 1 is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0293] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detection of contact, and then ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0294] Figure 5AAn exemplary personal electronic device 500 is shown. The device 500 includes a body 502. In some embodiments, the device 500 may include a body 502 relative to the devices 100 and 300 (e.g., Figures 1A-4B ) some or all of the features described in . In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. Alternatively or in addition to touch screen 504, device 500 also has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of applied contact (e.g., touch). The one or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on their intensity, which means that touches of different intensities can invoke different user interface operations on device 500.

[0295] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0296] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. Input mechanism 506 and input mechanism 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, wristbands, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the device 500 to be worn by the user.

[0297] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, the device 500 may include a Figure 1A 、 Figure 1B and Figure 3 Some or all of the components described. The device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. The I / O portion 514 can be connected to a display 504, which can have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, the I / O portion 514 can be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. The device 500 may include input mechanisms 506 and / or 508. For example, the input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, the input mechanism 508 is optionally a button.

[0298] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to I / O portion 514.

[0299] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, may cause the computer processors to perform the following techniques, including process 700 (Figure 7), process 900 (Figure 9), process 1100 (Figure 11), process 1300 (Figure 13), process 1500 (Figure 15), process 1700 (Figure 17), process 1800 (Figure 18), process 2000 (Figure 20), process 2200 (Figure 22), process 2400 (Figure 24), process 2600 (Figure 26), process 2800 (Figure 28), process 3000 (Figure 30), process 3200 (Figure 32), process 3400 (Figure 34), process 3600 (Figure 36), and process 3800 (Figure 38). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5B components and configurations, but rather may include other components or additional components in a variety of configurations.

[0300] As used herein, the term "indicator" refers to an indication that is optionally provided on device 100, 300, and / or 500 ( Figure 1A 、 Figure 3 and Figure 5A-5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.

[0301] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in FIG, the particular user interface element is adjusted according to the detected input. In the case of a touch screen display (e.g., a touch screen display) that enables direct interaction with user interface elements on the touch screen display Figure 1A touch-sensitive display system 112 or Figure 4A In some implementations of the touch screen 112 in FIG, 2 , a contact detected on the touch screen acts as a “focus selector” such that when input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0302] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples sampled during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after the contact is detected, before the contact is detected to be lifted, before or after the contact begins to move, before the contact ends, before or after the intensity of the contact is detected to increase, and / or before or after the intensity of the contact is detected to decrease). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0303] Figure 5C Detecting multiple contacts 552A-552E on touch-sensitive display 504 using multiple intensity sensors 524A-524D is shown. Figure 5C Also included is an intensity graph that shows the current intensity measurements of intensity sensors 524A-524D relative to intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are both 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are both 7 intensity units. In some implementations, the cumulative intensity is the sum of the intensity measurements of multiple intensity sensors 524A-524D, which in this example is 32 intensity units. In some embodiments, each contact is assigned a corresponding intensity, which is a portion of the cumulative intensity. Figure 5D554. The cumulative strengths assigned to contacts 552A-552E based on their distances from the center of force 554 are shown. In this example, each of contacts 552A, 552B, and 552E is assigned a strength of the contact of 8 strength units of the cumulative strength, and each of contacts 552C and 552D is assigned a strength of the contact of 4 strength units of the cumulative strength. More generally, in some implementations, each contact j is assigned a corresponding strength Ij that is a fraction of the cumulative strength A according to a predefined mathematical function Ij=A·(Dj / ΣDi), where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i=1 to the last) from the center of force. Reference may be performed using an electronic device similar to or identical to device 100, 300, or 500. Figure 5C-5D In some embodiments, the characteristic intensity of the contact is based on one or more intensities of the contact. In some embodiments, the intensity sensor is used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included in the Figure 5C-5D To help readers.

[0304] In some embodiments, a portion of a gesture is identified for determining the characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position and reaches an end position where the contact strength increases. In this example, the characteristic strength of the contact at the end position is optionally based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact for determining the characteristic strength.

[0305] The intensity of the contact on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below which are no longer detected), the device will move the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0306] An increase in the characteristic intensity of a contact from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in the characteristic intensity of a contact from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in the characteristic intensity of a contact from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in the characteristic intensity of a contact from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0307] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting an increase in the strength of the contact (or multiple contacts) to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting an increase in the strength of the corresponding contact to above the press input strength threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and a subsequent decrease in the strength of the contact to below the press input strength threshold, and the corresponding operation is performed in response to detecting a subsequent decrease in the strength of the corresponding contact to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0308] Figure 5E-5HDetection of a gesture is shown, the gesture comprising contact 562 with an intensity from below Figure 5E Light press intensity threshold in L ”) increases to a value higher than Figure 5H The deep press intensity threshold in D ”). On the displayed user interface 570 including application icons 572A-572D displayed in the predefined area 574, a cursor 576 is displayed above the application icon 572B corresponding to application 2, and a gesture performed using contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch-sensitive display 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines that the intensity of the contact 562 is above the deep press intensity threshold (for example, “IT D ”) reaches a peak above. Contact 562 is maintained on touch-sensitive surface 560. In response to detecting the gesture, and according to the intensity rising to the deep press intensity threshold (e.g., “IT D ”) above contact 562, displays reduced-scale representations 578A-578C (e.g., thumbnails) of documents recently opened for Application 2, as shown in FIG. Figure 5F-5H In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in the Figure 5E-5H To help readers.

[0309] In some embodiments, the display of representations 578A-578C includes animation. For example, representation 578A is initially displayed near application icon 572B, as shown in FIG. Figure 5F As the animation progresses, representation 578A moves upward and representation 578B is displayed near application icon 572B, as shown. Figure 5G Then, representation 578A moves upward, 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown in FIG. Figure 5H In some embodiments, the animation progresses according to the intensity of contact 562, as shown in FIG. Figure 5F-5G , where representations 578A-578C appear and decrease as the intensity of contact 562 moves toward a deep press intensity threshold (e.g., “IT D In some embodiments, the intensity according to which the animation progresses is the characteristic intensity of the contact. The reference may be performed using an electronic device similar to or identical to device 100, 300, or 500. Figure 5E-5H The operation described.

[0310] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units less than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the intensity of the contact increases from an intensity equal to or less than the hysteresis intensity threshold to an intensity equal to or greater than the press input intensity threshold and, optionally, that the intensity of the contact subsequently decreases to an intensity equal to or less than the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact, depending on the circumstances).

[0311] For ease of explanation, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting any of the following: contact intensity increasing above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing below the press input intensity threshold, and / or contact intensity decreasing below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0312] Figure 5I An exemplary electronic device 580 is shown. The device 580 includes a body 580A. In some embodiments, the device 580 may include a body 580A relative to the devices 100, 300, and 500 (e.g., Figures 1A-5B) some or all of the features described in the preceding claims. In some embodiments, device 580 has one or more speakers 580B (hidden in body 580A), one or more microphones 580C, one or more touch-sensitive surfaces 580D, and one or more displays 580E. Alternatively or in addition to the display and touch-sensitive surface 580D, the device also has a touch-sensitive display (also known as a touch screen). As with devices 100, 300, and 500, in some embodiments, the touch-sensitive surface 580D (or touch screen) optionally includes one or more intensity sensors for detecting the intensity of the applied contact (e.g., touch). The one or more intensity sensors of the touch-sensitive surface 580D (or touch screen) can provide output data representing the intensity of the touch. The user interface of device 580 can respond to touches based on their intensity, which means that touches of different intensities can invoke different user interface operations on device 580. In some embodiments, the one or more displays 580E are one or more light emitting diodes (LEDs). For example, a display can be a single LED, a cluster of LEDs (e.g., red, green, and blue LEDs), multiple discrete LEDs, multiple clusters of discrete LEDs, or other arrangements of one or more LEDs. For example, display 580E can be an array of nine discrete LED clusters arranged in a circular pattern (e.g., an annular pattern). In some examples, the one or more displays include one or more light-emitting elements of another type.

[0313] Figure 5J An exemplary personal electronic device 580 is depicted. In some embodiments, the device 580 may include a Figure 1A 、 Figure 1B 、 Figure 3 and Figure 5A-5B Some or all of the components described. The device 580 has a bus 592 that operatively couples an I / O portion 594 to one or more computer processors 596 and a memory 598. The I / O portion 594 can be connected to a display 582, which can have a touch-sensitive component 584 and optionally a strength sensor 585 (e.g., a contact strength sensor). In some embodiments, the touch-sensitive component 584 is a component separate from the display 582. In addition, the I / O portion 594 can be connected to a communication unit 590 for receiving application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. The device 580 may include an input mechanism 588. In some examples, the input mechanism 588 is optionally a button. In some examples, the input mechanism 588 is optionally a microphone. The input mechanism 588 is optionally a plurality of microphones (e.g., a microphone array).

[0314] The electronic device 580 includes a speaker 586 for outputting audio. The device 580 may include audio circuitry (e.g., in the I / O portion 594) that receives audio data, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 586. The speaker 586 converts the electrical signals into sound waves audible to humans. The audio circuitry (e.g., in the I / O portion 594) also receives electrical signals converted from sound waves by a microphone (e.g., input mechanism 588). The audio circuitry (e.g., in the I / O portion 594) converts the electrical signals into audio data. The audio data is optionally retrieved from and / or transmitted to a memory 598 and / or RF circuitry (e.g., in the communication unit 590) by the I / O portion 594.

[0315] The memory 598 of the personal electronic device 580 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 596, may cause the computer processors to perform the following techniques, including process 700 (Figure 7), process 900 (Figure 9), process 1100 (Figure 11), process 1300 (Figure 13), process 1500 (Figure 15), process 1700 (Figure 17), process 1800 (Figure 18), process 2000 (Figure 20), process 2200 (Figure 22), process 2400 (Figure 24), process 2600 (Figure 26), process 2800 (Figure 28), process 3000 (Figure 30), process 3200 (Figure 32), process 3400 (Figure 34), process 3600 (Figure 36), and process 3800 (Figure 38). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 580 is not limited to Figure 5J components and configurations, but rather may include other components or additional components in a variety of configurations.

[0316] As used herein, an "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched (e.g., become open) on the device. In some embodiments, a downloaded application becomes an installed application using an installer program that extracts program portions from the downloaded software package and integrates the extracted portions with the computer system's operating system.

[0317] As used herein, the term "open application" or "executing application" refers to a software application that has persisted state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or executing application is optionally any of the following types of applications:

[0318] The active application, which is currently displayed on the display of the device on which the application is being used;

[0320] Background applications (or background processes) that are not currently displayed but are part of the application

[0321] or multiple processes are being processed by one or more processors; and

[0322] • A suspended or dormant application that is not running but has state information stored in memory (volatile and non-volatile, respectively) that can be used to resume execution of the application.

[0323] As used herein, the term "closed application" refers to a software application that does not retain state information (e.g., the state information of a closed application is not stored in the device's memory). Thus, closing an application includes stopping and / or removing the application's application process and removing the application's state information from the device's memory. Generally speaking, while in a first application, opening a second application does not close the first application. When the second application is displayed and the first application ceases to be displayed, the first application becomes a background application.

[0324] Throughout this document, reference is made to functions, operations, and features related to digital assistants (also referred to as "voice assistants" or "personal digital assistants"). A digital assistant may refer to any information processing system that interprets natural language input in the form of speech and / or text to infer user intent and performs actions based on the inferred user intent. For example, to comply with the inferred user intent, the system may perform one or more of the following: identify a task flow with steps and parameters designed to implement the inferred user intent, input specific requirements from the inferred user intent into the task flow; execute the task flow by calling a program, method, service, API, etc.; and generate an output response to the user in audible (e.g., voice) and / or visual form.

[0325] Specifically, a digital assistant may be capable of accepting user requests that are at least partially in the form of natural language commands, requests, statements, narrations, and / or inquiries. Typically, a user request may seek an informational answer from the digital assistant or request the digital assistant perform a task. A satisfactory response to a user request may be providing the requested informational answer, performing the requested task, or a combination of the two. For example, a user may ask the digital assistant a question such as, "Where am I now?" Based on the user's current location, the digital assistant may respond, "You are near the West Gate of Central Park." A user may also request a task, such as, "Please invite my friends to my girlfriend's birthday party next week." In response, the digital assistant may confirm the request by saying, "Okay, right away," and then send appropriate calendar invitations on behalf of the user to each of the user's friends listed in the user's electronic address book. While performing the requested task, the digital assistant may sometimes interact with the user in a continuous conversation involving multiple information exchanges over an extended period of time. There are many other ways to interact with a digital assistant to request information or perform various tasks. In addition to providing verbal responses and taking programmed actions, digital assistants can also provide responses in other visual or audio forms, such as text, alerts, music, video, animation, etc.

[0326] In some examples, a digital assistant may be implemented according to a client-server model (e.g., the exemplary "digital assistant system"). The digital assistant may include a client-side portion (hereinafter referred to as a "DA client") that executes on a user device (e.g., device 100, 300, 500, or 580) and a server-side portion (hereinafter referred to as a "DA server") that executes on a server system (e.g., remote from the user device). The DA client may communicate with the DA server over one or more networks. The DA client may provide client-side functionality, such as input and output processing for the user, and communication with the DA server. The DA server may provide server-side functionality for any number of DA clients, each located on a respective user device.

[0327] In some examples, a DA server may include a client-facing I / O interface, one or more processing modules, data and models, and an I / O interface to external services. The client-facing I / O interface may facilitate the DA server to perform client-facing input and output processing. The one or more processing modules may utilize the data and models to process voice input and determine the user's intent based on the natural language input. In addition, the one or more processing modules perform tasks based on the inferred user intent. In some examples, the DA server may communicate with external services over one or more networks to complete tasks or collect information. The I / O interface to the external service may facilitate such communications.

[0328] The user device may be any suitable electronic device. For example, the user device may be a portable multi-function device (e.g., Figure 1A Device 100 as described above), multi-function device (e.g., Figure 3 Device 300 described), personal electronic devices (eg, Figure 5A-5B The device 500 described above) or an electronic device (eg, Figure 5I-5J The portable multifunction device may be, for example, a mobile phone that also includes other functions such as a PDA and / or music player. Specific examples of portable multifunction devices may include the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Equipment and Device. Other examples of portable multifunction devices may include, but are not limited to, laptop computers or tablet computers. Furthermore, in some examples, the user device may be a non-portable multifunction device. Specifically, the user device may be a desktop computer, a game console, a television, a smart speaker, or a television set-top box. In some examples, the user device may include a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). Furthermore, the user device may optionally include one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick. Various examples of electronic devices such as multifunction devices are described in more detail below.

[0329] Examples of the one or more communication networks may include a local area network (LAN) and a wide area network (WAN), such as the Internet. The one or more communication networks may be implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0330] The server system can be implemented on one or more stand-alone data processing devices or distributed networks of computers. In some examples, the server system can also use various virtual devices and / or services of third-party service providers (e.g., third-party cloud service providers) to provide potential computing resources and / or infrastructure resources of the server system.

[0331] In some examples, a user device (also referred to as a first user device) may communicate with a DA server via a second user device. The second user device may be similar or identical to the user device described above. For example, the second user device may be similar to the user device described above. Figure 1A 、 Figure 3 、 Figure 5A-5B and Figure 5I-5J Device 100, 300, 500 or 580 described. The first user device can be configured to be communicatively coupled to the second user device via a direct communication connection such as Bluetooth, NFC, BTLE, etc. or via a wired or wireless network such as a local Wi-Fi network. In some examples, the second user device can be configured to act as a proxy between the first user device and the DA server. For example, the DA client of the first user device can be configured to transmit information (e.g., a user request received at the first user device) to the DA server via the second user device. The DA server can process the information and return relevant data (e.g., data content in response to the user request) to the first user device via the second user device.

[0332] In some examples, the digital assistant system can be implemented on a stand-alone computer system. In some examples, the digital assistant system can be distributed across multiple computers. In some examples, as described above, some of the modules and functions of the digital assistant can be divided into a server portion and a client portion, where the client portion is located on one or more user devices (e.g., devices 100, 300, 500, 580) and communicates with the server portion via one or more networks. In some examples, the digital assistant system can be a specific implementation of a server system (and / or DA server).

[0333] In some examples, the digital assistant system includes a memory that can also store the digital assistant modules (or the server portion of the digital assistant). In some examples, the digital assistant modules can include the following submodules, or subsets or supersets thereof: an input / output processing module, a speech-to-text (STT) processing module, a natural language processing module, a dialog flow processing module, a task flow processing module, a service processing module, and a speech synthesis module. Each of these modules can have access to one or more of the following systems or data and models of the digital assistant modules, or subsets or supersets thereof: a knowledge ontology, a vocabulary index, user data, a task flow model, a service model, and an ASR system.

[0334] In some examples, using processing modules, data, and models implemented in the digital assistant module, the digital assistant may perform at least some of the following: convert speech input into text; recognize the user's intent expressed in natural language input received from the user; proactively elicit and obtain information needed to fully infer the user's intent (e.g., by disambiguating words, games, intents, etc.); determine a task flow for satisfying the inferred intent; and execute the task flow to satisfy the inferred intent.

[0335] In some examples, the I / O processing module can interact with the user via an I / O device or with a user device (e.g., device 100, 300, 500, or 580) via a network communication interface to obtain user input (e.g., voice input) and provide a response to the user input (e.g., as voice output). The I / O processing module can optionally obtain contextual information associated with the user input from the user device along with or shortly after receiving the user input. The contextual information can include user-specific data, vocabulary, and / or preferences related to the user input. In some examples, the contextual information also includes the software and hardware state of the user device at the time the user request was received, and / or information related to the user's surroundings at the time the user request was received. In some examples, the I / O processing module can also send follow-up questions regarding the user request to the user and receive answers from the user. When the user request is received by the I / O processing module and the user request includes voice input, the I / O processing module can forward the voice input to the STT processing module (or speech recognizer) for speech-to-text conversion.

[0336] The STT processing module may include one or more ASR systems. The one or more ASR systems may process the speech input received by the I / O processing module to generate a recognition result. Each ASR system may include a front-end speech preprocessor. The front-end speech preprocessor may extract a representative feature portion from the speech input. For example, the front-end speech preprocessor may perform a Fourier transform on the speech input to extract the spectral feature portion that characterizes the speech input as a sequence of representative multidimensional vectors. In addition, each ASR system may include one or more speech recognition models (e.g., an acoustic model and / or a language model) and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines may include an engine based on dynamic time warping and an engine based on weighted finite state transducer (WFST). One or more speech recognition models and one or more speech recognition engines may be used to process the extracted representative features of the front-end speech preprocessor to generate intermediate recognition results (e.g., phonemes, phoneme strings, and subwords), and ultimately generate text recognition results (e.g., words, word strings, or sequences of symbols). In some examples, the speech input may be at least partially processed by a third-party service or processed on the user's device (e.g., device 100, 300, 500, or 580) to generate recognition results. Once the STT processing module generates a recognition result containing a text string (e.g., a word, or a sequence of words, or a sequence of symbols), the recognition result may be passed to the natural language processing module for intent inference.

[0337] In some examples, the natural language processing module ("natural language processor") of the digital assistant may obtain a sequence of words or symbols ("symbol sequence") generated by the STT processing module and attempt to associate the symbol sequence with one or more "executable intents" recognized by the digital assistant. An "executable intent" may represent a task that can be executed by the digital assistant and may have an associated task flow implemented in the task flow model of the digital assistant system. The associated task flow may be a series of programmed actions and steps that the digital assistant takes to perform the task. The scope of the digital assistant's capabilities may depend on the number and variety of task flows that have been implemented and stored in the task flow model, or in other words, on the number and variety of "executable intents" recognized by the digital assistant. However, the effectiveness of the digital assistant may also depend on the assistant's ability to infer the correct "one or more executable intents" from a user request expressed in natural language.

[0338] In some examples, natural language processing may be based on, for example, a knowledge ontology. A knowledge ontology may be a hierarchical structure comprising many nodes, each node representing an "executable intent" or an "attribute" related to one or more of an "executable intent" or other "attributes". As described above, an "executable intent" may represent a task that the digital assistant is able to perform, i.e., the task is "executable" or can be performed. An "attribute" may represent a parameter associated with a sub-aspect of an executable intent or another attribute. The link between an executable intent node and an attribute node in the knowledge ontology may define how the parameter represented by the attribute node is subordinate to the task represented by the executable intent node.

[0339] In some examples, once the natural language processing module identifies an executable intent (or domain) based on a user request, the natural language processing module may generate a structured query representing the identified executable intent. In some examples, the structured query may include parameters for one or more nodes within the executable intent's domain, with at least some of the parameters populated with specific information and requirements specified in the user request. For example, a user may say, "Make me a reservation for 7 PM at the sushi restaurant." In this case, the natural language processing module may correctly identify the executable intent as "restaurant reservation" based on the user input. According to the knowledge ontology, a structured query for the "restaurant reservation" domain may include parameters such as {cuisine}, {time}, {date}, {number of people in the group}, etc. In some examples, based on speech input and text derived from the speech input using the STT processing module, the natural language processing module may generate a partial structured query for the restaurant reservation domain, where the partial structured query includes the parameters {cuisine = "sushi"} and {time = "7 PM"}. However, in this example, the user utterance does not contain sufficient information to complete the structured query associated with the domain. Therefore, based on the currently available information, other necessary parameters such as {number of people in the group} and {date} may not be specified in the structured query. In some examples, the natural language processing module can use the received contextual information to populate some parameters of the structured query. For example, in some examples, if the user requests a sushi restaurant "nearby," the natural language processing module can populate the {location} parameter in the structured query with the GPS coordinates from the user's device.

[0340] In some examples, the natural language processing module may pass the generated structured query (including any completed parameters) to a task flow processing module ("task flow processor"). The task flow processing module may be configured to receive the structured query from the natural language processing module, complete the structured query (if necessary), and perform the actions required to "complete" the user's final request. In some examples, the various processes necessary to complete these tasks may be provided in a task flow model. In some examples, the task flow model may include a process for obtaining additional information from the user, and a task flow for performing actions associated with the executable intent.

[0341] As described above, to complete a structured query, the task flow processing module may need to initiate additional conversations with the user to obtain additional information and / or clarify potentially ambiguous utterances. When such interaction is necessary, the task flow processing module may invoke the dialog flow processing module to participate in the conversation with the user. In some examples, the dialog flow processing module may determine how (and / or when) to request additional information from the user and receive and process user responses. Questions may be presented to the user and responses may be received from the user via the I / O processing module. In some examples, the dialog flow processing module may present dialog output to the user via audio and / or visual output and receive input from the user via verbal or physical (e.g., click) responses. Continuing with the above example, when the task flow processing module invokes the dialog flow processing module to determine the "number of people in your party" and "date" information for a structured query associated with the domain "restaurant reservation," the dialog flow processing module may generate questions such as "How many people in your party?" and "What date do you want to book?" to be passed to the user. Once responses are received from the user, the dialog flow processing module may fill in the missing information for the structured query or pass information to the task flow processing module to complete the missing information based on the structured query.

[0342] Once the task flow processing module has completed the structured query for the executable intent, the task flow processing module can proceed to execute the final task associated with the executable intent. Therefore, the task flow processing module can execute the steps and instructions in the task flow model according to the specific parameters contained in the structured query. For example, the task flow model for the executable intent "restaurant reservation" may include steps and instructions for contacting a restaurant and actually requesting a reservation for a specific number of people at a specific time. For example, using a structured query such as: {restaurant reservation, restaurant = ABC Cafe, date = 3 / 12 / 2012, time = 7pm, number of people = 5}, the task flow processing module can perform the following steps: (1) Log in to the server of ABC Cafe or a restaurant reservation system such as (2) entering date, time, and number of party members into a form on the website, (3) submitting the form, and (4) creating a calendar entry for the reservation on the user's calendar.

[0343] As discussed throughout this document, various techniques for receiving user input at a first device and causing an action or output at a second device are discussed. For example, a first user device (e.g., 580 or 3300) receives user input (e.g., "Play a Ramones song in my living room"). In some examples, the user input includes user speech, and the first user device uses a microphone of the first electronic device to receive the user input. In some examples, the first electronic device converts the user input into a representation of the audio input. The representation of the audio input can be an analog representation or a digital representation. In some examples, the representation is a text representation, and the first electronic device uses speech-to-text conversion to convert the user input into a text representation. Any known speech-to-text conversion process can be used to convert the user speech.

[0344] In some embodiments, the user input, or a representation thereof, is provided (e.g., transmitted) by the first electronic device to a virtual assistant server (e.g., a DA server) for processing.

[0345] Based on the user input, the virtual assistant server identifies one or more tasks and parameters. For example, the virtual assistant server interprets the textual representation of the user input to derive an intent and operationalizes the intent into one or more tasks. In the depicted example, based on the user input "play a song by the Ramones in my living room," the virtual assistant server identifies a task of adding one or more media items to a media playback queue at a desired electronic device (e.g., 580, 3304). The task can be identified in any number of ways. Technology for identifying tasks can be found in, for example, U.S. Utility Application No. 14 / 503,105, filed on September 30, 2014, entitled "Intelligent Assistant for Home Automation," and U.S. Provisional Patent Application Serial No. 62 / 348,015, filed on June 9, 2016, entitled "Intelligent Automated Assistant in a Home Environment," (Agent Docket No. 770003000100 (P30331USP1)), the entire disclosures of which are incorporated herein by reference.

[0346] In addition, the virtual assistant server identifies relevant parameters required to complete the task. In some examples, the parameters include an indication of the media item to be played, such as a song title and a media file name. In some examples, the parameters include an indication of the type of media item to be played, such as audio, video, and text. In some examples, the parameters include an indication of the location of the desired electronic device, such as a named location and configuration associated with the user ("living room," "garage," "upstairs") or GPS coordinates. In some examples, the parameters include an indication of the type of electronic device desired, such as a television, a speaker dock, and a smartwatch. In this example, the virtual assistant server identifies the parameters as "Ramones" and "living room."

[0347] The virtual assistant server can transmit a request for identifying one or more user devices to a device identification storage device (e.g., on the virtual assistant server). The request includes one or more parameters for identifying the one or more user devices. In some examples, the one or more parameters can specify a named location (e.g., "living room"), a device type (e.g., "Apple TV"), and / or one or more attributes of a media item, such as a type of media item (e.g., "audio" and "video").

[0348] A device identification storage device (e.g., on a virtual assistant server) may store data and / or models for a variety of attributes and related information associated with various electronic devices. Attributes that may be stored for an electronic device include, but are not limited to, a unique identifier, state, type, location, supported media types, and ownership. For example, a device identification storage device may store information associating a device with the type "AppleTV," the named location "living room," and the supported media types "audio" and "video." For another example, a device identification storage device may store information associating a device with the type "speaker," the named location "garage," and the supported media type "audio."

[0349] Device type is particularly relevant to smart device arbitration when the user's input specifies a task to be performed, such as, "Hey Siri, find me a TV episode" or "Hey Siri, drive me there." In some embodiments, the electronic device responds to the audio input if, based on the broadcasted value, the electronic device is the only device that can handle the type of task specified in the audio input.

[0350] In some examples, the electronic device obtains a list of acceptable device types that can handle a specified task (e.g., by locally analyzing the audio input and / or by receiving the list from one or more servers) and determines whether the device will respond to the audio input based on the broadcasted “device type” value and the list of acceptable device types. In some examples, one or more servers receive data representing sampled audio input from one or more of the electronic devices, derive user intent based on the data, and identify a task having one or more parameters based on the user intent. In some examples, once the one or more servers determine the task based on the audio input, the one or more servers transmit the task, parameters (if any), and a list of acceptable device types for processing the task. Additional details regarding identifying tasks from natural language input can be found, for example, in U.S. Utility Application No. 15 / 144,618, filed on May 2, 2016, entitled “Intelligent Device Identification,” which is hereby incorporated by reference in its entirety.

[0351] The device identification storage (e.g., on the virtual assistant server) can be implemented via hardware, software, or a combination thereof. In some examples, the device identification storage can be located on one or more of the aforementioned electronic devices, including the first user device (e.g., 580, 3300) and the second user device (e.g., 5380, 3304). The device identification storage can be located on the same electronic device that hosts either the media identification storage or the virtual assistant server.

[0352] At the device identification storage, one or more user devices are identified based on parameters provided in the request from the virtual assistant server. In some examples, the device identification storage determines whether any of the electronic devices having an entry in the database has attributes that match the parameters. Additional details regarding the identification of electronic devices can be found, for example, in U.S. Utility Application No. 14 / 503,105, filed on September 30, 2014, entitled “Intelligent Assistant for Home Automation,” the entire disclosure of which is incorporated herein by reference.

[0353] In response to the request for identifying one or more devices, the virtual assistant server receives identification information for the one or more devices. For example, the identification information includes a unique identifier corresponding to the identified user device (eg, "uid=123456").

[0354] The virtual assistant server transmits a request for identifying one or more media items to a media identification storage device. The request includes one or more parameters that allow for identification of the one or more media items, such as a parameter "Ramones." The media identification storage device includes a database that associates (e.g., catalogs) a set of media items by one or more attributes, including title, artist, genre, length, and any other relevant information associated with the media items. By way of example only, the one or more media items are identified based on the parameter "Ramones" at the media identification storage device.

[0355] In response to a request for identifying one or more media items, the virtual assistant server receives identification information for the one or more media items identified from the media identification storage device. In some embodiments, the identification information includes a unique identifier corresponding to the one or more identified media items, such as "artist: / / store / 56989". The unique identifiers of the one or more media items can be used, for example, to identify the one or more media items and subsequently access the one or more identified media items using one or more applications that support media playback. The media identification storage device can be located on any local or remote electronic device (such as the same electronic device where the virtual assistant server is located).

[0356] In some embodiments, the virtual assistant server provides a unified command to a first user device (e.g., 580, 3300). The command includes identification information of the one or more media items and identification information of a second user device (e.g., 3403). For example, the unified command includes the text string "SetMediaPlayerQueue artist: / / store / 56989AirplayRouteUid=12456". In response to receiving the unified command, the user device (e.g., 3300) triggers playback of the identified one or more media items (e.g., media about the Ramones) at the identified user device (e.g., second user device 3304). In some examples, the first user device (e.g., 3300) transmits relevant information derived from the unified command to the second user device (e.g., 3304).

[0357] Attention is now turned to an embodiment of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0358] Figures 6A-6AR An exemplary user interface for accessing media playback status of multiple available playback devices is shown according to some embodiments. The user interface in these figures is used to illustrate the media playback status of multiple available playback devices. Figures 7A-7DThe process is described below.

[0359] Figure 6A-6B An exemplary user interface for managing and controlling a group of devices is shown. In some embodiments, the group of devices includes devices throughout a user's home that perform various functions. Such functions may include controlling smart devices (e.g., light bulbs, thermostats, media devices), configuring home automation, and controlling home entertainment.

[0360] Figure 6A A home summary screen 601 (on display 602 of device 600) representing an exemplary interface for a home control application (also referred to as a "device management application") executed on device 600 is depicted. In some embodiments, device 600 includes one or more features of device 100, 300, or 500. An example of a home control application is "Home" (developed by Apple Inc. (Cupertino, California, USA)). The home summary screen 601 includes stored scenes (e.g., pre-configured sets of settings for one or more home devices) and indications of devices in the user's home (also referred to as "home devices", etc.). Home devices are represented by indicators 601A to 601C. Indicator 601A corresponds to a device named "Smart Speaker 1" in the room "Living Room". Indicator 601B corresponds to a device named "Smart Speaker 2" in the room "Kitchen". Indicator 601C corresponds to a device named "Smart Lightbulb" in the room "Bedroom". Exemplary interfaces and techniques for home control applications are described in U.S. patent application 62 / 349,057, entitled “User Interface for Managing Controllable External Devices,” filed on June 12, 2016, which is hereby incorporated by reference in its entirety.

[0361] Figure 6AAlso depicted is an example home configuration screen 603, which illustrates an example interface for managing configuration settings associated with the home devices represented in home summary screen 601. In some examples, in response to a user input selection of affordance 601D on home summary screen 601, device 600 displays home configuration screen 603. In home configuration screen 603, a user can edit the name of the home (in this example, "Home"), edit the members of the home (e.g., Users 1, 2, and 3 in this example), include notes for other members, and other functionality. As can be seen, "Home" in this example includes three members: User 1 (represented by indicator 603A), User 2 (represented by indicator 603B), and User 3 (represented by indicator 603C). In some examples, Users 1 through 3 represent users who reside at a location (collectively, "Home") that physically includes the home devices. Thus, Users 1 through 3 are members of the home. As members of the home (or other location), in some examples, users are provided with permission to control devices associated with the home. A user can control a home device using one or more personal devices associated with at least one of the users. In some examples, a device is associated with a user when the device is logged into an account associated with (e.g., owned by) the user. For example, a device logged into one or more user accounts associated with the exemplary Users 1 through 3 described above is associated with at least one user, and specifically, at least one user of Home (e.g., a home device).

[0362] Figure 6B An exemplary user interface of a portable electronic device 600 is depicted. Device home screen 605, displayed on display 602, is an exemplary default home screen for device 600. Device home screen 605 includes affordances associated with applications stored on the device. For example, affordance 605A corresponds to a music application. In some examples, selection of affordance 605A causes the device to display the user interface of the corresponding music application. Figure 6B Also depicted is exemplary user input 604 received by device 600 while displaying screen 603. In this example, user input 604 is a swipe gesture (e.g., a touch input followed by a drag) on ​​touch-sensitive display 602 of device 600. Figure 6BIn the depicted example, user input 604 is a swipe gesture that originates at the edge of touch-sensitive display 602. In some embodiments, user input 604 is an arc-shaped swipe gesture. For example, an arc-shaped swipe gesture is a gesture that includes both a horizontal component (e.g., in the x-direction) and a vertical component (e.g., in the y-direction) that meet the arc criteria. For example, user input 604 can be a touch and drag that follows a quarter circle on the touch-sensitive surface. In response to input 604, device 600 displays a control center interface (e.g., Figure 6C and Figure 6D ).

[0363] Figure 6C An exemplary control center interface is shown. Control center interface 606 includes various enable indicators for controlling various functions and / or components of device 600. In the depicted example, control center interface 606 includes exemplary illustrations of the following items: media control interface 606A, volume control interface 606C, screen brightness control interface 606D, camera enable indicator 606E, calculator enable indicator 606F, timer enable indicator 606G, flashlight enable indicator 606H, wireless media streaming enable indicator 606I, screen rotation lock enable indicator 606J, do not disturb enable indicator 606K, and wireless antenna control interface 606L. Exemplary media control interface 606A includes media playback controls 606M (e.g., previous track, pause / play, next track) and device selection enable indicator 606B. Exemplary wireless antenna control enable indicator 606L includes an enable indicator 606N for toggling (e.g., turning on and off) the Wi-Fi functionality of device 600.

[0364] Figure 6D Another exemplary control center interface 606 is shown when the device 600 is playing back media through an external device. Figure 6D As shown, device selection affordance 606B includes an indication that the media currently being played back (e.g., the song track "Same Thing" by the artist "Jambug") is being played back via an external device (or devices) named "AirPods." Thus, media control interface 606A indicates the media currently being played (otherwise referred to as being played back, currently playing, etc.) and the device to which the audio output is being routed. In this case, the audio output is being generated by wireless personal earbuds ("AirPods" developed by Apple Inc. (Cupertino, California, USA)). Figure 6D Also shown is a volume control enable indication 606C including an indicator 606M representing an “AirPods” device, indicating that the volume control enable indication 606C can be used to adjust the volume of the audio output of the external device “AirPods”.

[0365] Figure 6E Shown as Figure 6C , but with user input 608A on the device selection affordance 606B. In some embodiments, in response to receiving user input 608A, the device 600 displays a multi-device interface (e.g., Figure 6J For example, user input 608A is a touch (e.g., also referred to as a “tap” or “press”) on the device selection affordance.

[0366] Figure 6F Shown as Figure 6C 6B, but with user input 608B at the location on media control interface 606A. In some embodiments, in response to receiving user input 608B, device 600 displays a multi-device interface (e.g., Figure 6J ). In some examples, user input 608B is a deep press (e.g., a touch having a characteristic intensity that satisfies an intensity-dependent criterion). In some examples, the intensity-dependent criterion is whether the characteristic intensity of the touch exceeds a threshold intensity. In some examples, user input 608B is a tap and hold (e.g., continuous detection of touch input on the touch-sensitive surface of device 600 for a time period greater than a threshold). In some examples, user input 608B (e.g., whether a deep press or a tap and hold input) can be received anywhere on media control interface 606A (e.g., completely or partially within the media control interface). For example, in response to receiving user input 606B on a playback control affordance (e.g., 606M), device 600 performs an operation different from the operation it would have performed if the user input were a tap (e.g., device 600 performs a second operation instead of a playback control operation such as pausing playback).

[0367] Figure 6G An exemplary expanded media control interface of the device is shown. In some embodiments, in response to receiving user input 608B, the device 600 displays Figure 6G609 . As shown, the expanded media control interface 609 includes an indicator 609A representing the media currently being played back on the device 600. In this example, the device 600 is named "iPhone" and is currently playing the media track "Same Thing" by the artist "Jambug". The indicator 609A includes a graphic to the left of the track name, representing the album cover associated with the media track. In some examples, the graphic included in the indicator 609A can be any graphic associated with the media track or that can otherwise provide a visual representation of the media currently being played. For example, other suitable graphical indicators include, but are not limited to: artist photos, album covers, movie covers, movie posters, stylized text descriptions, etc. The exemplary expanded media control interface 609 includes a media progress indicator 609B (e.g., depicting the time progress (elapsed time) that the media track has been played and the amount of time (remaining time) until the current media ends). The exemplary expanded media control interface 609 also includes a playback control enable indication 609C (e.g., functioning similarly or identically to those described above), a volume control enable indication 609D (e.g., functioning similarly or identically to those described above), and a device selection enable indication 609E.

[0368] Figure 6H-Figure 6N An exemplary multi-device interface according to some embodiments is shown. Figure 6H Depicted is an exemplary multi-device interface 610 including affordances 610B through 610D for current device 600. In some embodiments, in response to receiving user input 608A, device 600 displays multi-device interface 610. In some embodiments, in response to receiving user input 608B, device 600 displays multi-device interface 610.

[0369] like Figure 6H As shown, as described above with respect to expanding media control interface 609, multi-device interface 610 includes an indicator 610A representing the media currently being played back on device 600 (named "iPhone"), a media progress indicator 610B (e.g., depicting the time progression (elapsed time) that the media track has been played and the amount of time until the current media ends (time remaining)). Exemplary multi-device interface 610 also includes playback control affordances 610C (e.g., functioning similarly or identically to those described above), volume control affordances 610D (e.g., functioning similarly or identically to those described above), and device selection affordances 610E. Figure 6G Compared to the expanded media control interface 609, the multiple device interface 610 provides the user with information about the media playback status of multiple devices and the ability to affect the media playback status of the multiple devices.

[0370] like Figure 6H As shown, the multi-device interface 610 also includes indicators associated with other playback devices. Indicator 610F corresponds to one or more devices named "Living Room" (or in a group of devices named "Living Room"). In some embodiments, a group of devices may include one or more devices. The name "Living Room" may be assigned to the group of devices based on their location in the user's home. For example, when one or more devices in the group named "Living Room" are configured (e.g., using device 600 and a home control application) (e.g., before first use), the name "Living Room" is assigned to the one or more devices in the group. Thus, when selecting or viewing playback status for devices throughout the home, the user is provided with intuitive identification of the group of devices by location. In some embodiments, the name of the device or device group is a default name. For example, if the user does not configure a device with a specific name, the device may be simply named "Device 1" or "Device 1-Living Room" (if location associated) (e.g., by the user or automatically). In some embodiments, the name of the device or device group is a user-configured name. For example, the user selects a name from a predefined set of names. In some examples, the name displayed for a device (e.g., in a multi-device interface) is the location of the device. For example, a device may be named "Tim's Speaker" (e.g., in a configuration setting or home control application) but appear as "Tim's Room" based on the location associated with the device. In some embodiments, a group of devices is custom named. For example, a user can assign a personalized name to a group of devices, such as "Tim's Room." In some embodiments, names other than location names are used. For example, a user can give a group of devices an arbitrary name.

[0371] In this example, a group of devices named Living Room, represented by indicator 610F, is currently playing back a media item titled "Heart Wave" by the artist "MKTS," with the artist and title indicated within indicator 610F. In some embodiments, the media item is one or more of a song, a video, a podcast, an audio stream, a video stream, an audio file, a video file, an electronic book (e.g., also known as an e-book), a text document, a web page, a phone call, a video call, and the like. Indicator 610G corresponds to a group of devices named "Bedroom" or included in a group named "Bedroom," which is currently playing back a song titled "Vibes" by the artist "John Appleseed." In some embodiments, indicators for a corresponding group of devices (e.g., indicators 610F and 610G) are displayed (e.g., in a multi-device interface) even when the group of devices is not currently playing back media. In some embodiments, indicators for a group of devices (e.g., indicator 610F and indicator 610G) are displayed (e.g., in a multi-device interface) even when not all devices in a group are currently playing media. For example, if a living room device group includes three devices (e.g., three smart speaker devices, such as device 580), and only one of those devices is currently playing a media item (e.g., a "Heart Wave" by "MKTS"), the corresponding indicator in the multi-device interface still includes an identification of the media currently being played back (e.g., a "Heart Wave" by "MKTS" is displayed inside indicator 610F for the device group "Living Room").

[0372] In some embodiments, if individual devices in the group of devices are playing back different media items at the same time, the multi-device interface displays an indication that multiple media items are being played on devices in the group of devices. For example, the multi-device interface may include a single indicator for the group (e.g., 610F) and an identification that multiple media items are being played back on these devices (e.g., "1 movie, 1 song"). For example, the multi-device interface may include two indicators for the group (e.g., 610F) and an identification of each corresponding media item being played back (e.g., an indicator for each media item such as 610F, both identified as "Living Room," or identified as "Living Room - Device 1" and "Living Room - Device 2").

[0373] A group of devices may also be referred to as a group of "available playback devices." In some embodiments, available playback devices include one or more (e.g., remote, external) devices on which a current device (e.g., device 600) can control or affect playback (e.g., using a multi-device interface such as 610). For example, Figure 6HAs shown, if a group of devices named Living Room includes three external devices (e.g., Figure 5I 580) and can be controlled by instructions issued by device 600 (e.g., representing user input), then the devices in the group named living room are "available playback devices". In some embodiments, affecting or controlling playback on available playback devices is subject to conditions or restrictions. For example, device 600 can see the devices in the living room represented by indicator 610F, but modifying the playback status of these devices may be subject to permissions (e.g., if the media being played back on the living room device is initiated by another device or user). In some embodiments, the available playback devices include the current device. For example, in the example where the living room includes three devices, the multiple available devices include device 600 and the three living room devices, for a total of at least four devices. Thus, the multi-device interface 610 provides a convenient interface that provides an indication of what is playing on the current device and other devices.

[0374] In some embodiments, one or more devices automatically detect another device. In some embodiments, the devices then form a network between them. For example, two devices (e.g., 580) may detect each other after first booting up (e.g., out of the box, not configured by the user), establish a connection link between them (e.g., and with device 600), and appear in a multi-device interface (e.g., with a name such as "Speaker 1" and / or a location such as "No Location"). In some embodiments, a device (e.g., 580) is configured to be part of a user's home (e.g., a home device). Those skilled in the art will appreciate that a variety of techniques may be used to interconnect and / or transmit information between devices (e.g., 580, 600), which are all contemplated to be within the scope of this disclosure and, therefore, are not included in additional detail herein.

[0375] exist Figure 6I , device 600 receives user input 612 representing a selection of device selection affordance 610E. For example, user input 612 can be a tap or touch at a location associated with displayed device selection affordance 610E.

[0376] Figure 6J In some embodiments, in response to receiving user input 608B, device 600 displays multi-device interface 610. In some embodiments, in response to receiving user input 612, device 600 displays multi-device interface 610. In some examples, in response to selecting an affordance for a device in any media application (e.g., Figure 6I 610E) selection to display the multi-device interface.

[0377] like Figure 6J As shown, the multi-device interface 610 includes separate indicators 610H through 610O for each of the available playback devices. In the depicted example, the separate indicators are visually associated with indicator 610A, which identifies the media item "Same Thing" by "Jambug" and the device that is currently playing back the media (in this case, only the device named "iPhone"). Indicator 610H identifies the current device (named "iPhone") and includes a selection indicator 610I. In this example, selection indicator 610I includes a check mark indicating that device 610H is currently playing back the media of indicator 610A. Indicator 610J identifies a group of devices named "Bedroom" and includes indicator 610K that does not include a check mark (e.g., the bedroom group devices are not currently playing back the media track "Same Thing" by "Jambug"). Indicator 610J also includes an indication that the bedroom group devices are currently playing back other media. For example, in Figure 6J , indicator 610J includes the title of the track "Vibes." The visual indication communicates to the user that the bedroom device group is currently busy. If the user wishes to add the bedroom device group to the current playback of "Same Thing" by "Jambug" (e.g., by selecting indicator 610K), doing so would interrupt the media currently playing in the bedroom. Indicator 610L identifies a group of devices named "Kitchen" and includes indicator 610M that does not include a check mark (e.g., the Kitchen group devices are not currently playing back the media track "Same Thing" by "Jambug"). Indicator 610N identifies a group of devices named "Living Room" and includes indicator 610O that does not include a check mark (e.g., the Living Room group devices are not currently playing back the media track "Same Thing" by "Jambug"). Indicator 610N also includes an indication that the Living Room group devices are currently playing back other media. For example, in Figure 6J , indicator 610J includes the title of the track "Vibes."

[0378] like Figure 6J As shown, the individual indicators are visually associated with indicator 610A. As can be seen, individual indicators 610H through 610O are included within the box outline of indicator 610A. In some examples, this provides a visual indication that the multi-device interface is currently "pointing to" or "focusing on" the "media session" identified by indicator 610A. In some embodiments, a media session includes devices and media items. Figure 6JIn the example shown, indicator 610A represents a media session that includes the media item "Same Thing" by "Jambug" and the device "iPhone" (e.g., device 600). In some embodiments, the media session includes multiple devices. In some embodiments, the media session includes multiple media items. For example, a media session may include playback of a playlist or other collection of multiple media items. Visually associating indicators for individual devices with indicators for media sessions visually indicates that selecting an individual device will add the device to the media session. This visual indication is useful, for example, when there are multiple concurrent media sessions in a user's home. For example, indicator 610F and indicator 610G each represent other media sessions (e.g., indicating playback of different media on different devices).

[0379] like Figure 6J As shown, the indicator for the separate device has replaced the media playback control enable representation (e.g., 610B to 610D). In some embodiments, the indicator for the separate device and the media playback control enable representation are displayed simultaneously.

[0380] In some embodiments, the indicator associated with a group of devices or an individual device includes an identification of the user who initiated playback of the media on the group of devices or the individual device, respectively. Figure 6J When a media session in the bedroom is initiated by a user (e.g., Steve) other than the user currently associated with device 600 (e.g., Tim), one or more of indicator 610G or indicator 610J may identify the user initiating the media playback in the bedroom—e.g., “Steve is playing Vibes by John Appleseed.”

[0381] Figure 6J The multi-device interface 610 may include an enable indication 610R. For example, selecting the enable indication 610R may cause the indicator for the individual device to cease being displayed (and, for example, the media playback control enable indication to be displayed again). In some examples, if the device 600 receives a selection of one or more individual indicators 610K, 610M, or 610O (e.g., user input causing the device 600 to display a check mark within the indicator), the selection of the enable indication 610R causes the device 600 to initiate media playback on the corresponding selected device in response. In some embodiments, the device 600 initiates playback in response to the selection of one or more individual indicators 610K, 610M, or 610O.

[0382] Figure 6KDevice 600 is shown receiving a selection of an individual indicator for a device from the available playback devices. User input 614 is received on indicator 610 associated with a kitchen group device (represented by 610L). In some examples, user input 614 is a tap or touch on a location associated with indicator 610M. In some examples, user input 614 is a deep press on a location associated with indicator 610M.

[0383] Figure 6L An exemplary multi-device interface 610 is shown after receiving a selection (eg, user input 614) of an individual indicator for a device among the available playback devices. Figure 6L As can be seen in the figure, indicator 610A now includes an indication that the current media session includes the kitchen group devices and the device named iPhone (e.g., device 600) ("Kitchen+iPhone"). In addition, indicator 610L has been updated to include a separate volume control enable representation, and indicator 610M now includes a check mark. In some examples, in response to a user input selection of its associated indicator 610M, the kitchen device group is added to the current media session of indicator 610A. In some examples, additional user input (e.g., user input 615) on enable representation 610R causes the kitchen devices to be added to the current media session of indicator 610A.

[0384] Figure 6M-Figure 6N FIG2 shows switching between media sessions when using an exemplary multi-device interface. Figure 6M , a multi-device interface 610 is shown with a first media session associated with indicator 610A (e.g., playback of "Same Thing" by artist "Jambug" on device "iPhone"). Figure 6M As shown, device 600 receives user input 616 associated with indicator 610F, which is associated with a second media session (e.g., playback of "Heart Wave" by artist "MKTS" on the "Living Room" group device). In some examples, user input 616 is a tap or touch at a location associated with indicator 610F. In some examples, user input 616 is a deep press at a location associated with indicator 610F.

[0385] In some embodiments, in response to user input 616, device 600 updates the display of the multi-device interface to "focus" (also referred to as "point to") the second media session. For example, Figure 6NIndicator 610A associated with the first media session is depicted as minimized (e.g., without visually associated indicators for individual available playback devices or media playback controls). However, indicator 610F has been expanded to include individual indicators for each of the plurality of available playback devices. Alternatively or in addition, expanded indicators for the media session (e.g., Figure 6N The indicator 610F) may include a media playback control affordance. Figure 6N As shown, indicators 610H, 610J, 610L, and 610N are now visually associated with indicator 610F for the second media session on the living room group device. In particular, indicator 610M and indicator 610O now include check marks, indicating that they are part of the currently selected media session (e.g., associated with expansion indicator 610F).

[0386] Switching between media sessions can be referred to as "re-pointing," "pointing," "refocusing," "focusing," etc. Thus, selection of indicator 610F can be referred to as "focusing" current device 600 or causing multi-device interface 610 to "focus" on a group of devices or media sessions.

[0387] As above relative to Figure 6H-Figure 6N As shown, you can use the multi-device interface to view the media playback status of multiple available playback devices. In addition, you can use the multi-device interface to view the content currently being played by each device in a group of available playback devices. In addition, you can use the multi-device interface to add or remove individual devices from the media session. Figure 6H-Figure 6N In addition to the examples shown, other exemplary techniques for invoking and interacting with a multi-device interface can be used, as described below.

[0388] Figure 6O-Figure 6Y Example techniques for interacting with a multi-device user interface are shown according to some embodiments. Figure 6O-6R Exemplary techniques for initiating playback directly onto one or a dynamically selected external device are shown. Figure 6O An exemplary browsing interface is shown. Browsing interface 620 depicts an interface for browsing media items. For example, a user can use the browsing interface to find and select one or more media items for playback. Exemplary browsing interfaces include interfaces for browsing a music library, a video library, media content that can be used for streaming, and the like. In this example, browsing interface 620 is a music browsing interface. Indicator 620A shows that the device is displaying a list of media items (e.g., a playlist) titled "New Music." Media item 620B (also referred to as song 620B) is a song titled "Monday Morning" by artist "The Fomoers."

[0389] exist Figure 6P , device 600 receives user input 622 on song 620B. In some examples, the user input representing the selection of the media item (e.g., a tap on a song) causes the playback of the song to be initiated on the current device. For example, if device 600 is currently only "pointing" to itself, a tap on the media item causes playback to be initiated on device 600. For another example, if device 600 is currently pointing to itself and the device group "living room", a tap on the media item causes playback to be initiated on device 600 and the group of devices named "living room". In some examples, a deep press user input or a tap and hold user input on a media item causes device 600 to perform a secondary operation (e.g., an operation other than immediately initiating playback of the corresponding media item on the one or more devices pointed to). In some examples, the secondary operation allows the user to select one or more devices on which media playback is to be initiated before playback of the corresponding media is initiated. In this way, the user can start playback directly on the desired set of devices without first initiating playback on the current device (e.g., device 600) and then transferring playback to one or more external devices (e.g., through a cumbersome interface arrangement). One of the disadvantages of first initiating playback on the current device (e.g., a personal device) and then transferring playback to other devices (e.g., a home theater system) is that the media item may have already started playing on the device before playback is initiated on the desired set of devices—an undesirable situation if playback is being performed for a room full of listeners. Even if playback is restarted (manually or automatically) in response to or after transferring playback to the desired set of devices, the original device that initiated playback will waste device resources by starting playback of the song, which is undesirable for the user. In some examples, a tap input invokes a primary operation, and a deep press or tap and hold input invokes a secondary operation. Those skilled in the art will recognize that any two user inputs that can be recognized by the device as different types of user inputs can be used to invoke corresponding primary and secondary operations.

[0390] Figure 6Q An exemplary media item menu is shown. For example, in response to user input 622, which is a deep press or a tap and hold input (e.g., to invoke a secondary action related to a media item), a media item menu 624 is displayed. Menu 624 includes a list of possible actions that can be selected for media item 620B (the song titled "Monday Morning" by "The Fomoers"). Media item menu 624 includes a play on speaker affordance 624A. Figure 6R, device 600 receives user input 626 on play-to-speaker enable indication 624A. For example, user input 626 is a tap. In some examples, in response to receiving user input 626 indicating a selection of play-to-speaker enable indication 624A, device 600 displays a multi-device interface (e.g., similar to Figure 6J ). For example, a multi-device interface may be displayed that includes an indicator (e.g., 610A) for a media session that includes the media item "Monday Morning" by "The Fomoers" and no selected devices (e.g., devices with indicators that do not include a check mark). In this example, after the one or more desired devices are selected (e.g., adding them to the media session), playback is then initiated on the desired devices (e.g., the device 600 initiates playback on itself and / or transmits instructions to other devices in the group of desired devices). In this example, user input (e.g., 622) associated with the media item is used to invoke the multi-device interface (e.g., 610) so that one or more external devices can be selected before the device 600 initiates playback of the media. Thus, the device 600 initiates playback of the media item (e.g., as shown in FIG. 1 ) from the beginning (e.g., from the beginning of the song) on ​​the selected one or more external devices. Figure 6P ” Monday Morning” shown), rather than first initiating playback on device 600 and then subsequently transmitting the playback (e.g., during playback).

[0391] Figure 6S-6Y Example techniques for providing an indication of device focus are shown. Figure 6S An exemplary current media interface is shown. Current media interface 630 (also referred to as the "now playing interface") identifies the media item currently being played. Current media interface 630 includes album art 630A, media identifier 630B (song title "Same Thing"), media identifier 630C (artist "Jambug"), media identifier 630D (album "Fly Away"), and device selection affordance 630E. Figure 6S As shown, device selection affordance 630E does not include an identifier for another device. Thus, in this example, the media item identified in current media interface 630A is currently playing on the current device (device 600). In some examples, the media identified in the current media interface is currently playing on another device (e.g., in addition to the current device or in a group that excludes the current device).

[0392] exist Figure 6TIn FIG. 6 , device 600 receives user input 632 on device selection enable indication 630E. In response to receiving user input 632, Figure 6U In the example, the device displays a multi-device interface 610. Figure 6U , a first area 611A of the display (e.g., shown with a first dashed pattern for illustrative purposes) includes the multi-device interface 610, and a second area 611B of the display (e.g., shown with a second dashed pattern for illustrative purposes) includes a visible indication of content that was displayed before the multi-device interface was displayed. In this case, the content that was displayed before the multi-device interface was displayed is the current media interface 630. For example, album art 630A is visible through the multi-device interface. In this example, the first area visually overlays the second area.

[0393] exist Figure 6V , device 600 receives user input 634 on indicator 610F. In some examples, user input 634 is a tap or touch at a location associated with indicator 610F. In some examples, user input 634 is a deep press at a location associated with indicator 610F.

[0394] like Figure 6W As shown, in some examples, in response to user input 634, device 600 displays multi-device interface 610, in which indicator 610F is expanded to show individual indicators for devices in the group of available playback devices that are visually associated with indicator 610F. Thus, device 600 is now focused on the media session that includes the device group living room. Figure 6W , the current media interface 636 (described below) is visible (via the multi-device interface 610). In particular, Figure 6W As seen in FIG, in response to input 634, device 600 stops displaying the visual indication associated with current media interface 630 and displays the visual indication associated with current media interface 636. This is a replacement of the current media interface, indicating that the device changes the media session being pointed to from the session associated with indicator 610A to the session associated with indicator 610F ( Figure 6U ).

[0395] like Figure 6X As shown, in some examples, in response to user input 634, device 600 stops displaying multi-device interface 610 and displays current media interface 636. Figure 6XAn alternative response of device 600 to receiving user input 634 is shown. Similar to interface 630, current media interface 636 includes album art 636A, media identifier 636B (song title "Heart Wave"), media identifier 636C (artist "MKTS"), media identifier 636D (album "MKTS"), and device selection affordance 636E. However, each of these indicators has been updated to reflect that the media items and devices of the selected media session have changed (e.g., because device 600 is now pointing to a media session that includes a different set of devices). In particular, device selection affordance 636E includes identifier 636F that identifies the currently pointed device. Figure 6X As shown, identifier 636F includes the text "Living Room," which indicates that current media interface 636 is the current media interface for media currently playing back on a group of devices named Living Room.

[0396] Even when viewing an interface for a non-media related application on their device, a user may want to view the media status of multiple devices in their home at any time. Therefore, this document describes a technology for easily and quickly accessing a multi-device interface from any application executing on a device. Figure 6Y-6AA Example techniques for accessing a multi-device interface are shown according to some embodiments. Figure 6Y An exemplary interface for an application is shown. Web browsing interface 640 depicts a currently displayed web browsing session (e.g., the home page of the website "Apple.com"). Web browsing interface 640 includes web page content 640A. Figure 6Z In some examples, device 600 receives user input 642. In some examples, user input 642 is a swipe gesture originating from an edge of a touch-sensitive surface of device 600. In some examples, in response to user input 642, device 600 displays a playback control interface. For example, in response to user input 642, device 600 displays Figure 6C In some embodiments, in response to receiving user input 608A on a device selection enable representation 606B at playback control interface 606A, device 600 displays a multi-device interface (e.g., Figure 6AA 610).

[0397] Figure 6AA An exemplary multi-device interface visually overlaid on an application interface is shown. Figure 6AA As shown, the multi-device interface 610 includes separate indicators for available playback devices. Figure 6AAIn the depicted example, only one media session is active on the available devices in the user's home, and that media session includes playback of "Same Thing" by "Jambug" on the current device 600. In some examples, the multi-device interface is displayed concurrently with the previously displayed content. For example, Figure 6AA As shown, the multi-device interface 610 (eg, displayed in the first area) is visually overlaid on the web browsing interface 640 (eg, displayed in the second area).

[0398] Figure 6AB-Figure 6AC Exemplary techniques for adjusting the volume of available playback devices are shown. Figure 6AB A multi-device interface 610 is depicted with a single media session including playback of a media item "Same Thing" by "Jambug" on three devices: an "iPhone" (eg, device 600), a kitchen group device, and a living room group device. Figure 6AB Depicted are individual volume control affordances 610S (corresponding to device iPhone), individual volume control affordances 610T (corresponding to device group Kitchen), and individual volume control affordances 610U (corresponding to device group Living Room). For example, the individual volume control affordances are selectable and can be manipulated (e.g., left and right) to adjust the volume of their respective devices. In some examples, individual volume control affordances are not displayed for devices not included in the current media session (e.g., Figure 6AB The device in the bedroom does not correspond to the separate volume control affordance shown). Figure 6AB Also depicted is a group volume control affordance 610D that is selectable and can be manipulated (e.g., left and right) to adjust the volume of multiple devices included in the media session represented by indicator 610A. Figure 6AB , device 600 receives user input 644 on group volume control affordance 610D, which represents a tap and slide gesture in a left direction (e.g., representing a request to turn down the volume). In response to user input 644, the device turns down the group volume represented by the position of sliding affordance 610D, as shown. Figure 6AC shown. Figure 6AC Also shown are individual volume control indicators for each device in the group, each having been correspondingly turned down.

[0399] Figures 6AD-6AGAn exemplary interface for initiating playback on a preferred device is shown. In some cases, a user may desire to automatically initiate playback of media at one or more preferred devices. For example, because media content may include visual aspects (e.g., if the media is a video), it is desirable to automatically initiate playback of such content on a device with a display or on a device with a large display (e.g., larger than the device receiving the user input request for initiating playback). For example, a user may browse media content using a personal device (e.g., device 600) with a relatively small screen, but desire to start playback of a selected media item (e.g., a feature film) on their large-screen TV. In other examples, a user may request playback of media via voice commands to a device (e.g., device 580) that does not have video playback capabilities. In this latter example, a device that does not have video playback capabilities is able to receive the user's request, but hands over video playback responsibility to another device.

[0400] Go to the attached figure, Figure 6AD An exemplary browsing interface is shown. Figure 6AD As shown, media browsing interface 650 includes media items with video content. In this example, media item 650A represents an episode of a television program titled "Super Heroes." Figure 6AE , device 600 receives user input 652 associated with media item 650A. In some examples, user input 652 is a tap or touch on media item 650A. In some examples, user input 652 is a deep press on media item 650A.

[0401] In response to receiving user input 652, device 600 displays Figure 6AFMessage interface 654 in . Message interface 654 includes an indication that playback of video content is or has been initiated on a different device than device 600 ("Start playback on Apple TV"). In this example, the device named "Apple TV" is different from device 600 and represents a preferred device for playback of video media. For example, device Apple TV can be connected to the user's large-screen television. In some embodiments, in response to user input (e.g., user input 652), a device (e.g., 600) determines whether it is currently connected to a home network. Based on determining that the device is currently connected to a home network, the device initiates playback on the preferred device (e.g., transmits an instruction to initiate playback on the preferred device). For example, if the device determines that the user is at home, it abandons initiating playback on itself and initiates playback on an external preferred device. Based on determining that the device is not currently connected to the home network, the device initiates playback on itself. For example, if the device determines that the user is not at home, it initiates playback on itself. In some examples, the preferred device is a group of preferred devices (e.g., one or more). In some examples, the preferred device (or group of devices) is predetermined. For example, a user may select a preferred device and save it to the device or user account settings. In another example, the preferred device is automatically selected. For example, if the user's home includes only one identified external device (e.g., Apple TV) that is capable of playing back videos, it may be designated as the preferred device. In some embodiments, the device prompts the user whether to initiate playback on the preferred device. For example, regardless of whether the user has pre-configured a preferred device, in response to user input 652, the device 600 may prompt (e.g., visually or audibly) and ask, "Do you want to initiate playback of "Super Heroes" on your Apple TV?"

[0402] For example, each device (e.g., a group of devices such as each device in a group of preferred devices, such as device 600 and device Apple TV) generates and broadcasts a set of one or more values ​​(e.g., in response to user input such as 652). In this example, each group of one or more values ​​includes a "device type" value. The device (e.g., 600) analyzes the broadcasted "device type" value based on the task specified in the user request (e.g., 652). The device (e.g., 600) makes a determination based on the device type broadcast by other electronic devices (e.g., Apple TV) and a list of acceptable device types. In some examples, if the device (e.g., 600) determines that none of the other electronic devices has an acceptable device type for processing the task, the device outputs an error message to the user or prompts the user for additional input (e.g., "Do you want to play back this video on your iPhone?").

[0403] Figure 6AGAn exemplary playback control interface for video content is shown. Playback control interface 656 includes an indication 656A of the media content currently being played ("Super Heroes"), playback control affordances 656B (e.g., rewind 10 seconds, pause, and skip 10 seconds), and an indication 656C of the device currently playing the media content ("Apple TV"). In some embodiments, interface 656 is displayed after message interface 654.

[0404] Figure 6AH-Figure 6AQ An exemplary multi-device interface is shown. As described herein, a multi-device interface can be used to view and control the media playback status of multiple available playback devices in a group of interconnected devices (e.g., devices in the user's home associated with the user's account). Thus, a variety of different types of devices can be controlled, and each device can have a different interface. For example, a group of devices in a home may include a first smart speaker (e.g., 580) manufactured by a first manufacturer, a second smart speaker (e.g., 580) manufactured by a second manufacturer different from the first manufacturer, and a home media hub (e.g., Apple TV developed by Apple Inc. (Cupertino, California, USA)). The exemplary home media hub may include or otherwise be connected to a large display (e.g., a display larger than the personal device 600) and therefore may have interfaces that are the same, similar, or different than those described above with respect to the device 600. Additional exemplary interfaces are described in detail below.

[0405] Figure 6AH An exemplary current media interface 664 is shown displayed on display 660A of device 660. In some embodiments, device 660 includes one or more features of device 100, 300, or 500. Device 660 is itself a device included in a set of devices associated with the user's home (or is otherwise configured to act as a display for a device included in a set of devices associated with the user's home). Current media interface 664 includes device selection affordance 664A (e.g., which is associated with Figure 6SDevice selection affordance 664A is associated with the text "Living Room Apple TV," which indicates that device 660 is currently pointing at itself (i.e., device 660). Current media interface 664 also includes an indication of the media currently playing (e.g., indicator 664C), which is a song titled "Heart Wave" (e.g., indicator 664D) by the artist "MKTS" (e.g., indicator 664E). In some examples, indicator 664C of currently playing interface 664 is an animated indicator that further conveys that the associated media item is currently playing. Indicator 664F and indicator 664G correspond to one or more other media items available for playback.

[0406] Figure 6AH Also shown is an exemplary remote control for controlling device 660. In some examples, remote control 662 is connected to device 660 (e.g., paired) and used by a user to interact with an interface displayed on device 660. Because the remote control is effectively an extension of the device, used to remotely control the device, for purposes of this disclosure, user input received at device 662 is considered the same as user input received at device 660, unless otherwise noted herein. For example, a tap input received on device 662 representing a request to select an affordance is considered the same as a tap on device 660 representing a request to select the same affordance (e.g., if device 660 includes a touch-sensitive surface), unless otherwise noted. However, those skilled in the art will understand that user input may be received at any device and communicated to another device (e.g., for processing and performing an action in response), and that the particular device that physically receives the user input is unimportant unless specifically noted. Similarly, in the example where device 660 is an external display device (e.g., a television) used by another device (e.g., an Apple TV that is a "home device" that receives and processes user input playback commands), because device 660 is effectively an extension of the other device, references herein to device 660 and the other device are interchangeable unless otherwise noted. For example, with respect to controlling media, references to the "focus" or "pointing" of device 660 refer to the focus or pointing of the other device (e.g., Apple TV) unless otherwise noted.

[0407] exist Figure 6AH, device 662 receives user input 668 on device 662. User input 668 represents a selection of device selection enable representation 664A. In some examples, user input 668 is a button press on device 662, and device selection enable representation 664A is highlighted or otherwise selected at interface 664. In some examples, user input 662 is a tap, tap and hold, or deep press on a touch-sensitive surface. In some embodiments, in response to receiving user input 668, device 660 displays a multi-device interface (e.g., Figure 6AI of 670).

[0408] Figure 6AI An exemplary multi-device interface is shown according to some embodiments. The multi-device interface 670 includes an identification of the current focus of the device 660 (e.g., the currently selected media session). Header 670A indicates that the current focus of the device is the living room Apple TV (e.g., device 660). The multi-device interface 670 also includes a first indicator 670B, which indicates the currently pointed media session. For example, the media session represented by indicator 670B includes playback of a song titled "Heart Wave" by the artist "MKTS" from the album "MKTS". Header 670A and indicator 670B (which may be collectively referred to as indicators) together indicate the media session that the interface 670 is currently pointed to (e.g., the media being played back and the device performing the playback). The multi-device interface 670 also includes a second indicator 670C, which corresponds to a media session that is different from the currently selected media session. As Figure 6AI As shown, indicator 670C is displayed (eg, partially displayed) on an edge of the display.

[0409] As described above with respect to multi-device interface 610, multi-device interface 670 includes separate indicators representing available playback devices (e.g., devices in the user's home). Indicator 670D identifies the current device (named "Living Room Apple TV") and includes a selection indicator 670K. In this example, selection indicator 670K includes a check mark indicating that device 670D is currently playing back the media...

Claims

1. A computer-implemented method comprising: At a first device (2700) associated with a user account, wherein the first device (2700) is logged into the user account: receiving user input representing a request to access data from the user account; In response to receiving the user input, determining whether a physical location of the first device (2700) and a second device (2702) associated with the user account satisfies a licensing condition, wherein the second device (2702) is logged into the user account; Based on determining that the physical location satisfies the permission condition, providing the data from the user account via the first device (2700); and Based on determining that the physical location does not satisfy the permission condition, abandoning the provision of the data from the user account through the first device (2700).

2. The method of claim 1, wherein providing, by the first device (2700), the data from the user account comprises: An audio signal representative of the data from the user account is output.

3. The method of claim 2, wherein outputting an audio signal representing the data from the user account comprises: A text-to-speech algorithm is used to cause an audible output of a dictation representing the data from the user account.

4. The method of claim 3, wherein the data from the user account includes data from one or more of: calendar entries, notifications, emails, text messages, stored text documents, stored contact information, telephone call audio, video call audio or video, voice messages, addresses, and media items.

5. The method according to any one of claims 1 to 4, wherein determining whether the physical locations of the first device and the second device satisfy the permission condition comprises: A determination is made as to whether a physical proximity between the first device and the second device satisfies a proximity condition.

6. The method according to any one of claims 1 to 5, wherein determining whether the physical locations of the first device and the second device satisfy the permission condition comprises: A determination is made as to whether a physical orientation of the first device satisfies an orientation condition.

7. The method according to any one of claims 1 to 6, further comprising: Further based on determining that the physical location does not satisfy the permission condition, an indication that the permission condition is not satisfied is output.

8. The method of claim 7, wherein outputting the indication that the permission condition is not satisfied comprises: An audible indication of a user action required to satisfy the permission condition is output.

9. The method of any one of claims 1-8, wherein the user input representing a request to access data from the user account is a first user input, the method further comprising: After providing the data from the user account via the first device: receiving a second user input representing a request to access additional data from the user account; in response to receiving the second user input, determining whether the physical locations of the first device and a second device associated with the user account satisfy the permission condition; providing, via the first device, the additional data from the user account based on determining that the physical location satisfies the permission condition; Based on determining that the physical location does not satisfy the permission condition, providing the additional data from the user account through the first device is foregone.

10. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device (2700), wherein the electronic device is a first electronic device associated with a user account, wherein the first electronic device is logged into the user account, and the one or more programs include instructions for executing a method according to any one of claims 1-9.

11. An electronic device (2700), comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1-9.

Images