User interface for audio routing

Abstract

The present disclosure generally relates to user interfaces for audio routing. In some examples, a computer system detects an audio device and displays selectable options in accordance with a determination that a set of criteria is satisfied. The selectable option, when selected, enables the computer system to connect to and transmit audio data to the audio device.

Description

[0001] This application is a divisional application of an invention patent application with an international application date of May 27, 2022, a national application number of 202280040345.2 (international application number PCT / US2022 / 031252), and an invention name of “User Interface for Audio Routing”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. application serial number 17 / 824,510, filed on May 25, 2022, entitled “USER INTERFACES FOR AUDIOROUTING,” and U.S. provisional application serial number 63 / 197,425, filed on June 6, 2021, entitled “USER INTERFACESFOR AUDIO ROUTING,” the entire contents of each of which are hereby incorporated by reference. Technical Field

[0004] The present disclosure relates generally to computer user interfaces and, more particularly, to techniques for managing audio output. Background Art

[0005] Electronic devices can play various types of media, including audio, such as music tracks, podcasts, audiobooks, and videos. Electronic devices can also provide audio data to audio output devices such as wireless speakers and wireless headphones via wireless connections. Summary of the invention

[0006] However, some techniques for managing audio output using electronic devices are often cumbersome and inefficient. For example, some prior art techniques use complex and time-consuming user interfaces that may include multiple keystrokes or keystrokes. Prior art techniques require more time than necessary, which results in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, the technology of the present invention provides a faster and more efficient method and interface for managing audio output for electronic devices. Such methods and interfaces optionally supplement or replace other methods for managing audio output. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. Such methods and interfaces also reduce the number of inputs necessary to connect audio devices to electronic devices. For battery-driven computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0008] According to some embodiments, a method is described. The method is performed at a computer system in communication with a display generation component. The method includes: while displaying a first type of content via the display generation component, detecting an audio device; and in response to detecting the audio device: based on determining that a set of criteria are met, displaying a selectable option, the selectable option causes the computer system to provide output content audio to the audio device when selected via user input; and based on determining that the set of criteria are not met, abandoning displaying the selectable option.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, and the one or more programs include instructions for performing the following operations: when displaying a first type of content via the display generation component, detecting an audio device; and in response to detecting the audio device: based on determining that a set of criteria are met, displaying a selectable option, the selectable option causes the computer system to provide output content audio to the audio device when selected via user input; and based on determining that the set of criteria are not met, abandoning displaying the selectable option.

[0010] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component, the one or more programs including instructions for performing the following operations: when displaying a first type of content via the display generation component, detecting an audio device; and in response to detecting the audio device: based on determining that a set of criteria are met, displaying a selectable option, the selectable option causes the computer system to provide output content audio to the audio device when selected via user input; and based on determining that the set of criteria are not met, abandoning display of the selectable option.

[0011] According to some embodiments, a computer system is described. The computer system is configured to communicate with a display generation component. The computer system includes: one or more processors; and a memory, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: when displaying a first type of content via the display generation component, detecting an audio device; and in response to detecting the audio device: displaying a selectable option based on determining that a set of criteria are met, the selectable option causes the computer system to provide output content audio to the audio device when selected via user input; and abandoning displaying the selectable option based on determining that the set of criteria are not met.

[0012] According to some embodiments, a computer system is described. The computer system is configured to communicate with a display generation component. The computer system includes: a device for detecting an audio device when displaying a first type of content via the display generation component; and a device for performing the following operations in response to detecting the audio device: displaying a selectable option based on determining that a set of criteria are met, the selectable option causes the computer system to provide output content audio to the audio device when selected via user input; and abandoning displaying the selectable option based on determining that the set of criteria are not met.

[0013] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component, the one or more programs including instructions for performing the following operations: when displaying a first type of content via the display generation component, detecting an audio device; and in response to detecting the audio device: based on determining that a set of criteria are met, displaying a selectable option, the selectable option causes the computer system to provide output content audio to the audio device when selected via user input; and based on determining that the set of criteria are not met, abandoning display of the selectable option.

[0014] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0015] Thus, a faster, more efficient method and interface for managing audio output is provided for devices, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace other methods for managing audio output. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0017] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0018] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0019] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0020] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0021] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to some embodiments.

[0022] Figure 4B An exemplary user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0023] Figure 5A A personal electronic device according to some embodiments is shown.

[0024] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0025] FIG. 6A to FIG. 6I An exemplary user interface for managing audio output according to some embodiments is shown.

[0026] 7A to 7C is a flowchart illustrating a method for managing audio output according to some embodiments. DETAILED DESCRIPTION

[0027] The following description sets forth exemplary methods, parameters, etc. However, it should be appreciated that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0028] There is a need for electronic devices that provide efficient methods and interfaces for managing audio output. For example, when viewing content on an electronic device, a user typically must provide a set of user inputs to cause the electronic device to switch audio output between a first audio device and a second audio device. The disclosed techniques reduce the number of user inputs required for a user to switch audio output between audio devices. Such techniques can reduce the cognitive burden on a user managing audio output, thereby increasing productivity. In addition, such techniques can reduce processor power and battery power that would otherwise be wasted on redundant user inputs.

[0029] under Figure 1A to Figure 1B , Figure 2 , Figure 3 , FIG. 4A to FIG. 4B and FIG. 5A to FIG. 5B A description of an exemplary device for performing techniques for management event notifications is provided. FIG. 6A to FIG. 6I An exemplary user interface for managing audio output is shown. 7A to 7C is a flow chart illustrating a method of managing audio output according to some embodiments. FIG. 6A to FIG. 6I The user interface in is used to illustrate the process described below, including 7A to 7C process.

[0030] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device more quickly and more efficiently.

[0031] In addition, in the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0032] Although the following description uses the terms "first", "second", etc. to describe various elements, these elements should not be limited by the terms. In some embodiments, these terms are used to distinguish one element from another element. For example, a first touch can be named a second touch and similarly a second touch can be named a first touch without departing from the scope of the various described embodiments. In some embodiments, the first touch and the second touch are two separate references to the same touch. In some embodiments, both the first touch and the second touch are touches, but they are not the same touch.

[0033] The terms used in the description of various described embodiments in this article are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" and / or "comprising" are used in this specification to specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0034] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]," or "in response to detecting [a stated condition or event]," depending on the context.

[0035] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Optionally use other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads). In some embodiments, the electronic device is a computer system that communicates with the display generation component (e.g., via wireless communication, via wired communication). The display generation component is configured to provide visual output, such as display via a CRT display, display via an LED display, or display via image projection. In some embodiments, the display generation component is integrated with the computer system. In some embodiments, the display generation component is separated from the computer system. As used herein, "display" content includes transmitting data (e.g., image data or video data) to an integrated or external display generation component via a wired or wireless connection to visually generate content to display content (e.g., video data rendered or decoded by display controller 156).

[0036] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0037] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0038] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0039] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1Ais a block diagram showing a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0040] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited real estate, which is used to display an enable indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0041] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0042] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the EMBODIMENTS 2000 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0043] Memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0044] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs (such as computer programs (e.g., including instructions)) and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0045] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memories, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuits for detecting near field communications (NFC) fields, such as through short-range communications radio components. Wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11d), IEEE 802.11e, IEEE 802.11f, IEEE 802.11g, IEEE 802.11g). 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols that have not been developed as of the filing date of this document.

[0046] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0047] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, a depth camera controller 169, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some embodiments, the input controller 160 is optionally coupled to any of the following (or none of the following): a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an increase / decrease button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2206 in). In some embodiments, the electronic device is a computer system that communicates with one or more input devices (e.g., via wireless communication, via wired communication). In some embodiments, the one or more input devices include a touch-sensitive surface (e.g., a touchpad as part of a touch-sensitive display). In some embodiments, the one or more input devices include one or more camera sensors (e.g., one or more optical sensors 164 and / or one or more depth camera sensors 175), such as for tracking a user's gestures (e.g., hand gestures and / or air gestures) as input. In some embodiments, the one or more input devices are integrated with the computer system. In some embodiments, the one or more input devices are separate from the computer system. In some embodiments, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and is based on detected movement of a part of the user's body through the air, including movement of the user's body relative to an absolute reference (for example, the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (for example, movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (for example, a tap gesture involving moving the hand in a predetermined posture by a predetermined amount and / or speed, or a shake gesture involving a predetermined speed or amount of rotation of a part of the user's body).

[0048] A quick press of the push button optionally disengages the lock of the touch screen 112 or optionally begins the process of unlocking the device using gestures on the touch screen, as described in U.S. Patent Application 11 / 322,549, filed on December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image" (i.e., U.S. Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0049] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0050] The touch screen 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact on the touch screen 112 (and any movement or interruption of the contact) and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to a finger of the user.

[0051] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® from Apple Inc. (Cupertino, California). and iPod The technology used in.

[0052] The touch-sensitive display in some embodiments of touch screen 112 is optionally similar to the multi-touch-sensitive touchpad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, touch screen 112 displays visual output from device 100, while a touch-sensitive touchpad does not provide visual output.

[0053] The touch-sensitive display in some embodiments of the touch screen 112 is described in the following applications: (1) U.S. patent application No. 11 / 381,313, filed on May 2, 2006, "Multipoint Touch Surface Controller"; (2) U.S. patent application No. 10 / 840,862, filed on May 6, 2004, "Multipoint Touchscreen"; (3) U.S. patent application No. 10 / 903,964, filed on July 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) U.S. patent application No. 11 / 048,264, filed on January 31, 2005, "Gestures For Touch Sensitive Input Devices"; (5) U.S. patent application No. 11 / 038,590, filed on January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices”; (6) U.S. patent application No. 11 / 228,758, filed on September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. patent application No. 11 / 228,700, filed on September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. patent application No. 11 / 228,737, filed on September 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. patent application No. 11 / 367,749, filed on March 3, 2006, “Multi-Functional Hand-Held Device”. All of these applications are incorporated herein by reference in their entirety.

[0054] The touch screen 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or attachment such as a stylus, a finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0055] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch screen 112 or is an extension of the touch-sensitive surface formed by the touch screen.

[0056] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0057] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures a static image or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to the touch screen display 112 on the front of the device, so that the touch screen display can be used as a viewfinder for static images and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device so that the user's image is optionally obtained for video conferencing while the user views other video conference participants on the touch screen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that a single optical sensor 164 is used with the touch screen display for both video conferencing and static image and / or video image acquisition.

[0058] Device 100 optionally also includes one or more depth camera sensors 175 . Figure 1A A depth camera sensor coupled to a depth camera controller 169 in the I / O subsystem 106 is shown. The depth camera sensor 175 receives data from the environment to create a three-dimensional model of an object (e.g., a face) within the scene from a viewpoint (e.g., a depth camera sensor). In some embodiments, in conjunction with the imaging module 143 (also referred to as a camera module), the depth camera sensor 175 is optionally used to determine a depth map of different portions of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is located at the front of the device 100, so that an image of the user with depth information is optionally obtained for video conferencing while the user is viewing other video conference participants on the touch screen display, and a selfie with depth map data is captured. In some embodiments, the depth camera sensor 175 is located at the rear of the device, or at both the rear and front of the device 100. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that the depth camera sensor 175 is used with the touch screen display for both video conferencing and still image and / or video image acquisition.

[0059] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1AA contact force sensor is shown coupled to a force sensor controller 159 in the I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite to touch screen display 112 located on the front of device 100.

[0060] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent application numbers: 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0061] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in parallel or adjacent to a touch-sensitive surface (e.g., a touch-sensitive display system 112) and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0062] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 is optionally implemented as described in the following U.S. Patent Publication Nos.: 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0063] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and position information related to the device's position and / or posture.

[0064] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitating communication between various hardware components and software components.

[0065] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to (trademark of Apple Inc.) devices.

[0066] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touch pad.

[0067] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0068] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0069] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0070] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0071] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0072] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0073] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services, such as a weather widget, a local yellow pages widget, and a map / navigation widget).

[0074] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0075] ● Contacts module 137 (sometimes called address book or contact list);

[0076] ● Telephone module 138;

[0077] ● Video conferencing module 139;

[0078] ● Email client module 140;

[0079] ●Instant messaging (IM) module 141;

[0080] ●Fitness support module 142;

[0081] ● Camera module 143 for still images and / or video images;

[0082] ●Image management module 144;

[0083] ●Video player module;

[0084] ●Music player module;

[0085] ●Browser module 147;

[0086] ●Calendar module 148;

[0087] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock market widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0088] A widget creator module 150 for forming a user-created widget 149-6;

[0089] ●Search module 151;

[0090] ● Video and music player module 152, which merges the video player module and the music player module;

[0091] ●Note module 153;

[0092] ● Map module 154; and / or

[0093] ●Online video module 155.

[0094] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0095] In conjunction with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and the like.

[0096] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a sequence of characters corresponding to a phone number, access one or more phone numbers in contact module 137, modify an entered phone number, dial a corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0097] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, the text input module 134, the contact module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0098] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0099] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0100] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154 and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing and transmitting fitness data.

[0101] In conjunction with the touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132 and image management module 144, the camera module 143 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0102] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0103] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with user instructions, including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0104] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0105] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is a mini-application that is optionally downloaded and used by a user (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm widget 149-4, and dictionary widget 149-5) or a mini-application created by a user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! widget).

[0106] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the widget creator module 150 is optionally used by a user to create widgets (e.g., converting a user-specified portion of a web page into a widget).

[0107] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0108] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0109] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the note module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0110] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the GPS module 135 and the browser module 147, the map module 154 is optionally used to receive, display, modify and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0111] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for performing the following operations: allowing a user to access, browse, receive (e.g., by streaming and / or downloading), playback (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a specific online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a specific online video. Additional descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0112] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs (such as computer programs (e.g., including instructions)), processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0113] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0114] A predefined set of functions that are performed exclusively through the touch screen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0115] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0116] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event distributor module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0117] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0118] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0119] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0120] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0121] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0122] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events that are recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0123] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0124] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0125] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receivers 182.

[0126] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0127] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0128] The corresponding event identifier 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event based on the event information. The event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0129] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (for example, from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0130] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sub-event sequence), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in events (e.g., 187-1 and / or 187-2) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0131] In some embodiments, event definition 186 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and the object that triggered the hit test.

[0132] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0133] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0134] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0135] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and executes a predefined process.

[0136] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and executes a predetermined process.

[0137] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0138] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0139] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events that define the event to be distinguished.

[0140] Figure 2A portable multifunction device 100 with a touch screen 112 according to some embodiments is shown. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more of these graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, when the user interrupts contact with one or more graphics, selection of one or more graphics will occur. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic will not select the graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0141] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0142] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the strength of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0143] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference Figure 1A The tactile output generator 167 described above), sensor 359 (e.g., optical sensor, acceleration sensor, proximity sensor, touch sensor and / or contact intensity sensor (similar to the above reference Figure 1A The memory 370 may include a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and may optionally include a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 may optionally include one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores data related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0144] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the previously mentioned memory device. Each module in the above-mentioned modules corresponds to an instruction set for performing the above-mentioned functions. The above-mentioned modules or computer programs (for example, instruction sets or including instructions) do not have to be implemented with separate software programs (such as computer programs (for example, including instructions)), processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0145] Attention is now turned to an embodiment of a user interface that is optionally implemented on, for example, portable multifunction device 100.

[0146] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 according to some embodiments is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0147] ● Signal strength indicators 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0148] ●Time 404;

[0149] ●Bluetooth indicator 405;

[0150] ●Battery status indicator 406;

[0151] A tray 408 with icons for commonly used applications, such as:

[0152] o An icon 416 of the phone module 138 labeled "Phone," which optionally includes an indicator 414 of the number of missed calls or voicemails;

[0153] o an icon 418 of the email client module 140 labeled “Mail”, the icon 418 optionally including an indicator 410 of the number of unread emails;

[0154] o An icon 420 labeled "Browser" of the browser module 147; and

[0155] o An icon 422 labeled “iPod” for a video and music player module 152 (also referred to as an iPod (trademark of Apple Inc.) module 152); and

[0156] ● Icons for other applications, such as:

[0157] o Icon 424 labeled "Message" of IM module 141;

[0158] o An icon 426 labeled “Calendar” of the calendar module 148;

[0159] o Icon 428 labeled “Photos” of the image management module 144;

[0160] o An icon 430 labeled “Camera” of the camera module 143;

[0161] ○ Icon 432 labeled “Online Video” of the online video module 155;

[0162] ○ Icon 434 labeled “Stock Market” of the Stock Market Widget 149 - 2 ;

[0163] o An icon 436 labeled “Map” of the map module 154;

[0164] ○ Icon 438 labeled “Weather” of the weather widget 149-1;

[0165] ○ Icon 440 labeled as “Clock” of the alarm clock widget 149 - 4 ;

[0166] o An icon 442 labeled “Fitness Support” of the fitness support module 142;

[0167] o An icon 444 labeled "Notes" of the notes module 153; and

[0168] o An icon 446 labeled “Settings” for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0169] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0170] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, Figure 3 tablet computer or touch pad 355) device (e.g., Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0171] Although some of the examples below are given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the principal axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B 460 corresponds to 468 and 462 corresponds to 470) in contact with touch-sensitive surface 451 (e.g., Figure 4B Thus, when the touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B When the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is separated, the device is used to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0172] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0173] Figure 5AAn exemplary personal electronic device 500 is shown. Device 500 includes a body 502. In some embodiments, device 500 may include a body 502 relative to devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in the foregoing. In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. As an alternative or in addition to touch screen 504, device 500 has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more strength sensors for detecting the intensity of contact (e.g., touch) applied. One or more strength sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of touch. The user interface of device 500 can respond to touch based on the intensity of the touch, which means that touches of different intensities can invoke different user interface operations on device 500.

[0174] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International patent application serial number PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO patent publication number WO / 2013 / 169849; and International patent application serial number PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO patent publication number WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0175] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0176] Figure 5B An exemplary personal electronic device 500 is depicted. In some embodiments, the device 500 may include a reference Figure 1A , Figure 1B and Figure 3 Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. I / O portion 514 may be connected to a display 504, which may have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, I / O portion 514 may be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, input mechanism 508 is optionally a button.

[0177] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as GPS sensor 532, accelerometer 534, orientation sensor 540 (e.g., compass), gyroscope 536, motion sensor 538, and / or combinations thereof, all of which are operably connected to I / O portion 514.

[0178] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the techniques described below, including process 700 ( 7A to 7C). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5B components and configurations, but may include other components or additional components in a variety of configurations.

[0179] As used herein, the term "indicative representation" refers to an optional feature of the apparatus 100, 300, and / or 500 ( Figure 1A , Figure 3 and FIG. 5A to FIG. 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.

[0180] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface that a user is interacting with. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that a touch-screen display (e.g., Figure 1A A touch-sensitive display system 112 or Figure 4AIn some implementations of the touch screen 112 in FIG. 1 , a contact detected on the touch screen acts as a “focus selector” such that when an input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted in accordance with the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of the contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0181] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0182] Attention is now turned to embodiments of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0183] FIG. 6A to FIG. 6I An exemplary user interface for managing audio output according to some embodiments is shown. The user interface in these figures is used to illustrate the process described below, including 7A to 7C process.

[0184] Fig. 6AA first electronic device 600 (e.g., device 100, 300, and / or 500), a second electronic device 602 (e.g., device 100, 300, and / or 500), and an audio device 604 (e.g., a wireless headset) are shown, where the audio device 604 communicates with the first electronic device 600 and the second electronic device 602, as represented by communication 606 and communication 608, respectively. Fig. 6A , the first electronic device 600 is in communication with the audio device 604, but the first electronic device 600 is not transmitting audio signals to the audio device 604, as shown by the dashed line representing communication 606. In contrast, the second electronic device 602 is connected to the audio device 604 and is transmitting audio signals to the audio device 604 to cause the audio device 604 to output audio, as shown by the solid line representing communication 608. FIG. 6A to FIG. 6I Audio device 604 is shown connected to (eg, actively communicating with) a corresponding electronic device via solid lines representing communication 606 and / or communication 608 and is receiving audio data from the corresponding electronic device. FIG. 6A to FIG. 6I The audio device 604 is shown communicating with the corresponding electronic device (eg, passively communicating with the corresponding electronic device) via dashed lines representing communication 606 and / or communication 608, but the audio device 604 does not receive audio data from the corresponding electronic device.

[0185] exist Fig. 6A At , the first electronic device 600 is displaying a video 610, such as a television program, on a display 612 (e.g., a television), and is outputting the audio of the video 610 at a speaker of the display 612. In some embodiments, the first electronic device 600 is in communication (e.g., wirelessly or wired) with the display 612. In some embodiments, the video 610 is played back from local media stored at the first electronic device 600. In some embodiments, the video 610 is played via a video stream (e.g., a real-time video stream) received over a network.

[0186] Fig. 6A Also shown is a remote control 614 that is configured to transmit data to the first electronic device 600 (e.g., via RF communication, via Bluetooth, via infrared) based on user input detected at the remote control 614. The remote control 614 includes a selection area 614a (which includes a touch-sensitive surface for detecting tap, press, and swipe gestures), a back key 614b, a TV button 614c, a microphone button 614d, a play / pause button, volume control buttons, and a power button.

[0187] exist Fig. 6AAt , the second electronic device 602 is displaying an audio application user interface 618 corresponding to an audio application of the second electronic device 602 on the display 616. The audio application of the second electronic device 602 is configured to cause the second electronic device 602 to output audio (e.g., audio data corresponding to one or more audio tracks). Fig. 6A At , the second electronic device 602 is connected to the audio device 604 (e.g., via a wireless communication connection, such as Bluetooth, Wi-Fi, and / or Zigbee), such that the second electronic device 602 causes the audio device 604 to output audio associated with the audio application user interface 618, as shown by the solid line representing communication 608. In some embodiments, the second electronic device 602 transmits audio data to the audio device 604 via the wireless communication connection, such that the audio device 604 outputs audio associated with the audio application user interface 618 via a speaker of the audio device 604.

[0188] exist Fig. 6A At , the audio application user interface 618 includes a playback user interface object 618a, a fast forward user interface object 618b, a fast rewind user interface object 618c, and a volume user interface object 618d. In some embodiments, when the second electronic device 602 is causing the playback of audio and in response to detecting a user input that selects the playback user interface object 618a, the second electronic device 602 stops (e.g., pauses) outputting audio and / or stops (e.g., pauses) transmitting audio data to the audio device 604. In some embodiments, when the second electronic device 602 is not currently causing the playback of audio and in response to detecting a user input that selects the playback user interface object 618a, the second electronic device 602 initiates (e.g., resumes) the output of audio and / or initiates (e.g., resumes) the transmission of audio data to the audio device 604. In some embodiments, in response to detecting a user input that selects the volume user interface object 618d, the second electronic device 602 causes the audio device 604 to adjust the volume level of the audio output via the speaker of the audio device 604.

[0189] exist Fig. 6A At , the audio device 604 communicates with the first electronic device 600 (as represented by communication 606), but the audio device 604 does not receive audio data from the first electronic device 600 and / or does not output audio associated with the first electronic device. In some embodiments, the audio device 604 is configured to communicate (e.g., passive communication) with multiple devices (e.g., the first electronic device 600 and the second electronic device 602) concurrently and / or simultaneously, but the audio device 604 is configured to establish a communication with a single device (e.g., Fig. 6AThe present invention may also be used to establish a connection (eg, active communication and / or audio data communication) with a second electronic device 602 at a location and / or receive audio data from a single device.

[0190] As explained below, reference Fig.6D and Fig. 6E In some embodiments, the first electronic device 600 and the second electronic device 602 communicate with each other so that the first electronic device 600 and the second electronic device 602 send and / or receive data indicating whether the first electronic device 600 and / or the second electronic device 602 are currently causing the playback of audio on the audio device 604 (e.g., whether the first electronic device 600 and / or the second electronic device 602 are connected to the audio device 604 and / or are in active communication with the audio device). In some embodiments, the first electronic device 600 and / or the second electronic device 602 initiates a connection with the audio device 604 based on the communication between the first electronic device 600, the second electronic device 602, and the audio device 604 (e.g., sends audio data to the audio device 604 so that the audio device 604 outputs audio associated with the first electronic device 600 and / or the second electronic device 602). In some such embodiments, when the first electronic device 600 detects that the audio device 604 is outputting audio corresponding to the audio data received from the second electronic device 602, the first electronic device 600 is not connected to the audio device 604 even if the first electronic device 600 is outputting audio corresponding to the video 610. Similarly, in some embodiments, when the second electronic device 602 detects that the audio device 604 is outputting audio corresponding to audio data received from the first electronic device 600, the second electronic device 602 is not connected to the audio device 604 even if the second electronic device 602 is outputting audio corresponding to the audio application user interface 618.

[0191] In some embodiments, the first electronic device 600 and / or the second electronic device 602 are designated (e.g., programmatically designated relative to each other) as having different priority levels for connecting to the audio device 604 and / or for otherwise transmitting audio data to the audio device 604. For example, in some embodiments, when the first electronic device 600 is connected to the audio device 604 and transmitting audio data to the audio device 604, the first electronic device 600 disconnects from the audio device 604 in response to detecting a disconnection event. In some embodiments, such as when the disconnection event includes the second electronic device 602 receiving and / or detecting a user input accepting an incoming call and / or real-time communication (e.g., a video call), the second electronic device 602 connects to the audio device 604 in response to detecting a disconnection event. In some such embodiments, the first electronic device 600 includes a lower priority designation than the second electronic device 602, such that the disconnection event causes the first electronic device 600 to disconnect from the audio device 604 and stop transmitting audio signals to the audio device 604.

[0192] In some embodiments, the first electronic device and / or the second electronic device 602 communicate with additional electronic devices that also have priority designations of different levels for connecting to the audio device 604. In some embodiments, the electronic devices that communicate with each other log in to and / or log in to the same account. In some embodiments, the electronic devices that communicate with each other perform wireless communication (e.g., direct wireless communication and / or indirect wireless communication via a server) with each other. In some embodiments, the electronic devices that the user uses most frequently (and / or is more likely to use) include priority level designations higher than the electronic devices that the user uses less frequently (and / or is less likely to use). For example, users who log in to the same account on mobile devices (e.g., smart phones), tablet devices, laptop computers, and television devices may use mobile devices more frequently than other devices, because users may carry the mobile device at home and when users leave home. In this way, the mobile device may include priority designations higher than tablet devices, laptop computers, and television devices. Similarly, users may not use television devices as frequently as mobile devices, tablet devices, and / or laptop computers, because television devices have less mobility compared to other devices. Therefore, in some embodiments, television devices include priority designations lower than mobile devices, tablet devices, and laptop computers. As explained below, in some embodiments, a device having a lower priority designation than other devices is disconnected from the audio device 604 in response to a disconnect event associated with a device having a higher priority designation. Establishing such a device hierarchy with respect to the audio device 604 enables a user of the audio device 604 to receive audio data from the electronic device with which the user is most likely to interact.

[0193] exist Figure 6B At , the second electronic device 602 has stopped displaying the audio application user interface 618 and is not actively displaying the user interface on the display 616 (e.g., the second electronic device 602 is in an idle and / or locked mode). Figure 6B At , second electronic device 602 has detected one or more user inputs, such as a user input selecting playback user interface object 618a, to stop outputting audio associated with audio application user interface 618. Therefore, second electronic device 602 remains in communication with audio device 604, but stops transmitting audio data to audio device 604, such that audio device 604 does not output audio associated with audio application user interface 618, as shown by the dashed line representing communication 608.

[0194] exist Figure 6B At , the first electronic device 600 detects that the audio device 604 is communicating with the audio device 604, as shown by the dashed line representing the communication 606. Figure 6B At this time, the first electronic device 600 does not transmit audio data to the audio device 604. The first electronic device 600 receives an indication that the audio device 604 is not receiving audio data from the second electronic device 602. In some embodiments, the first electronic device 600 receives an indication from the second electronic device 602 and / or from the audio device 604 that the audio device 604 is not outputting audio associated with the audio data received from the second electronic device 602. In some embodiments, the first electronic device 600 receives an indication from the second electronic device 602, the audio device 604, and / or another electronic device that the audio device 604 is not connected to any other electronic device different from the first electronic device 600 and / or is receiving audio data from the any other electronic device.

[0195] exist Figure 6BAt, in response to detecting the audio device 604, the first electronic device 600 determines whether a set of criteria for prompting the user to connect the first electronic device 600 to the audio device 604 is met. In some embodiments, the set of criteria includes a first criterion that is met when the first electronic device 600 receives an indication that the audio device 604 does not receive audio data from an electronic device different from the first electronic device 600. In some embodiments, the set of criteria includes a second criterion that is met when the first electronic device 600 receives an indication that the audio device 604 is powered on and / or is in an active listening position. In some embodiments, the active listening position includes the audio device 604 being positioned in the user's ear (e.g., for in-ear headphones) and / or positioned above the user's ear (e.g., for over-ear headphones). In some embodiments, the set of criteria includes a third criterion that is met when the first electronic device 600 detects that the audio device 604 is within a threshold distance of the first electronic device 600 (e.g., within 50 feet, within 30 feet, within 25 feet, and / or within 10 feet of the first electronic device 600). In some embodiments, the set of criteria includes a fourth criterion that is satisfied when the first electronic device 600 detects and / or receives an indication that the audio device 604 is associated with the first electronic device 600 (such as the audio device 604 is paired with the first electronic device 600 and / or has been previously connected to the first electronic device).

[0196] exist Figure 6B At , the first electronic device 600 determines that the set of criteria is met, and based on the determination that the set of criteria is met, the first electronic device 600 displays a selectable option 620 on the display 612. Figure 6B At 610, the selectable option 620 is a pop-up and / or push notification overlaid on the video 610. The selectable option 620 includes an indicator 620a indicating that the selectable option 620 corresponds to an audio device and / or audio device 604 (e.g., an image and / or icon representing headphones and / or audio device 604). The selectable option 620 also includes a prompt 620b, which prompts and / or instructs the user to provide a first user input via the remote control 614 to connect the first electronic device 600 to the audio device 604 and cause the first electronic device 600 to transmit audio data to the audio device 604.

[0197] exist Figure 6BAt , the selectable option 620 is displayed in the upper right portion of the display 612. In some embodiments, the selectable option 620 is displayed in the upper left portion of the display 612, the top center portion of the display 612, the lower right portion of the display 612, the lower left portion of the display 612, and / or the bottom center portion of the display 612. In some embodiments, while the selectable option 620 is displayed, the first electronic device 600 continues to cause playback of the video 610 on the display 612, so that displaying the selectable option 620 does not interrupt playback of the video 610. In some embodiments, before detecting a user input that causes the first electronic device 600 to connect to the audio device 604, the first electronic device 600 continues to cause audio output via the speakers of the display 612 while the selectable option 620 is displayed. In some such embodiments, displaying the selectable option 620 does not interrupt playback of the video 610 on the display 612 and / or playback of audio associated with the video 610 via the speakers of the display 612.

[0198] exist Figure 6B At , while the video 610 continues to play and while the selectable options 620 are displayed, the remote control 614 detects activation of the TV button 614c via the press gesture 650a and transmits an indication of the input to the first electronic device 600. While the video 610 continues to play and while the selectable options 620 are displayed, the first electronic device 600 receives an indication of the press gesture 650a corresponding to the TV button 614c from the remote control 614, and in response, the first electronic device 600 connects to the audio device 604 and initiates a process for transmitting audio data to the audio device 604, as shown. Figure 6C shown.

[0199] In some embodiments, in response to detecting the absence of user input (such as a press gesture 650a) within a predefined time period after displaying the selectable option 620 (e.g., within 45 seconds, within 30 seconds, and / or within 20 seconds of displaying the selectable option 620), the first electronic device 600 stops displaying the selectable option 620 while continuing to display and / or cause playback of the video 610.

[0200] In some embodiments, in response to detecting the audio device 604 and when the first electronic device 600 determines that the second set of criteria is met, the first electronic device 600 connects to the audio device 604 and transmits audio data to the audio device 604 without displaying the selectable option 620. In some such embodiments, the first electronic device 600 connects to the audio device 604 and transmits audio data to the audio device 604 without receiving an indication of a user input from the remote control 614. In some embodiments, the first electronic device 600 includes one or more sensors that provide data indicating the user's intention to connect the audio device 604 to the first electronic device 600. In some embodiments, the second set of criteria includes a first criterion that is met when the first electronic device 600 receives an indication from a camera sensor and / or a motion sensor that a user (e.g., a user who owns the audio device 604) is located in front of the first electronic device 600 and / or within a threshold distance of the first electronic device 600. Feedback received from the camera sensor and / or motion sensor of the first electronic device 600 confirms that the user is located in the same area and / or room as the first electronic device 600. In contrast, detecting that the audio device 604 is within a threshold distance of the first electronic device 600 indicates that the user is close to the first electronic device 600 but is not necessarily in an area where the user can view the display 612 (e.g., when the user is in an adjacent room but the user's view of the display 612 is blocked by a wall and / or another obstacle). Therefore, feedback received from the camera sensor and / or motion sensor of the first electronic device 600 indicates that the user is more likely to intend to connect the audio device 604 to the first electronic device 600, and in some embodiments, the first electronic device 600 forgoes displaying the selectable option 620 based on the feedback from the camera sensor and / or motion sensor.

[0201] In some embodiments, the second set of criteria includes a second criterion that is satisfied when the first electronic device 600 determines that the signal strength between the first electronic device 600 and the audio device 604 is equal to and / or exceeds a threshold strength indicating that the audio device 604 is within a threshold distance of the first electronic device 600. In some embodiments, the second set of criteria includes a third criterion that is satisfied when the first electronic device 600 receives an indication from the audio device 604 that the audio device is not receiving audio data from an electronic device different from the first electronic device 600. In some embodiments, the second set of criteria includes a fourth criterion that is satisfied when the first electronic device 600 detects that the audio device 604 is associated with the first electronic device 600 (such as the audio device 604 is paired with the first electronic device 600 and / or has been previously connected to the first electronic device). In some embodiments, the second set of criteria includes a fifth criterion that is satisfied when the first electronic device 600 receives from the audio device 604 an indication that the audio device 604 is powered on and / or is in an active listening position.

[0202] exist Figure 6C In response to receiving an indication of the press gesture 650a, the first electronic device 600 connects to the audio device 604 and transmits audio data corresponding to the video 610 to the audio device 604, as shown by the solid line representing communication 606. Figure 6C 604, the second electronic device 602 is in communication with the audio device 604, but the second electronic device 602 is not transmitting audio data to the audio device 604, as shown by the dashed line representing communication 608. Therefore, in response to receiving an indication of a user input corresponding to the selectable option 620, the first electronic device 600 transmits audio data to the audio device 604. Displaying the selectable option 620 prompting the user to provide the press gesture 650a enables the user of the first electronic device 600 to quickly cause the first electronic device 600 to connect to the audio device 604 without requiring the user to navigate to a settings user interface and / or otherwise interrupt the playback of the video 610.

[0203] As described above, in some embodiments, the first electronic device 600 initiates a process of disconnecting the audio device 604 in response to detecting and / or receiving an indication of a disconnection event. Fig.6D At , the first electronic device 600 is connected to the audio device 604 and is transmitting audio data corresponding to the video 610 to the audio device 604, as shown by the solid line representing the communication 606. Fig.6D At , the second electronic device 602 is in communication with the audio device 604 , but the second electronic device 602 does not transmit audio data to the audio device 604 , as shown by the dashed line representing communication 608 .

[0204] exist Fig.6DAt , the second electronic device 602 receives the incoming call and displays the incoming call user interface 622. The incoming call user interface 622 includes an accept user interface object 622a, a reject user interface object 622b, a reminder user interface object 622c, a message user interface object 622d, and a participant indicator 622e. The participant indicator 622e includes an identifier and / or name (e.g., "Jane Appleseed") of the user who initiated the call to the second electronic device 602. In some embodiments, in response to detecting a user input corresponding to the reject user interface object 622b, the second electronic device 602 rejects and / or ignores the incoming call, and optionally provides an indication that the incoming call is rejected to the first electronic device 600 (e.g., so that the first electronic device 600 remains connected to the audio device 604 and continues to provide audio data to the audio device 604). In some embodiments, in response to detecting user input corresponding to the reminder user interface object 622c, the second electronic device 602 rejects and / or ignores the incoming call and displays a reminder user interface for scheduling a reminder to call the user associated with the participant indicator 622e at a time after the current time (and optionally, provides an indication to the first electronic device 600 that the incoming call was rejected). In some embodiments, in response to detecting user input corresponding to the message user interface object 622d, the second electronic device 602 rejects and / or ignores the incoming call and displays a message user interface for sending a message to the device of the user associated with the participant indicator 622e (and optionally, provides an indication to the first electronic device 600 that the incoming call was rejected).

[0205] exist Fig.6D At , the second electronic device 602 detects user input 650b (e.g., a tap gesture) corresponding to selection of the acceptance user interface object 622a. In response to detecting the user input 650b, the second electronic device 602 accepts the incoming call, initiates a phone call between the second electronic device 602 and the device of the user associated with the participant indicator 622e, and provides an indication to the first electronic device 600 that the incoming call is accepted. In response to receiving the indication that the incoming call is accepted via the second electronic device 602, the first electronic device 600 initiates a process for disconnecting the audio device 604 and stopping transmission of audio data corresponding to the video 610 to the audio device 604. The second electronic device 602 also initiates a process for connecting to the audio device 604 such that the second electronic device 602 transmits audio data corresponding to the phone call to the audio device 604, as shown. Fig. 6E as shown.

[0206] exist Fig. 6E At , the second electronic device 602 is connected to the audio device 604 and transmits audio data corresponding to the phone call to the audio device 604, as shown by the solid line representing communication 608. Fig. 6E 606. At , the first electronic device 600 communicates with the audio device 604, but the first electronic device 600 does not transmit audio data to the audio device 604, as shown by the dotted line representing communication 606. The second electronic device 602 also displays an active call user interface 624 indicating that a phone call is ongoing between the second electronic device 602 and the device of the user associated with the participant indicator 622e. In some embodiments, in response to detecting the end call user interface object 624a of the active call user interface 624, the second electronic device 602 initiates a process for disconnecting from the audio device 604 and stopping the transmission of audio data to the audio device 604. In some such embodiments, the first electronic device 600 (e.g., from the second electronic device 602 and / or the audio device 624) receives an indication that the second electronic device 602 is disconnected from the audio device 604. In some embodiments, in response to receiving the indication, the first electronic device 600 displays a selectable option 620 and / or connects to the audio device 604 to initiate the transmission of audio data corresponding to the video 610 to the audio device 604.

[0207] Acceptance of the incoming call via the second electronic device 602 is a disconnection event that causes the first electronic device 600 to disconnect from the audio device 604 and stop transmitting audio data corresponding to the video 610 to the audio device 604. Enabling the first electronic device 600 to disconnect from the audio device 604 and enabling the second electronic device 602 to connect to the audio device 604 in response to the user input 650b allows the user to quickly and easily switch the audio output of the audio device 604 between devices. In some embodiments, acceptance of the incoming audio phone call received via the second electronic device 602 is a disconnection event for the first electronic device 600 because the first electronic device 600 includes a lower priority designation than the second electronic device 602. In some embodiments, when the first electronic device 600 includes a higher priority designation than the second electronic device 602, acceptance of the incoming audio phone call received via the second electronic device 602 is not a disconnection event for the first electronic device 600.

[0208] exist Fig.6D and Fig. 6EAt , the disconnection event that causes the first electronic device 600 to disconnect from the audio device 604 (and causes the second electronic device 602 to connect to the audio device 604) is the acceptance of an incoming audio telephone call received via the second electronic device 602. In some embodiments, the disconnection event includes receiving an incoming audio telephone call via the second electronic device 602 (e.g., regardless of whether the user accepts the incoming audio telephone call via user input 650b), accepting and / or receiving an incoming real-time communication (e.g., a video call) via the second electronic device 602, initiating an outgoing audio telephone call and / or real-time communication via the second electronic device 602, and / or playing back a media file via the second electronic device 602.

[0209] As described above, when the set of criteria is not met, the first electronic device 600 abandons displaying the selectable option 620. In some embodiments, the set of criteria includes a first criterion that is met when the first electronic device 600 detects that the audio device 604 is within a threshold distance of the first electronic device 600. Fig. 6F At , the first electronic device 600 communicates with the audio device 604, but the first electronic device 600 does not transmit audio data to the audio device 604, as shown by the dashed line representing communication 606. Fig. 6F At , the second electronic device 602 communicates with the audio device 604, but the second electronic device 602 does not transmit audio data to the audio device 604, as shown by the dashed line representing communication 608. Fig. 6F At this point, the audio device 604 is not outputting audio received from the first electronic device 600 or the second electronic device 602.

[0210] exist Fig. 6F and Figure 6GAt , arrangement 626 shows the positioning of the first electronic device 600, the second electronic device 602, and the audio device 604 relative to each other. Arrangement 626 is provided for a more thorough understanding, but is not part of a user interface displayed via the first electronic device 600 and / or the second electronic device 602. Arrangement 626 includes a distance indicator 626a indicating a distance threshold (e.g., a distance extending from the first electronic device 600) between the first electronic device 600 and the audio device 604 (and optionally, between the first electronic device 600 and the second electronic device 602). When the audio device 604 is within the distance threshold of the first electronic device 600, the first electronic device 600 detects that the first criterion in the set of criteria is met, for example, the audio device 604 is within the distance threshold of the first electronic device 600. In some embodiments, the distance indicator 626a indicates the maximum distance at which a short-range communication connection can be established between the first electronic device 600 and the audio device 604 (e.g., the maximum distance at which communication 606 can be reliably established between the first electronic device 600 and the audio device 604). The distance threshold may be non-uniform, may be affected by a variety of variables (e.g., wireless interference, air humidity, etc.), and may include points in three-dimensional space, all of which are intended to be within the scope of the present disclosure. Therefore, the graphical representation of distance indicator 626a is merely exemplary. In addition, the drawings are not necessarily drawn to scale and are included herein only as a visual aid. Therefore, the sizes and proportions of the features depicted in the drawings are not intended to be limitations on the distance required to establish a connection.

[0211] exist Fig. 6F At , the arrangement 626 indicates that the audio device 604 is located outside (eg, at a distance greater than) a distance threshold from the first electronic device 600 represented by the distance indicator 626a. Fig. 6F At , the first electronic device 600 detects the audio device 604 and determines that the audio device 604 is located farther than a threshold distance from the first electronic device 600. In some embodiments, the first electronic device 600 determines that the audio device 604 is located farther than a threshold distance from the first electronic device 600 based on the strength of the signal between the first electronic device 604 and the audio device 600 (for example, the strength of the signal between the first electronic device 600 and the audio device 604 is lower than a threshold signal strength indicating a threshold distance). Fig. 6FIn response to determining that the audio device 604 is located outside the threshold distance from the first electronic device 600 (e.g., farther than the threshold distance), the first electronic device 600 abandons displaying the selectable option 620 and continues to display and / or play the video 610. When the audio device 600 is located outside the threshold distance from the first electronic device 600, the first electronic device 604 also abandons displaying the selectable option 620 even if the second electronic device 602 is not connected to the audio device 604 and / or transmitting audio data to the audio device 604.

[0212] In some embodiments, the first electronic device 600 forgoes displaying the selectable option 620 based on determining that the audio device 604 is outside a threshold distance of the first electronic device 600 because it is unlikely that the user of the audio device 604 intended to connect the audio device 604 to the first electronic device 600 (e.g., the user is not within an appropriate distance for viewing the display 612).

[0213] In some embodiments, when the first electronic device 600 is displaying a particular type of content, the first electronic device 600 abandons displaying the selectable option 620. In some embodiments, the set of standards includes a second standard that is satisfied when the first electronic device 600 is displaying a first type of content. In some embodiments, when the first electronic device 600 is not displaying the first type of content, the first electronic device 600 abandons displaying the selectable option 620 regardless of whether the set of standards is satisfied. In some such embodiments, the first electronic device 600 displaying the first type of content is not a standard in the set of standards, so that when the first electronic device 600 displays a second type of content that is different from the first type of content, the first electronic device 600 abandons making a determination as to whether the set of standards is satisfied.

[0214] exist Figure 6G At , the first electronic device 600 communicates with the audio device 604, but the first electronic device 600 does not transmit audio data to the audio device 604, as shown by the dashed line representing communication 606. Figure 6G At , the second electronic device 602 communicates with the audio device 604, but the second electronic device 602 does not transmit audio data to the audio device 604, as shown by the dashed line representing communication 608. Figure 6G At this point, the audio device 604 is not outputting audio received from the first electronic device 600 or the second electronic device 602.

[0215] exist Figure 6GAt, the first electronic device 600 is displaying a screen saver 628. In some embodiments, the first electronic device 600 determines whether the first type of content is being displayed before determining whether a set of criteria for displaying selectable options 620 is met. In some embodiments, the first type of content includes playback of media files and / or a content library for selecting media files. In some embodiments, media files include video 610 and / or another video, video game, music track and / or other multimedia currently being played, streamed, paused and / or output via the first electronic device 600. In some embodiments, the content library for selecting media files includes a user interface having one or more user interface objects corresponding to applications and / or media files, which cause the first electronic device 600 to playback and / or otherwise display media files when selected via user input. In some embodiments, the first type of content does not include a screen saver and / or user interface associated with the inactivity of the first electronic device 600.

[0216] exist Figure 6G At this point, the first electronic device 600 determines that the screen saver 628 is not the first type of content. In some embodiments, based on determining that the screen saver 628 is not the first type of content, the first electronic device 600 abandons determining whether a set of criteria for displaying the selectable options 620 is met. In some embodiments, the first electronic device 600 determines that a set of criteria for displaying the selectable options 620 is not met based on determining that the screen saver 628 is not the first type of content. Figure 6G At, based on determining that the screen saver 628 is not the first type of content, the first electronic device 600 forgoes displaying the selectable option 620 even if the audio device 604 is within a threshold distance of the first electronic device 600 (e.g., as indicated by the arrangement 626 and the distance indicator 626a) and even if the audio device 604 is not outputting audio associated with audio data received from the second electronic device 602.

[0217] In some embodiments, upon determining that the audio routing setting of the first electronic device 600 is disabled, the first electronic device 600 abandons displaying the selectable option 620. Figure 6H At , the first electronic device 600 displays a setting user interface 630. The setting user interface 630 is associated with audio settings of content output by the first electronic device 600. Figure 6HAt, the setting user interface 630 includes an audio routing user interface object 630a that is currently in an enabled and / or open position. When the audio routing user interface object 630a is in the enabled and / or open position, the first electronic device 600 displays the selectable option 620 when displaying the first type of content and based on determining that the set of standards are met. When the audio routing user interface object 630a is in the disabled and / or closed position, the first electronic device 600 abandons displaying the selectable option 620 when displaying the first type of content regardless of whether the set of standards are met. In some embodiments, when the audio routing user interface object 630a is in the disabled and / or closed position, the first electronic device 600 abandons determining whether the set of standards are met. In some embodiments, when the audio routing user interface object 630a is in the disabled and / or closed position, the first electronic device 600 abandons detecting the audio device 604 and / or communicating with the audio device 604 via communication 606.

[0218] exist Figure 6H At , when the first electronic device 600 displays the settings user interface 630 and focuses on the audio routing user interface object 630a, the remote control 614 detects activation of the selection area 614a via the press gesture 650c and transmits an indication of the input to the first electronic device 600. When the first electronic device 600 displays the settings user interface 630 and focuses on the audio routing user interface object 630a, the first electronic device 600 receives an indication of the press gesture 650c corresponding to the activation area 614a from the remote control 614, and in response, the first electronic device 600 disables the audio routing settings and displays the audio routing user interface object 630a in a disabled and / or closed position.

[0219] In some embodiments, the first electronic device 600 is configured to transmit audio data to multiple audio devices concurrently and / or simultaneously (e.g., when the set of criteria is met for each audio device). Fig.6I At , the first electronic device 600 is connected to the audio device 604 and is transmitting audio data corresponding to the video 610 to the audio device 604, as shown by the solid line representing the communication 606. Fig.6I At , the first electronic device 600 detects the second audio device 632 via communication 634 with the second audio device 632. However, the first electronic device is not connected to the second audio device 632, and does not transmit audio data corresponding to the video 610 to the second audio device 632.

[0220] exist Fig.6IAt, the first electronic device 600 determines that a third set of criteria associated with the second audio device 632 is satisfied. In some embodiments, the third set of criteria includes a first criterion that is satisfied when the first electronic device 600 receives an indication that the second audio device 632 is not connected to an electronic device different from the first electronic device 600. In some embodiments, the third set of criteria includes a second criterion that is satisfied when the first electronic device 600 receives an indication that the second audio device 632 is powered on and / or is in an active listening position. In some embodiments, the third set of criteria includes a third criterion that is satisfied when the first electronic device 600 detects that the second audio device 632 is within a threshold distance of the first electronic device 600. In some embodiments, the third set of criteria includes a fourth criterion that is satisfied when the first electronic device 600 detects and / or receives an indication that the second audio device 632 is associated with the first electronic device 600 (such as the second audio device 632 is paired with the first electronic device 600 and / or has been previously connected to the first electronic device).

[0221] exist Fig.6I At , in response to detecting the second audio device 632 and based on determining that the third set of criteria are met, the first electronic device 600 displays a selectable option 636 corresponding to the second audio device 632 on the display 612. Fig.6I In response to receiving an indication from the remote control 614 corresponding to a press gesture 650d on the TV button 614c, the first electronic device 600 connects to the second audio device 632 and transmits audio data to the second audio device 632 and the audio device 604 simultaneously.

[0222] 7A to 7C 700 is a flow chart illustrating a method for managing audio output using a computer system according to some embodiments. Method 700 is performed at a computer system (e.g., 100, 300, 500, or 600) (e.g., an electronic device; a set-top device, a smart audio speaker, and / or a digital media player) that communicates (e.g., wired communication and / or wireless communication) with a display generation component (e.g., 612). Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0223] As described below, method 700 provides an intuitive way to manage audio output. The method reduces the cognitive burden on the user when switching audio output between devices, thereby creating a more efficient human-computer interface. For battery-powered computing devices, enabling users to switch audio output between devices more quickly and more efficiently saves power and increases the time between battery charges.

[0224] While displaying (e.g., causing the display generation component to display) a first type of content (e.g., 610) via a display generation component (e.g., 612) (e.g., actively displaying the content, such as playing, streaming, and / or otherwise displaying a multimedia file and / or causing playback of audio, displaying a content library for selecting content to actively display, and / or not displaying a second type of content, such as a screen saver and / or an inactive display screen), a computer system (e.g., 600) detects (702) an audio device (e.g., 604) (e.g., an external audio device configured to output audio generated and / or received from the computer system, such as a wireless headset).

[0225] In response to detecting the audio device (704) and in accordance with determining that a set of criteria is satisfied (e.g., the set of criteria is satisfied when the audio device is not currently outputting audio received from a device other than the computer system, when the audio device is powered on and / or in a listening position (e.g., within and / or over a user's ear), when the audio device is associated with the computer system (e.g., the audio device is paired with and / or otherwise in communication with and / or has been in communication with the computer system), and / or when the audio device is within a threshold distance of the computer system (e.g., within 50 feet, within 25 feet, and / or within 10 feet of the computer system), the computer system (e.g., 600) displays (706) (e.g., by displaying to a display device the audio device). In some embodiments, the computer system transmits audio data corresponding to the content audio to a speaker of the computer system before detecting the user input selecting the selectable option, and the computer system subsequently abandons transmitting the audio data corresponding to the content audio to the speaker, so that the speaker does not output the content audio in response to detecting the user input selecting the selectable option. In some embodiments, the computer system transmits audio data to both the speaker and the audio device of the computer system in response to detecting the user input selecting the selectable option.

[0226] In response to detecting an audio device (704) and based on determining that the set of criteria is not met (e.g., the set of criteria is not met when the audio device is currently outputting audio received from a device different from the computer system (e.g., an electronic device such as a smart phone, tablet computer and / or laptop computer and / or desktop computer), when the audio device is not powered on and / or in a listening position, when the audio device is not associated with the computer system and / or when the audio device is outside a threshold distance of the computer system), the computer system (e.g., 600) abandons (708) displaying selectable options (e.g., 620) (e.g., the computer system abandons transmitting audio data corresponding to the content audio to the audio device and optionally continues transmitting audio data corresponding to the content audio to the speaker so that the speaker continues to output the content audio).

[0227] Displaying selectable options based on determining that the set of criteria is met enables the user to connect the audio device to the computer system without navigating to a setup user interface and / or another user interface that would interrupt the first type of content, which reduces the number of inputs required to perform the operation and / or provides the user with an improved experience in viewing and / or listening to content.

[0228] In some embodiments, while displaying (e.g., by transmitting data (e.g., image data and / or video data) to the display generating component to cause the display generating component to display) a selectable option (e.g., 620) via a display generating component (e.g., 612), the computer system (e.g., 600) detects (710) a first user input (e.g., 650a) corresponding to the selectable option (e.g., 620) (e.g., corresponding to a first user input of a remote control device in communication with the computer system) (e.g., the first user input is a user input prompted by the selectable option and / or a user input selecting the selectable option (e.g., when the focus is on the selectable option)). In response to detecting the first user input (e.g., 650a) (e.g., receiving an indication of the first user input via a remote control device in communication with the computer system), the computer system (e.g., 600) provides (712) output content audio to an audio device (e.g., 604) (e.g., causing the audio device to output the output content audio via a speaker of the audio device).

[0229] Providing output content audio to an audio device in response to detecting a first user input enables a user to listen to the output content audio corresponding to a first type of content without having to navigate to another user interface and / or interrupt the first type of content, which reduces the number of inputs required to perform an operation and / or provides the user with an improved experience of viewing and / or listening to content.

[0230] In some embodiments, while providing output content audio to an audio device (e.g., 604) (e.g., causing the audio device to output the output content audio via a speaker of the audio device), the computer system (e.g., 600) detects (714) a disconnection condition (e.g., an incoming call received via the second electronic device 602) (e.g., receiving an indication of a disconnection event from an external device that communicates with the audio device (e.g., the external device is paired with the audio device and / or the external device is within a predefined distance of the audio device), wherein the disconnection event includes an incoming call received via the external device, an incoming real-time communication received via the external device, initiating playback of content via the external device, initiating a connection between the audio device and the external device, and / or initiating disconnection of the audio device from the computer system). In response to detecting the disconnection condition, the computer system (e.g., 600) initiates (716) a process of stopping providing output content audio to the audio device (e.g., 604) (e.g., stopping causing the audio device to output the output content audio via the speaker of the audio device, stopping transmitting audio data associated with the output content audio to the audio device, and / or stopping the connection between the computer system and the audio device).

[0231] Initiating a process of ceasing to provide output content audio to an audio device in response to detecting a disconnect condition enables a user to quickly switch a connection between the audio device and a computer system to a connection between the audio device and an external device without having to provide additional user input, which reduces the number of inputs required to perform an operation and / or performs an operation without further user input when a set of conditions have been met.

[0232] In some embodiments, while displaying (e.g., by transmitting data (e.g., image data and / or video data) to the display generation component to cause the display generation component to display) a selectable option (e.g., 620) via a display generation component (e.g., 612), the computer system (e.g., 600) detects (718) that a predefined amount of time (e.g., five seconds, ten seconds, twenty seconds, and / or thirty seconds) has passed since the selectable option (e.g., 620) was first displayed (e.g., a predefined amount of time starting from when the computer system caused the selectable option to be displayed via the display generation component and no user input corresponding to the selectable option (e.g., user input selecting the selectable option (e.g., when focus is on the selectable option) and / or user input prompted by the selectable option) has been detected). In response to detecting that the predefined amount of time has passed since the selectable option (e.g., 620) was first displayed, the computer system (e.g., 600) stops (720) displaying the selectable option (e.g., 620) (e.g., by transmitting data to the display generation component to cause the display generation component to stop displaying).

[0233] Ceasing to display the selectable option in response to detecting that a predefined amount of time has passed since the selectable option was first displayed reduces the amount of interruption of the first type of content caused by the selectable option, performing the operation when a set of conditions are met without further user input and / or providing the user with an improved experience of viewing and / or listening to the content.

[0234] In some embodiments, a computer system (e.g., 600) displays a selectable option (e.g., 620) simultaneously with a first type of content (e.g., 610) (e.g., the selectable option is overlaid on a portion of the first type of content, but playback of the first type of content is not otherwise interrupted). In some embodiments, the first type of content is a video currently playing on a display generation component, and when the selectable option is displayed, the video does not pause and / or otherwise stop playing and / or displaying (e.g., except for the portion covered and / or blocked by the selectable option). Displaying the selectable option simultaneously with the first type of content enables a user to connect an audio device to the computer system without having to navigate to another user interface and interrupt the first type of content, which reduces the number of inputs required to perform operations and / or provides the user with an improved experience of watching and / or listening to content.

[0235] In some embodiments, the set of criteria includes a first criterion that is satisfied when the computer system (e.g., 600) detects that the audio device (e.g., 604) is not outputting audio associated with an external device (e.g., 602) (e.g., the audio device is not caused to output audio via a speaker of the audio device via data transmission from the external device) (e.g., based on determining that the computer system detects that the audio device is not outputting audio associated with the external device). In some embodiments, the first criterion is a sufficient condition for satisfying the set of criteria. In some embodiments, the first criterion is a necessary but not sufficient condition for satisfying the set of criteria. Displaying a selectable option when the audio device is not outputting audio associated with an external device reduces the likelihood of inadvertently switching the audio output of the audio device to the content output audio, which provides the user with an improved experience of viewing and / or listening to content.

[0236] In some embodiments, the set of criteria includes a second criterion that is satisfied when the audio device (e.g., 604) is in an active listening position (e.g., the audio device includes an indication of whether the audio device is positioned within an ear of a user (e.g., a person) (e.g., when the audio device includes in-ear headphones) and / or positioned above an ear of a user (e.g., a person) (e.g., when the audio device includes over-ear headphones) and the computer system receives an indication from the audio device that the audio device is positioned within the user's ear and / or above the user's ear) (e.g., based on determining that the audio device is in an active listening position or the computer system detects and / or receives an indication that the device is positioned within the user's ear and / or above the user's ear). In some embodiments, the second criterion is a sufficient condition for satisfying the set of criteria. In some embodiments, the second criterion is a necessary but not sufficient condition for satisfying the set of criteria.

[0237] Displaying selectable options when the audio device is in an active listening position increases the likelihood that a user associated with the audio device intends to listen to output content audio corresponding to the first type of content and reduces the amount of interruption of the first type of content, which reduces the number of inputs required to perform operations and / or provides the user with an improved experience in viewing and / or listening to content.

[0238] In some embodiments, the set of criteria includes a third criterion that is satisfied when the computer system (e.g., 600) detects that the audio device (e.g., 604) has been previously associated with the computer system (e.g., 600) (e.g., the computer system is paired with the audio device and / or has been otherwise connected to the audio device and caused the audio device to output audio associated with audio data of the computer system) (e.g., based on determining that the computer system has detected that the audio device has been previously associated with the computer system). In some embodiments, the third criterion is a sufficient condition for satisfying the set of criteria. In some embodiments, the third criterion is a necessary but not sufficient condition for satisfying the set of criteria. Displaying selectable options when the audio device has been previously associated with the computer system reduces interruption of the first type of content to audio devices that are unlikely to be connected to the computer system, which reduces the processing power of the computer system.

[0239] In some embodiments, the set of criteria includes a fourth criterion that is satisfied when the computer system (e.g., 600) detects that the audio device (e.g., 604) is within a threshold distance (e.g., the distance indicated by distance indicator 626a) of the computer system (e.g., 600) (e.g., within 50 feet, within 30 feet, within 20 feet, and / or within 10 feet) (e.g., the computer system detects that the strength of the connection signal between the computer system and the audio device exceeds a threshold amount indicating that the audio device is within the threshold distance of the computer system) (e.g., based on determining that the computer system detects that the audio device is within the threshold distance of the computer system). In some embodiments, the fourth criterion is a sufficient condition for satisfying the set of criteria. In some embodiments, the fourth criterion is a necessary but not sufficient condition for satisfying the set of criteria. Displaying selectable options when the audio device is within the threshold distance of the computer system reduces interruption of the first type of content for audio devices that are unlikely to be connected to the computer system, which reduces the processing power of the computer system.

[0240] In some embodiments, the first type of content (e.g., 610) includes playback of media files (e.g., videos, video games, music tracks, and / or other multimedia that is currently playing, streaming, paused, and / or outputted) and / or a content library for selecting media files (e.g., a content library including one or more applications and / or one or more media files that the computer system can cause to be played back and displayed on the display generating component). Displaying selectable options when the first type of content includes media files and / or a content library for selecting media files increases the likelihood that a user associated with the audio device intends to connect the audio device to the computer system, which reduces the processing power of the computer system.

[0241] In some embodiments, the first type of content (e.g., 610) does not include a screen saver (e.g., 628) (e.g., an image, a set of images, and / or a video displayed via a display generation component in response to the computer system detecting an absence of user input and / or an absence of active playback of media). In some embodiments, the set of criteria is not met based on determining that the content is a screen saver. Abandoning detection of an audio device when the computer system displays a screen saver via the display generation component reduces power consumption and / or processing power of the computer system when an audio device is unlikely to be connected to the computer system, which improves battery life of the computer system and / or reduces the amount of power consumed by the computer system.

[0242] In some embodiments, based on determining that an audio output setting (e.g., an audio output setting that, when enabled, causes the computer system to detect an audio device and / or other audio devices) is disabled (e.g., via audio routing user interface object 630a), the computer system (e.g., 600) abandons (722) displaying a selectable option (e.g., 620) (e.g., the computer system abandons transmitting data to a display generation component that would cause the display generation component to display the selectable option regardless of whether the set of criteria is met). Abandoning displaying the selectable option when the audio output setting is disabled enables the computer system to save power when the user does not intend to connect an audio device to the computer system, which improves the battery life of the computer system and / or reduces the amount of power consumed by the computer system.

[0243] In some embodiments, in response to detecting the audio device (724) (e.g., 604) and in accordance with determining that a second set of criteria (726) is satisfied (e.g., the second set of criteria is satisfied when the computer system includes one or more sensors that provide an indication of a user's intent to connect the audio device to the computer system, wherein the indication includes a camera sensor and / or a motion sensor detecting that the user is in front of the computer system and / or within a threshold distance of the computer system, the computer system detecting that a strength of a connection signal between the computer system and the audio device is equal to and / or exceeds a threshold indicating that the audio device is within the threshold distance of the computer system, and the computer system receiving from the audio device that the audio device is not outputting audio that is connected to the computer system. The computer system (e.g., 600) forgoes (728) displaying the selectable option (e.g., 620) (e.g., the computer system forgoes transmitting data to the display generation component that would cause the display generation component to display the selectable option) and provides (730) output content audio to the audio device (e.g., 604) (e.g., causing the audio device to output the content audio via a speaker output of the audio device without detecting user input).

[0244] Abandoning displaying the selectable options and providing output content to the audio device based on determining that the second set of criteria is met enables the computer system to connect to the audio device without user input, which performs operations without further user input when a set of conditions have been met.

[0245] In some embodiments, when a computer system (e.g., 600) displays (e.g., by transmitting data (e.g., image data and / or video data) to a display generating component to cause the display generating component to display) a first type of content (e.g., 610) via a display generating component (e.g., 612) (e.g., actively displaying the content, such as playing, streaming, and / or otherwise displaying a multimedia file and / or causing playback of audio, displaying a content library for selecting content to be actively displayed, and / or not displaying a second type of content, such as a screen saver and / or an inactive display screen), the computer system (e.g., 600) detects (732) a second audio device (632) that is different from the audio device (604) (e.g., an external audio device, such as a wireless headset, configured to output audio generated and / or received from the computer system). In response to detecting a second audio device (734) (e.g., 632) that is different from the audio device (e.g., 604) and based on determining that a third set of criteria is satisfied (e.g., the third set of criteria is satisfied when the second audio device is not currently outputting audio received from a device different from the computer system, when the second audio device is powered on and / or in a listening position (e.g., within and / or over the user's ear), when the second audio device is associated with the computer system (e.g., the second audio device is paired with and / or otherwise in communication with and / or has been in communication with the computer system), and / or when the second audio device is within a threshold distance of the computer system (e.g., within 50 feet, within 25 feet, and / or within 10 feet of the computer system), the computer system (e.g., 600) displays (736) (e.g., by Data (e.g., image data and / or video data) is transmitted to the display generation component to cause the display generation component to display) a second selectable option (e.g., 636) (e.g., a pop-up and / or push notification prompting a user to provide user input), which second selectable option, when selected via user input (e.g., 650d) (e.g., selected via user input from a remote control when focus is displayed on the second selectable option, and / or selected via user input prompted by the user providing the second selectable option) causes the computer system (e.g., 600) to provide output content audio to a second audio device (e.g., 632) (e.g., the computer system transmits audio data to the second audio device so that the second audio device outputs content audio corresponding to the first type of content displayed via the display generation component of the computer system (e.g., the same content audio as the audio device)).In response to detecting a second audio device (734) (e.g., 632) that is different from the audio device (e.g., 604) and based on determining that a third set of criteria is not met (e.g., the third set of criteria is not met when the second audio device is currently outputting audio received from a device that is different from the computer system (e.g., an electronic device such as a smart phone, a tablet computer and / or a laptop computer and / or a desktop computer), when the second audio device is not powered on and / or in a listening position, when the second audio device is not associated with the computer system, and / or when the second audio device is outside a threshold distance of the computer system), the computer system (e.g., 600) abandons (738) displaying a second selectable option (e.g., 636) (e.g., the computer system abandons transmitting audio data corresponding to the content audio to the second audio device, and optionally continues transmitting audio data corresponding to the content audio to the speaker so that the speaker continues to output the content audio).

[0246] Displaying a second selectable option based on determining that the third set of criteria is met enables the user to connect a second audio device to the computer system without navigating to a settings user interface and / or another user interface that would interrupt the first type of content, which reduces the number of inputs required to perform the operation and / or provides the user with an improved experience in viewing and / or listening to content.

[0247] For the purpose of explanation, the preceding description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to best explain the principles of these technologies and their practical applications. Others skilled in the art can thus best utilize these technologies and various embodiments with various modifications suitable for the intended specific use.

[0248] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the disclosure and examples defined by the claims.

[0249] As described above, one aspect of the present technology is to collect and use data from various sources to improve the audio delivered to the user. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, social network IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0250] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to cause an audio device to output audio from a device that the user may interact with. In addition, the present disclosure also anticipates other uses of the personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health or can be used as positive feedback to individuals using technology to pursue health goals.

[0251] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and protect access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0252] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to routing audio output, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0253] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0254] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, a determination as to whether to switch audio output between devices may be based on non-personal information data or a minimal amount of personal information, such as content requested by a device associated with a user or publicly available information.

Claims

1. A method comprising At a computer system in communication with a display generating component and one or more input devices: detecting an audio device while displaying content of a first type via the display generating component; and In response to detecting the audio device: providing output content audio to the audio device based on determining that a set of criteria is satisfied, wherein the set of criteria includes a first criterion that is satisfied when the computer system receives, via the one or more input devices, information indicating that a user is located at a predefined location relative to the computer system; and Based on determining that the set of criteria is not met, output content audio is provided to the audio device.

2. The method of claim 1, wherein the set of criteria includes a second criterion that is satisfied when the computer system detects that a signal strength between the computer system and the audio device exceeds a threshold signal strength.

3. The method of any one of claims 1 to 2, wherein the set of criteria includes a third criterion that is met when the computer system detects that the audio device is not outputting audio associated with a second external device.

4. The method of any one of claims 1 to 2, wherein the set of criteria includes a fourth criterion that is satisfied when the computer system detects that the audio device is associated with the computer system.

5. The method of any one of claims 1 to 2, wherein the set of criteria includes a fifth criterion that is satisfied when the computer system receives an indication that the audio device is in a first state. The method of claim 5 , wherein the first state comprises a power-on state. The method of claim 5 , wherein the first state comprises the audio device being in an active listening position.

8. The method of any one of claims 1-2, wherein the one or more input devices include a motion sensor and / or a camera sensor.

9. The method of any one of claims 1-2, wherein the information received via the one or more input devices indicating that the user is located at the predefined position relative to the computer system includes information indicating that the user is within a threshold distance of the computer system.

10. The method of any one of claims 1-2, wherein the information received via the one or more input devices indicating that the user is located at the predefined position relative to the computer system comprises information indicating that the user is located in front of the computer system.

11. The method of any of claims 1-2, wherein providing the output content audio to the audio device comprises providing the output content audio to the audio device without receiving user input.

12. The method according to any one of claims 1-2, further comprising: detecting a disconnect condition while providing output content audio to the audio device; as well as In response to detecting the disconnect condition, a process is initiated to stop providing output content audio to the audio device.

13. The method according to any one of claims 1-2, wherein the first type of content comprises playback of media files and / or a content library for selecting media files.

14. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 13.

15. A computer system configured to communicate with a display generation component and one or more input devices, the computer system comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 13.

16. A computer system configured to communicate with a display generation component and one or more input devices, comprising: Device for carrying out the method according to any one of claims 1 to 13.

17. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and one or more input devices, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 13.

Images