User interface for managing audio of media items

Abstract

The present disclosure relates generally to techniques and user interfaces for transmitting audio using a computer system, according to some embodiments.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 075,130, filed September 5, 2020, entitled “USER INTERFACES FOR MANAGING AUDIOFOR MEDIA ITEMS”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to computer user interfaces, and more specifically, to techniques for managing audio associated with media projects. Background Technology

[0004] Users are increasingly watching and listening to media. Some users enjoy watching and listening to media with their friends. Some existing technologies allow users to watch media on a display system while simultaneously listening to the audio associated with the media via audio output devices such as headphones and / or one or more speakers. Summary of the Invention

[0005] However, some technologies used to manage audio tracks in media projects using electronic devices are often cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may involve multiple keystrokes or button presses. These technologies require more time than necessary, resulting in wasted user time and device power. This latter consideration is particularly important in battery-powered devices.

[0006] Therefore, the present invention provides electronic devices with a faster and more efficient method and interface for managing audio tracks in media projects. Such methods and interfaces optionally complement or replace other methods for managing audio tracks in media projects. These methods and interfaces reduce the cognitive burden on users and result in a more effective human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charging cycles.

[0007] In some examples, the present invention enables users to transmit different audio files corresponding to a media project to different devices in a time-efficient and input-efficient manner, thereby reducing the amount of processing required by the devices and / or reducing the number of playback instances of the media project.

[0008] According to some embodiments, a method is described. The method is executed at a computer system communicating with one or more input devices. The method includes: transmitting, when causing playback of a media item corresponding to a plurality of audio tracks, a first audio track corresponding to a first audio track among the plurality of audio tracks to a first device and a second device; while transmitting the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device and the second device, receiving, via the one or more input devices, a request to transmit, to the second device, a second audio track corresponding to a second audio track among the plurality of audio tracks that is different from the first audio track; and in response to receiving the request to transmit the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device: transmitting the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device; and continuing to transmit the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system, wherein the computer system communicates with one or more input devices, the one or more programs including instructions for: transmitting, when causing playback of a media item corresponding to a plurality of audio tracks, a first audio track corresponding to a first audio track among the plurality of audio tracks to a first device and a second device; while transmitting the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device and the second device, receiving via the one or more input devices a request to transmit a second audio track corresponding to a second audio track among the plurality of audio tracks that is different from the first audio track to the second device; and in response to receiving the request to transmit the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device: transmitting the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device; and continuing to transmit the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device.

[0010] According to some embodiments, a transient computer-readable storage medium is described. This transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system, wherein the computer system communicates with one or more input devices, the one or more programs including instructions for: transmitting, when causing playback of a media item corresponding to a plurality of audio tracks, a first audio track corresponding to a first audio track among the plurality of audio tracks to a first device and a second device; while transmitting the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device and the second device, receiving via the one or more input devices a request to transmit a second audio track corresponding to a second audio track among the plurality of audio tracks that is different from the first audio track to the second device; and in response to receiving the request to transmit the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device: transmitting the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device; and continuing to transmit the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device.

[0011] According to some embodiments, a computer system is described. The computer system includes: one or more processors, wherein the computer system communicates with one or more input devices; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: transmitting, when causing playback of a media item corresponding to a plurality of audio tracks, a first audio track corresponding to a first audio track among the plurality of audio tracks to a first device and a second device; while transmitting the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device and the second device, receiving via the one or more input devices a request to transmit a second audio track corresponding to a second audio track among the plurality of audio tracks that is different from the first audio track to the second device; and in response to receiving the request to transmit the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device: transmitting the second audio track corresponding to the second audio track among the plurality of audio tracks to the second device; and continuing to transmit the first audio track corresponding to the first audio track among the plurality of audio tracks to the first device.

[0012] According to some embodiments, a computer system is described. The computer system communicates with one or more input devices. The computer system includes: means for transmitting first audio corresponding to a first audio track among the plurality of audio tracks to a first device and a second device when causing playback of a media item corresponding to a plurality of audio tracks; means for receiving, via the one or more input devices, a request to transmit second audio corresponding to a second audio track among the plurality of audio tracks that is different from the first audio track to the second device while transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the first device and the second device; and means for performing the following operations in response to receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the second device: transmitting the second audio corresponding to the second audio track among the plurality of audio tracks to the second device; and continuing to transmit the first audio corresponding to the first audio track among the plurality of audio tracks to the first device.

[0013] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0014] Therefore, devices are provided with faster and more efficient methods and interfaces for managing audio associated with media projects, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. These methods and interfaces can complement or replace other methods used for managing audio associated with media projects. Attached Figure Description

[0015] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0016] Figure 1A This is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some embodiments.

[0017] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some implementation schemes.

[0018] Figure 2 A portable multi-functional device with a touchscreen is shown according to some embodiments.

[0019] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some implementation schemes.

[0020] Figure 4A An exemplary user interface for a menu on an application on a portable multifunction device, according to some implementation schemes, is shown.

[0021] Figure 4B An exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0022] Figure 5A A personal electronic device according to some implementation schemes is shown.

[0023] Figure 5B This is a block diagram illustrating a personal electronic device according to some implementation schemes.

[0024] Figure 5C An electronic device according to some implementation schemes is shown.

[0025] Figure 5D This is a block diagram illustrating an electronic device according to some implementation schemes.

[0026] Figures 6A to 6K Exemplary techniques and user interfaces for transmitting audio using a computer system, according to some implementation schemes, are shown.

[0027] Figures 7A to 7B This is a flowchart illustrating a method for transmitting audio using a computer system according to some implementation schemes. Detailed Implementation

[0028] The following description illustrates exemplary methods, parameters, etc. However, it should be understood that such description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.

[0029] Electronic devices need to provide efficient methods and interfaces for managing audio associated with media items. For example, new, efficient methods and interfaces are needed to transmit different audio to different devices while a user is watching and listening to a media item. Such technologies can reduce the cognitive load on users watching and listening to media items, thereby increasing productivity. Furthermore, such technologies can reduce processor and battery power wasted otherwise on redundant user input.

[0030] In the following text, Figures 1A to 1B , Figure 2 , Figure 3 , Figures 4A to 4B and Figures 5A to 5D A description of an exemplary device for performing techniques for transmitting audio is provided. Figures 6A to 6K Exemplary techniques and user interfaces for transmitting audio using a computer system, according to some implementation schemes, are shown. Figures 7A to 7BThis is a flowchart illustrating a method for transmitting audio using a computer system according to some implementation schemes. Figures 6A to 6K The user interface in the document is used to illustrate the process described below, including Figures 7A to 7B The process in.

[0031] The processes described below enhance device operability and make the user-device interface more efficient through various technologies (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device), including providing users with improved visual feedback, reducing the amount of input required to perform operations, providing additional control options without cluttering the user interface with additional display controls, and performing operations without further user input and / or additional techniques when a set of conditions have been met. These technologies also reduce power consumption and extend device battery life by enabling users to use the device faster and more efficiently.

[0032] Furthermore, in methods described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the method may be repeated in multiple repetitions such that, during the repetitions, all conditions determining the steps in the method are satisfied in different repetitions of the method. For example, if a method requires performing a first step (if a condition is satisfied) and a second step (if a condition is not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions determining the steps in the method are satisfied. Those skilled in the art will also understand that, similar to methods having discretionary steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.

[0033] Although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch may be named a second touch and similarly, a second touch may be named a first touch, without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0034] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0035] Depending on the context, the term "if" may optionally be interpreted as meaning "when," "at," or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" may optionally be interpreted as meaning "in response to determination..." or "in response to detection [the stated condition or event]."

[0036] This document describes implementations of electronic devices, user interfaces for such devices, and related processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). In some embodiments, the electronic device is a computer system that communicates with a display generating component (e.g., via wireless or wired communication). The display generating component is configured to provide visual output, such as a display via a CRT monitor, a display via an LED monitor, or a display via image projection. In some embodiments, the display generating component is integrated with the computer system. In some embodiments, the display generating component is separate from the computer system. As used herein, “display” content includes displaying content (e.g., video data rendered or decoded by display controller 156) by transmitting data (e.g., image data or video data) to an integrated or external display generating component via a wired or wireless connection to visually generate content.

[0037] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0038] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0039] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0040] Now let’s turn our attention to implementation schemes for portable devices with touch-sensitive displays. Figure 1AThis is a block diagram illustrating a portable multi-functional device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and is sometimes referred to as or called a “touch-sensitive display system.” Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (e.g., generating haptic output on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0041] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute (alternative) for the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface near the contact, and / or the resistance and / or variation of the touch-sensitive surface near the contact may optionally be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement) has been exceeded. In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure) has been exceeded. Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited physical space, which is used (e.g., on a touch-sensitive display) to display an indication and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0042] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. For example, even when the smoothness of the tactile surface remains unchanged, the movement of the tactile surface can optionally be interpreted or sensed by the user as the "roughness" of the tactile surface. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user.

[0043] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits.

[0044] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0045] Peripheral interface 118 can be used to couple the device's input and output peripherals to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs (such as computer programs (e.g., including instructions)) and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0046] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals and vice versa, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via near-field communication radio components. Wireless communication may optionally employ any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution, Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE...). 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol that has not been developed as of the date of this document submission.

[0047] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a single-ear or dual-ear headphone) and input (e.g., a microphone).

[0048] I / O subsystem 106 couples input / output peripherals on device 100, such as touchscreen 112 and other input control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, depth camera controller 169, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / send electrical signals to the other input control device 116. The other input control device 116 optionally includes physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click dials, etc. In some embodiments, input controller 160 is optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing device such as mouse. One or more buttons (e.g., Figure 2 Optionally, 208) includes an increase / decrease button for volume control of speaker 111 and / or microphone 113. The one or more buttons optionally include a push-button (e.g., Figure 2(Ref. 206 in the original text). In some embodiments, the electronic device is a computer system that communicates with one or more input devices (e.g., via wireless communication, via wired communication). In some embodiments, the one or more input devices include a touch-sensitive surface (e.g., a touchpad, as part of a touch-sensitive display). In some embodiments, the one or more input devices include one or more camera sensors (e.g., one or more optical sensors 164 and / or one or more depth camera sensors 175), such as for tracking user gestures (e.g., hand gestures) as input. In some embodiments, the one or more input devices are integrated with the computer system. In some embodiments, the one or more input devices are separate from the computer system.

[0049] A quick press of the push button optionally disengages the touchscreen 112 from its lock or optionally initiates a process of unlocking the device using gestures on the touchscreen, as described in U.S. Patent Application 11 / 322,549 (i.e., U.S. Patent No. 7,657,849), filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," the entire contents of which are incorporated herein by reference. A long press of the push button (e.g., 206) optionally powers the device 100 on or off. The function of one or more buttons is optionally user-customizable. The touchscreen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0050] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0051] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In an exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.

[0052] Touchscreen 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. Touchscreen 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that from Apple Inc. (Cupertino, California). and iPod The technology used.

[0053] In some embodiments of the touchscreen 112, the touch-sensitive display optionally resembles a multi-touch touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touchscreen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.

[0054] The touch-sensitive display in some embodiments of the touchscreen 112 is described in the following applications: (1) U.S. Patent Application 11 / 381,313, filed May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. Patent Application 10 / 840,862, filed May 6, 2004, “Multipoint Touchscreen”; (3) U.S. Patent Application 10 / 903,964, filed July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. Patent Application 11 / 048,264, filed January 31, 2005, “Gestures For Touch Sensitive Input Devices”; and (5) U.S. Patent Application 11 / 038,590, filed January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input”. (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, “Multi-Functional Hand-Held Device”. The full text of all these applications is incorporated herein by reference.

[0055] Touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. Users optionally use any suitable object or accessory such as a stylus, finger, etc., to interact with touchscreen 112. In some embodiments, the user interface is designed to operate primarily through finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0056] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. Optionally, the touchpad is a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0057] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0058] The device 100 may optionally also include one or more optical sensors 164. Figure 1AAn optical sensor 164 is shown coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing images of the user to be optionally acquired for video conferencing while the user views other video conferencing participants on the touchscreen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lenses and sensor within the device housing), allowing a single optical sensor 164 to be used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.

[0059] The device 100 optionally also includes one or more depth camera sensors 175. Figure 1A A depth camera sensor is shown coupled to a depth camera controller 169 in I / O subsystem 106. Depth camera sensor 175 receives data from the environment to create a 3D model of an object (e.g., a face) within the scene from a viewpoint (e.g., the depth camera sensor). In some embodiments, in conjunction with imaging module 143 (also referred to as a camera module), depth camera sensor 175 may optionally be used to determine depth maps of different portions of an image captured by imaging module 143. In some embodiments, the depth camera sensor is located at the front of device 100, such that user images with depth information are optionally acquired for video conferencing while the user views other video conferencing participants on a touchscreen display, and selfies with depth map data are captured. In some embodiments, depth camera sensor 175 is located at the rear of the device, or both the rear and front of device 100. In some embodiments, the position of depth camera sensor 175 may be changed by the user (e.g., by rotating a lens and sensor within the device housing), such that depth camera sensor 175 is used in conjunction with a touchscreen display for both video conferencing and still image and / or video image acquisition.

[0060] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1AA contact strength sensor is shown coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100, opposite to the touchscreen display 112 located on the front of device 100.

[0061] The device 100 optionally also includes one or more proximity sensors 166. Figure 1A A proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 may optionally be coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 may optionally perform as described in the following U.S. patent applications: No. 11 / 241,839, entitled "Proximity Detector In Handheld Device"; No. 11 / 240,788, entitled "Proximity Detector In Handheld Device"; No. 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and No. 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, the proximity sensor is turned off and the touchscreen 112 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0062] The device 100 may optionally also include one or more tactile output generators 167. Figure 1AA haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.

[0063] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may be coupled to an input controller 160 in an I / O subsystem 106. The accelerometer 168 may optionally perform as described in the following U.S. patent publications: U.S. Patent Publication No. 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication No. 20060017692, entitled "Methods and Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 100 may optionally include, in addition to the accelerometer 168, a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for acquiring information about the location and orientation (e.g., portrait or landscape) of device 100.

[0064] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 ( Figure 1A ) or 370 ( Figure 3 Storage device / global internal state 157, such as Figure 1A and Figure 3 As shown in the figure. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, indicating what applications, views or other information occupy various areas of the touch screen display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and position information relating to the device's position and / or orientation.

[0065] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0066] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to… (Trademark of Apple Inc.) The same or similar and / or compatible multi-pin (e.g., 30-pin) connectors used in Apple Inc. devices.

[0067] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., touchpads or physical click-based rotary dials). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to single-point contact (e.g., single-finger contact) or multi-point simultaneous contact (e.g., "multi-touch" / multiple-finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0068] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0069] The touch / motion module 130 optionally detects gesture input performed by the user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0070] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0071] In some implementations, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like to specify the graphic to be displayed, and, if necessary, also receives coordinate data and other graphic attribute data, and then generates screen image data for output to the display controller 156.

[0072] The haptic feedback module 133 includes various software components for generating instructions which are used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0073] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0074] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to phone 138 for use in location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services, such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0075] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0076] • Contacts module 137 (sometimes called address book or contact list);

[0077] • Telephone module 138;

[0078] • Video conferencing module 139;

[0079] • Email client module 140;

[0080] • Instant Messaging (IM) module 141;

[0081] Fitness support module 142;

[0082] • Camera module 143 for still images and / or video images;

[0083] • Image management module 144;

[0084] • Video player module;

[0085] Music player module;

[0086] • Browser module 147;

[0087] • Calendar module 148;

[0088] • Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets acquired by the user, and user-created widgets 149-6;

[0089] • Widget creator module 150 for creating user-created widgets 149-6;

[0090] • Search module 151;

[0091] • Video and music player module 152, which combines a video player module and a music player module;

[0092] • Notes module 153;

[0093] • Map module 154; and / or

[0094] • Online video module 155.

[0095] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0096] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., in the application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0097] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 137, modify entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0098] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0099] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0100] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving an instant message, and viewing a received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0101] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes executable instructions for creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (executive devices); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing music for fitness activities; and displaying, storing, and transmitting fitness data.

[0102] In conjunction with the touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0103] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0104] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet according to user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as links to attachments and other files on web pages.

[0105] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0106] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is optionally a microapplication downloaded and used by a user (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or a user-created microapplication (e.g., user-created widget 149-6). In some embodiments, the widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! widgets).

[0107] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget creator module 150 can optionally be used by the user to create widgets (e.g., to convert user-specified portions of a webpage into widgets).

[0108] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0109] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0110] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the note-taking module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0111] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data related to shops and other points of interest at or near a specific location, and other location-based data) according to user instructions.

[0112] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions for performing the following actions: allowing users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Further descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.

[0113] Each of the above modules and applications corresponds to an executable set of instructions for performing one or more of the functions described above and the methods described in this patent application (e.g., computer-implemented methods and other information processing methods as described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs (such as computer programs (e.g., including instructions)), processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. For example, a video player module may optionally be combined with a music player module into a single module (e.g., Figure 1A (e.g., video and music player module 152). In some embodiments, memory 102 optionally stores subgroups of the aforementioned modules and data structures. Additionally, memory 102 optionally stores other modules and data structures not described above.

[0114] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on device 100 can be optionally reduced.

[0115] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main menu, home menu, or root menu. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0116] Figure 1B This is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0117] Event classifier 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0118] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0119] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral device interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.

[0120] In some implementations, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other implementations, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0121] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0122] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0123] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0124] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.

[0125] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.

[0126] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0127] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.

[0128] In some implementations, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other implementations, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit or a higher-level object from which application 136-1 inherits methods and other properties. In some implementations, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0129] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event based on the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0130] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a longitudinal orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).

[0131] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 112, and lifting off the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0132] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0133] In some implementations, the definition of the corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0134] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0135] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0136] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and delaying) the sub-event to the corresponding hit view. In some implementations, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with that tag retrieves the tag and executes a predefined procedure.

[0137] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined procedure.

[0138] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on touch-sensitive display.

[0139] In some implementations, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.

[0140] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0141] Figure 2A portable multifunction device 100 with a touchscreen 112 is shown according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some specific embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0142] Device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.

[0143] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic outputs for a user of device 100.

[0144] Figure 3This is a block diagram of an exemplary multi-functional device with a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the reference above). Figure 1A The tactile output generator 167 and sensor 359 (e.g., optical sensor, accelerometer, proximity sensor, touch sensor and / or contact intensity sensor (similar to the one mentioned above)) are described. Figure 1A The contact strength sensor 165 is described above. The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100. Figure 1A The memory 370 stores programs, modules, and data structures similar to those in the memory 102 of the portable multifunction device 100, or subsets thereof. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 (… Figure 1A The memory 102 may optionally not store these modules.

[0145] Figure 3Each of the elements described above is optionally stored in one or more memory devices of the previously mentioned memory devices. Each of the modules described above corresponds to a set of instructions for performing the functions described above. The modules or programs (e.g., instruction sets) described above need not be implemented as separate software programs (such as computer programs (e.g., including instructions)), processes, or modules, and therefore various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subgroup of the modules and data structures described above. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0146] Now let’s turn our attention to the implementation of the user interface, which is optionally implemented on, for example, a portable multifunction device 100.

[0147] Figure 4A An exemplary user interface for an application menu on a portable multifunction device 100 according to some embodiments is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof:

[0148] • Signal strength indicator 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0149] • Time 404;

[0150] Bluetooth indicator 405;

[0151] • Battery status indicator 406;

[0152] • Tray tray 408 with icons for commonly used applications, such as:

[0153] ○ The telephone module 138 has an icon 416 labeled "telephone", which optionally includes an indicator 414 indicating the number of missed calls or voicemail messages;

[0154] ○ An icon 418 labeled "Mail" in the email client module 140, which optionally includes an indicator 410 for the number of unread emails;

[0155] ○ The icon 420 labeled "Browser" in browser module 147; and

[0156] ○ The icon 422 labeled "iPod" for the video and music player module 152 (also known as the iPod module 152, a trademark of Apple Inc.); and

[0157] • Icons of other applications, such as:

[0158] ○The icon 424 of the IM module 141 marked as "Message";

[0159] ○The calendar module 148 has an icon 426 labeled "Calendar";

[0160] ○ The icon 428 of the image management module 144 is labeled "Photo".

[0161] ○ The icon 430 of camera module 143, which is labeled "camera";

[0162] ○ The icon 432 of the online video module 155, which is labeled "Online Video";

[0163] ○ The icon 434 labeled "Stock Market" in the Stock Market widget 149-2;

[0164] ○The icon 436 of the map module 154 that is labeled "map";

[0165] ○The weather widget 149-1 has icon 438 labeled "weather";

[0166] ○ The alarm clock widget 149-4 has an icon 440 labeled "clock";

[0167] ○ The icon 442 of the fitness support module 142 is labeled "fitness support";

[0168] ○ The icon 444 labeled "Notes" in Notes module 153; and

[0169] ○ The icon 446, labeled "Settings," is used to set the settings of the device 100 and its various applications 136.

[0170] It should be pointed out that, Figure 4A The icon labels shown are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels may be optionally used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0171] Figure 4B A touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)) is shown. Figure 3 Devices such as tablets or touchpads (e.g., 355) Figure 3An exemplary user interface on the device 300. The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 359) for detecting the intensity of contact on the tactile surface 451 and / or one or more tactile output generators 357 for generating tactile outputs for the user of the device 300.

[0172] While some examples of input on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given below, in some implementations the device detects input on a touch-sensitive surface separate from the display, such as... Figure 4B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a spindle (e.g., on the display (e.g., 450)). Figure 4B The spindle corresponding to 453 in the middle (e.g., Figure 4B (452 in the example). According to these embodiments, the device detects the position corresponding to the corresponding position on the display (e.g., in the example). Figure 4B In the middle, 460 corresponds to 468 and 462 corresponds to 470) is in contact with the touch-sensitive surface 451 (e.g., Figure 4B (460 and 462 in the text). Thus, when the touch-sensitive surface (e.g., ...) Figure 4B 451) and the display of a multi-functional device (e.g., Figure 4B When 450 is separated from the touch-sensitive surface, user input detected by the device on that touch-sensitive surface (e.g., touches 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may optionally be used for other user interfaces described herein.

[0173] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger touch, single-finger tap, finger swipe), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0174] Figure 5AAn exemplary personal electronic device 500 is illustrated. Device 500 includes a body 502. In some embodiments, device 500 may include components relative to devices 100 and 300 (e.g., Figures 1A to 4B Some or all of the features described herein. In some embodiments, device 500 has a touch-sensitive display 504, referred to below as touchscreen 504. As an alternative to or complement to touchscreen 504, device 500 has a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., touch). The one or more intensity sensors of touchscreen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on the intensity of the touch, meaning that touches of different intensities can invoke different user interface operations on device 500.

[0175] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, filed May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is incorporated herein by reference in its entirety.

[0176] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) may be physical. Examples of physical input mechanisms include push-buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watch straps, bangles, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow a user to wear device 500.

[0177] Figure 5B An exemplary personal electronic device 500 is depicted. In some embodiments, device 500 may include a reference retrieval system. Figure 1A , Figure 1B and Figure 3 Some or all of the components described herein. Device 500 has a bus 512 that operatively couples I / O portion 514 to one or more computer processors 516 and memory 518. I / O portion 514 may be connected to display 504, which may have touch-sensitive components 522 and optionally have an intensity sensor 524 (e.g., a contact intensity sensor). Furthermore, I / O portion 514 may be connected to communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. Input mechanism 506 is optionally a rotatable input device. In some examples, input mechanism 508 is optionally a button.

[0178] In some examples, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to the I / O section 514.

[0179] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, may cause the computer processors to perform the techniques described below, including process 700 (e.g., Figures 7A to 7BA computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. Personal electronic devices are not limited to... Figure 5B It can be the components and configurations, or it can include other components or additional components in a variety of configurations.

[0180] As used herein, the term "power indication" refers optionally to the power indication in devices 100, 300, and / or 500 ( Figure 1A , Figure 3 and Figures 5A to 5B A user-interactive graphical user interface object displayed on a screen. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) optionally each constitute a functional representation.

[0181] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other positional marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3 The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of enabling direct interaction with user interface elements on the touchscreen display) Figure 1A The touch-sensitive display system 112 or Figure 4AIn some embodiments of the touchscreen 112, a touch detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that delivers the user-expected interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0182] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted away, before or after contact begins to move, before contact ends, before or after contact intensity is detected to increase and / or before or after contact intensity decreases). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some implementations, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the feature intensity and one or more thresholds is used to determine whether to perform one or more actions (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.

[0183] In some implementations, a portion of the gesture is identified to determine the characteristic intensity. For example, a touch-sensitive surface optionally receives a series of swipe contacts that transition from a starting position to an ending position, where the contact intensity increases. In this example, the characteristic intensity of the contact at the ending position is optionally based only on a portion of the series of swipe contacts, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the ending position). In some implementations, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: unweighted moving average smoothing algorithm, triangular smoothing algorithm, median filter smoothing algorithm, and / or exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic intensity.

[0184] Optionally, the contact intensity on a touch-sensitive surface is characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device performs an operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device performs an operation different from the operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity lower than the light press intensity threshold (e.g., and higher than the nominal contact detection intensity threshold, where contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector based on the movement of the contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.

[0185] An increase in contact intensity from below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact intensity from below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact intensity from below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact intensity from above a contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact being lifted off the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0186] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., a "downward stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing to above a press input intensity threshold and the intensity of the contact subsequently decreasing to below the press input intensity threshold, and the corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the press input threshold (e.g., an "upward stroke" of the corresponding press input).

[0187] In some implementations, the device employs intensity hysteresis to avoid unintended inputs sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Therefore, in some implementations, a press input includes an increase in the intensity of the corresponding contact above the press input intensity threshold and a subsequent decrease in the intensity of that contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some implementations, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and optionally the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity threshold, and a corresponding operation is performed in response to detecting a press input (e.g., an increase or decrease in contact intensity depending on the environment).

[0188] Figure 5C An exemplary electronic device 580 is illustrated. Device 580 includes a body 580a. In some embodiments, device 580 may include components relative to devices 100, 300, and 500 (e.g., Figures 1A to 5BThe device 580 may include some or all of the features described herein. In some embodiments, the device 580 has one or more speakers 580b (hidden within a body 580a), one or more microphones 580c, one or more touch-sensitive surfaces 580d, and one or more displays 580e. Alternatively, or as a supplement to the displays and touch-sensitive surfaces 580d, the device may also have a touch-sensitive display (also referred to as a touchscreen). Similar to devices 100, 300, and 500, in some embodiments, the touch-sensitive surface 580d (or touchscreen) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., a touch). The one or more intensity sensors of the touch-sensitive surface 580d (or touchscreen) may provide output data representing the intensity of the touch. The user interface of the device 580 may respond to these touches based on the intensity of the touches, meaning that touches of different intensities may invoke different user interface operations on the device 580. In some embodiments, the one or more displays 580e are one or more light-emitting diodes (LEDs). For example, the display may be a single LED, an LED cluster (e.g., red, green, and blue LEDs), multiple discrete LEDs, multiple discrete LED clusters, or other arrangements of one or more LEDs. For example, display 580e may be an array of nine discrete LED clusters arranged in a circular shape (e.g., a ring). In some examples, the one or more displays may contain one or more light-emitting elements of another type.

[0189] Figure 5D An exemplary personal electronic device 580 is depicted. In some embodiments, device 580 may include, relative to... Figure 1A , Figure 1B , Figure 3 and Figures 5A to 5B Some or all of the components described herein. Device 580 has a bus 592 operatively coupling I / O portion 594 to one or more computer processors 596 and memory 598. I / O portion 594 may be connected to display 582, which may have touch-sensitive component 584 and optionally have intensity sensor 585 (e.g., contact intensity sensor). In some embodiments, touch-sensitive component 584 is a component separate from display 582. Furthermore, I / O portion 594 may be connected to communication unit 590 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 580 may include input mechanism 588. In some examples, input mechanism 588 is optionally a button. In some examples, input mechanism 588 is optionally a microphone. Input mechanism 588 is optionally a plurality of microphones (e.g., microphone array).

[0190] Electronic device 580 includes a speaker 586 for outputting audio. Device 580 may include audio circuitry (e.g., in I / O section 594) that receives audio data, converts the audio data into electrical signals, and transmits the electrical signals to speaker 586. Speaker 586 converts the electrical signals into sound waves that are audible to humans. The audio circuitry (e.g., in I / O section 594) also receives electrical signals converted from sound waves by a microphone (e.g., input mechanism 588). The audio circuitry (e.g., in I / O section 594) converts these electrical signals into audio data. The audio data may optionally be retrieved by I / O section 594 from and / or transmitted to memory 598 and / or RF circuitry (e.g., in communication unit 590).

[0191] The memory 598 of the personal electronic device 580 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 596, may cause the computer processors to perform the techniques described below, including process 700 (e.g., Figures 7A to 7B A computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but is not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. Personal electronic devices are not limited to... Figure 5D It can be the components and configurations, or it can include other components or additional components for various configurations.

[0192] For ease of explanation, optionally, the description of an operation triggered in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is provided in response to detecting any of the following conditions: the contact strength increases to above the press input strength threshold, the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold, the contact strength decreases to below the press input strength threshold, and / or the contact strength decreases to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example where the operation is described as being performed in response to detecting a decrease in contact strength below the press input strength threshold, the operation is optionally performed in response to detecting a decrease in contact strength below a hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0193] Now let’s turn our attention to the implementation of user interfaces (“UIs”) and associated processes on electronic devices such as portable multifunction devices 100, 300 and / or 500.

[0194] Figures 6A to 6K Exemplary techniques and user interfaces for transmitting audio using a computer system, according to some embodiments, are illustrated. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 7A to 7B The process in.

[0195] Figure 6A An exemplary display system 602 (e.g., "Jane's TV") is shown in operational communication with a remote control 604, earbuds 606a (e.g., "Jane's Earbuds"), earbuds 606b (e.g., "Nick's Earbuds"), and a kitchen speaker 606c. Figure 6A In this embodiment, display system 602 is a multimedia device (e.g., a device for storing and / or generating multimedia content for output at the device and / or at another output device such as a display). Display system 602 is linked to earbuds 606a, 606b, and a kitchen speaker 606c, respectively represented by communication links 608a-608c. Remote control 604 is also linked to display system 602, such that gestures received (or detected) via remote control 604 are transmitted to display system 602 for processing. In some embodiments, devices (e.g., display system, a pair of headphones (e.g., earbuds), a telephone, a speaker) are devices that are in operative communication with another device (e.g., via wireless or wired media) or connected (paired) with that other device (e.g., via Bluetooth, NFC, or a network connection).

[0196] In some implementations, the display system 602, rather than the multimedia device itself, is directly linked to the multimedia device that provides multimedia content for output at the display system 602. Figure 6A In this context, 670 represents multimedia content (e.g., media items) displayed on the display system 602. In some embodiments, the display system 602 or a multimedia device linked to the display system 602 includes one or more features of devices 100, 300, and / or 500. In some embodiments, the multimedia device is configured to share media with the display system 602 and one or more earbuds, including earbuds 606a, earbuds 606b, and / or a kitchen speaker 606c.

[0197] In some embodiments, the multimedia device and / or display system 602 is also linked to a remote control 604, such that gestures detected via the remote control 604 are transmitted to the multimedia device for processing in a scenario where the display system 602 is linked to the multimedia device. In some embodiments, the multimedia device (e.g., linked to the display system 602) causes the display system 602 to display one or more user interface elements in response to receiving one or more gestures (or instructions for one or more gestures transmitted from the remote control 604). In some embodiments, the multimedia devices share media such that media (e.g., audio) is available for output by earbuds 606a, earbuds 606b, and / or kitchen speaker 606c. Therefore, for simplicity, the following description will generally refer to the display system 602 performing actions, belonging to an entity (e.g., associated with an account), or linked to one or more devices (e.g., headphones, telephone). However, the following description envisions and does not exclude scenarios where one or more multimedia devices are linked to the display system 602 and one or more devices. In these scenarios, the multimedia device (rather than the display system) may be an entity and / or perform one or more actions (e.g., such as causing one or more user interfaces to be displayed on the display system 602, receiving one or more gestures, sharing media, and / or performing other logic described herein with respect to the display system 602).

[0198] exist Figure 6A In this embodiment, display system 602 belongs to Jane (e.g., "Jane's TV"). Furthermore, earbud 606a is a headset belonging to Jane (e.g., associated with a user account that includes an identifier or name, i.e., "Jane"), and earbud 606b is a headset belonging to Nick. Therefore, in some embodiments, when a device is associated with a user account that includes an identifier or name representing an entity (e.g., login), the device belongs to an entity (e.g., a person, building, house). In some embodiments, kitchen speaker 606c is associated with the same user account as Jane. In some embodiments, kitchen speaker 606c includes one or more features of devices 100, 300, 500, and / or 580.

[0199] like Figure 6A As shown, display system 602 is configured to share audio corresponding to multimedia content being presented on display system 602. As indicated by audio outputs 610a and 612a, earbuds 606a and 606b concurrently output audio including the phrase "THE CAR," synchronized with the multimedia content shown by representation 670 and presented on display system 602. Therefore, in Figure 6AAt this point, earbuds 606a and 606b effectively output the same audio content as the multimedia content shown by representation 670 (e.g., one or more audio tracks included in or associated with the content).

[0200] The following section describes Figures 6B to 6C This illustrates that earbuds 606a and 606b are configured to output the same audio content as the multimedia content represented by representation 670. Figure 6A At this point, the remote control 604 receives the downward gesture 650a and transmits the instruction of the downward gesture 650a to the display system 602 for processing.

[0201] like Figure 6B As shown, in response to receiving a downward gesture 650a, the display system 602 displays an audio settings overlay 620 on top of the multimedia content display 670. Figure 6B Because audio configuration indicator 622 is selected, audio settings overlay 620 is displayed, as indicated by the ellipse surrounding audio configuration indicator 622. Audio settings overlay 620 includes speaker header 624 and headphone header 626. Below speaker header 624, a number of speaker controls are listed, each of which is a control for sharing media from display system 602 to a specific speaker. Speaker controls include kitchen speaker control 624a and dining room speaker control 624b. Each speaker control is displayed below speaker header 624 because the speaker (e.g., kitchen speaker 606c) is linked to display system 602, previously linked to display system, or configured to receive audio from display system. Headphone header 626 lists headphones linked to display system 602, previously linked to display system, or configured to receive audio from display system. Headphone header 626 includes earbud controls 626a (“Jane’s Earbuds”) and earbuds 626b. Figure 6B In the display system 602, checkmarks are displayed to the right of each earbud control in earbud controls 626a and 626b. The checkmarks displayed next to earbud controls 626a and 626b indicate that the display system 602 is configured to share audio (e.g., send audio data) for output to earbuds 606a and 606b.

[0202] like Figure 6B As shown, display system 602 displays focus selector 628 around earbud control 626a. Because focus selector 628 is displayed around earbud control 626a, display system 602 displays a set of audio configurations for sharing audio output to earbud 606a. Figure 6BIn the middle, the audio configuration for sharing the audio output to the earpiece 606a is displayed to the left of the speaker title 624 and the headphone title 626, and includes language configuration 630, audio description configuration 632, sound configuration 634 and narration configuration 636.

[0203] The language configuration 630 includes multiple controls for selecting a specific type of language for the audio, which is transmitted to and output by a set of headphones. Figure 6B In the language configuration 630, there are English language setting controls 630a, French language setting controls 630b, and Spanish language setting controls 630c. For example... Figure 6B As shown, the English language setting control 630a is selected (for example, as indicated by the check mark next to the English language setting control 630a). Therefore, in Figure 6B In this configuration, display system 602 is currently configured to output audio in English to earpiece 606a corresponding to the multimedia content being presented on display system 602. In some embodiments, display system 602 outputs audio including a specific audio track included with the media item. For example, when English language setting control 630a is selected, display system 602 transmits an English audio track included to (or corresponding to) the multimedia content being presented on display system 602, but when French language setting control 630b is selected, display system 602 transmits a French audio track included to (or corresponding to) the multimedia content being presented on display system 602 (e.g., the same multimedia content).

[0204] Audio description configuration 632 includes multiple controls for selecting a specific type of descriptive audio that is transmitted to and output by a set of headphones. Figure 6B In the audio description configuration 632, there are standard audio description setting controls 632a and descriptive audio description setting controls 632b. Therefore, in Figure 6BBecause the standard audio description setting control 632a is selected (e.g., as indicated by the check mark next to the standard audio description setting control 632a), the display system 602 is currently configured to output standard audio corresponding to the multimedia content being presented on the display system 602 to the earpiece 606b. When the standard audio description setting control 632a is selected, the display system 602 transmits less audio content (e.g., less audio content per second, per minute, or per hour) compared to the audio transmitted by the display system 602 when the descriptive audio description setting control 632b is selected. In some embodiments, when the descriptive audio description setting control 632b is selected, the audio transmitted by the display system 602 includes more audio content describing the scene and / or the actions of people or objects compared to the audio transmitted by the display system 602 when the standard audio description setting control 632a is selected. In some embodiments, the display system 602 selects a specific audio track of the multimedia content to transmit to a set of output devices (e.g., earpieces, speakers) based on whether a specific type of audio description setting control is selected.

[0205] The sound configuration group 634 includes multiple controls for selecting a specific type of sound configuration for audio, which is transmitted to and output by a set of headphones. Figure 6B In the sound configuration group 634, there are full dynamic range setting controls 634a and louder sound reduction setting controls 634b. Figure 6B Because the full dynamic range setting control 634a is selected (e.g., as indicated by the check mark next to the full dynamic range setting control 634a), the display system 602 is currently configured to output audio to the earpiece 606b with a larger sound range compared to when the reduce loud volume setting control 634b is selected. In some embodiments, the display system 602 selects a specific audio track of multimedia content to be transmitted to a set of output devices (e.g., earpieces, speakers) based on whether a specific type of sound configuration setting control is selected.

[0206] The narration configuration 636 includes multiple controls for selecting whether to transmit narration audio to and from a set of headphones. Figure 6B In the narration configuration group 636, there are an on control 636a and an off control 636b. Figure 6BIn some embodiments, because the off control 636b is selected (e.g., as indicated by the check mark next to the off control 636b), no narration audio is transmitted to and output by the earpiece 606a. In some embodiments, when the on control 636a is selected, the display system 602 generates narration audio unrelated to the multimedia content. In some embodiments, the display system 602 transmits the narration audio to the earpiece 606a along with an audio track corresponding to the multimedia content displayed on the display system 602. In some embodiments, the narration audio includes audio content corresponding to instructions for navigation and / or interaction with the display system 602 and / or the remote control 604. In some embodiments, the narration audio includes audio content unrelated to the multimedia content. In some embodiments, the narration audio content is not included in the audio track corresponding to the multimedia content. Figure 6B At this point, the remote control 604 receives the downward gesture 650b and transmits the instruction of the downward gesture 650b to the display system 602 for processing.

[0207] like Figure 6C As shown, in response to receiving a downward gesture 650b, the display system 602 moves the focus selector 628 downwards, causing the focus selector 628 to appear around the earbud control 626b instead of the earbud control 626a. Because the focus selector 628 is displayed around the earbud control 626a, the display system 602 displays a set of audio configurations for output to the earbud 606b. It should be noted that in Figure 6C In this configuration, the audio configuration for outputting audio to earbud 606b is the same as the audio configuration for outputting audio to earbud 606a. Therefore, in Figure 6C In this configuration, the display system 602 is configured to transmit standard audio in English with a full dynamic range and no narration to earbuds 606a and 606b (e.g., as indicated by checkmarks next to the English language setting control 630a, standard audio description setting control 632a, full dynamic range setting control 634a, and turn-off control 636b). Furthermore, because earbuds 606a and 606b have the same audio configuration, earbuds 606a and 606b... Figures 6B to 6C The outputs the same audio (e.g., "IS" (e.g., 610b and 612b) and "GOING" (e.g., 610c and 612c)). In some embodiments, earbuds 606a and 606b have the same (or different) audio configurations when display system 602 is playing back multimedia content different from the multimedia content represented by representation 670 and / or at the start of a media playback session. In some embodiments, the audio configuration settings remain unchanged between media playback sessions (e.g., turning the device off / on, stopping / initiating the start of media playback).

[0208] Figures 6D to 6G An exemplary scenario is shown where the audio configuration of earbud 606b is changed so that the audio configuration of earbud 606b differs from that of earbud 606a. Figure 6C At this point, the remote control 604 receives the left gesture 650c and transmits the instruction of the left gesture 650c to the display system 602 for processing.

[0209] like Figure 6D As shown, in response to receiving a leftward gesture 650c, the display system 602 horizontally moves the focus selector 628, positioning the focus selector 628 around the English language setting control 630a. Figure 6D Because earbuds 606a and 606b have the same audio configuration, the display system 602 continues to transmit the same audio (e.g., "VERY" 610d and 612d) to earbuds 606a and 606b. Figure 6D At this point, the remote control 604 receives the downward gesture 650d and transmits the instruction of the downward gesture 650d to the display system 602 for processing.

[0210] like Figure 6E As shown, in response to receiving a downward gesture 650d, the display system 602 moves the focus selector 628 downward along the display, positioning the focus selector 628 around the Spanish language setting control 630c. Figure 6E Because earbuds 606a and 606b have the same audio configuration, the display system 602 continues to transmit the same audio (e.g., "VERY" 610e and 612e) to earbuds 606a and 606b. Figure 6E At this point, the remote control 604 receives the pressing gesture 650e and transmits the instruction of the pressing gesture 650e to the display system 602 for processing.

[0211] like Figure 6F As shown, in response to detecting a press gesture 650e, the display system 602 displays the Spanish language setting control 630c as selected (e.g., a checkmark next to the Spanish language setting control 630c) and stops displaying the English language setting control 630a as selected. Additionally, in response to detecting a press gesture 650e, the display system 602 changes the audio configuration of the audio being sent to the earphone 606b. Therefore, when the audio configuration of the audio sent to the earphone 606b changes, the earphone 606b outputs audio corresponding to the Spanish track of the multimedia content (e.g., ...). 614f). On the other hand, because the audio configuration of earbud 606a has not yet changed in response to the detection of a press gesture 650e, earbud 606a continues to output audio corresponding to the English track of the multimedia content (e.g., "FAST" 610f). Figure 6F At this point, the remote control receives the pressing gesture 650f on the menu button and transmits the instruction of the pressing gesture 650f to the display system 602 for processing.

[0212] like Figure 6G As shown, in response to receiving a press gesture 650f, the display system 602 stops displaying the audio settings overlay 620. In response to receiving the press gesture 650f, the display system 602 also maintains the display of the representation 670 indicating the multimedia content being displayed. Figure 6G In this process, the display system 602 continues to transmit English audio (e.g., "GOODBYE" 610g) to earbud 606a and Spanish audio (e.g., "ADIOS" 614g) to earbud 606b based on the audio configuration of each earphone. It should be noted that... Figure 6G Meanwhile, the audio continues to be synchronized with the multimedia content being displayed on the display system 602.

[0213] Figures 6G to 6K An exemplary scenario is shown where the display system 602 is configured to share media with the kitchen speaker 606c using an audio configuration based on historical usage settings. Figure 6G At this point, the remote control 604 receives the downward gesture 650g and transmits the instruction of the downward gesture 650g to the display system 602 for processing.

[0214] like Figure 6H As shown, in response to receiving a downward gesture 650g, the display system 602 redisplays the audio settings overlay 620 on top of the multimedia content display 670. Figure 6H In the center, the focus selector 628 is displayed around the earbud control 626b. Figure 6H At the same time, the remote control 604 receives a pressing gesture 650h pointing at the earpiece control 626b and transmits the instruction of the pressing gesture 650h to the display system 602 for processing.

[0215] like Figure 6I As shown, in response to receiving a press gesture 650h, the display system 602 stops being configured to share audio with the earpiece 606b. It should be noted that... Figure 6IIn this configuration, because the display system 602 does not share multimedia with the earphone 606b, the earphone 606b does not output any audio. On the other hand, because the display system 602 remains configured to share audio with the earphone 606a, the earphone 606a continues to output audio (e.g., "SOON" 610i). To indicate that the display system 602 is not configured to share audio with the earphone 606b, the display system 602 also displays the earphone control 626b as unselected (e.g., removing the check mark previously next to the earphone control 626b). Figure 6I At this point, the remote control 604 receives the upward gesture 650i and transmits the instruction of the upward gesture 650i to the display system 602 for processing.

[0216] like Figure 6J As shown, in response to receiving an upward gesture 650i, the display system 602 moves the focus selector 628 upward, positioning the focus selector 628 around the kitchen speaker control 624a. Figure 6J For reasons similar to those mentioned above, the display system 602 continues to transmit English audio (e.g., “UNTIL NEXT” 610j) to earpiece 606a, but does not transmit audio to earpiece 606b. Figure 6J At the same time, the remote control 604 receives a pressing gesture 650j directed at the kitchen speaker controller 624a and transmits the instruction of the pressing gesture 650j to the display system 602 for processing.

[0217] like Figure 6K As shown, in response to receiving a press gesture 650j, the display system 602 is configured to share audio with the kitchen speaker 606c (e.g., as indicated by the check mark appearing next to the kitchen speaker control 624a). Additionally, in Figure 6K At this location, display system 602 displays the audio configuration settings of kitchen speaker control 624a. The audio configuration settings of kitchen speaker control 624a are automatically selected based on the historical usage of kitchen speaker 606c (e.g., without user input). For example, because kitchen speaker 606c was connected to display system 602 during a previous session (e.g., a previous time when display system 602 was connected to display system 602), it is determined that the speaker was previously used when the entity associated with kitchen speaker 606c viewed specific multimedia content presented on display system 602. Figure 6K The audio configuration settings are set in the middle, so those audio settings can be displayed.

[0218] like Figure 6KAs shown, the display system 602 is configured to transmit descriptive audio in English with a full dynamic range and narration (e.g., indicated by checkmarks next to the English language setting control 630a, descriptive audio description setting control 632b, full dynamic range setting control 634a, and on control 636a). Audio output 616k indicates audio configuration settings based on the kitchen speaker 606c. Figure 6K As shown, the English audio output 616k includes descriptive audio (e.g., audio about a scene or action, such as “THE CAR EXITS THE SCENE”) and narration audio (e.g., audio about the user interface or navigation of the user interface, such as “CLICK 'MENU' TO EXIT”).

[0219] While the above scenario has been described as using a display system 602 to change the audio configuration of output devices (e.g., earbuds 606a, earbuds 606b, and kitchen speaker 606c), the remote control 604 may display one or more portions of the audio settings overlay 620, allowing a user to change the audio configuration settings by selecting one or more user interface objects displayed on the remote control 604. In some embodiments, the user may change the audio configuration settings using only the user interface elements displayed on the remote control 604. In some embodiments, the user interface elements are displayed on the top portion of the remote control 604 (e.g., above the menu button) or on a portion of the remote control 604 not surrounded by hardware buttons (such as menu buttons, volume controls, etc.).

[0220] Figures 7A to 7BThis is a flowchart illustrating a method 700 for transmitting audio using a computer system according to some embodiments. Method 700 is performed at a computer system (e.g., 100, 300, 500, 602) communicating with one or more input devices (e.g., 604) (e.g., an external remote control; an external computing device). (In some embodiments, the computer system communicates with display generation components (e.g., a display controller, a touch-sensitive display system, a television screen).) In some embodiments, the one or more input devices are configured to receive input, such as a touch-sensitive surface and / or a remote control that receives user input. In some embodiments, the one or more input devices are integrated with the computer system. In some embodiments, the one or more input devices are separate from the computer system. Therefore, the computer system can transmit data (e.g., image data or video data) via wired or wireless connections to integrated or external display generation components to visually generate content (e.g., using a display device), and can receive input from one or more input devices via wired or wireless connections. Some operations in method 700 may be arbitrarily combined, the order of some operations may be arbitrarily changed, and some operations may be arbitrarily omitted.

[0221] In some embodiments, the computer system optionally communicates with a display generation component (602 in some embodiments) (e.g., wired communication, wireless communication). The display generation component is configured to provide visual output, such as a display via a CRT monitor, a display via an LED monitor, or a display via image projection. In some embodiments, the display generation component is integrated with the computer system. In some embodiments, the display generation component is separate from the computer system.

[0222] As described below, method 700 provides an intuitive way to transmit audio using a computer system. This method reduces the cognitive burden on users when transmitting audio using a computer system, thereby creating a more efficient human-computer interface. For battery-powered computing devices, enabling users to transmit audio quickly and more efficiently saves power and increases the time interval between battery charges.

[0223] When a media project (e.g., represented by 670) corresponding to multiple audio tracks (e.g., audio media (e.g., music), video media (e.g., movies, television programs)) (e.g., playback media project) is played back (e.g., on one or more external speaker devices, or on one or more display generation components in communication with a computer system), the computer system transmits (702) (e.g., indirectly (e.g., via an intermediate device and / or server) or directly) the first audio (e.g., 610a-610k, 612a-612e) corresponding to the first audio track among the multiple audio tracks to a first device (e.g., 606a) (e.g., headphones, one or more speakers) and a second device (e.g., 606b) (e.g., headphones, one or more speakers) to a second device (e.g., 606b) (e.g., headphones, one or more speakers) (e.g., 702) (e.g., indirectly (e.g., via an intermediate device and / or server) or directly ... (e.g., audio signal) to a second audio track among the multiple audio tracks, to a first device (e.g., 610a-610k, 612a-612e). In some implementations, the multiple audio tracks include different language tracks (e.g., English audio track, Spanish audio track), different levels of detail (e.g., descriptive audio tracks (e.g., for the visually impaired), standard audio tracks), audio tracks that identify one or more user interface elements and / or provide details about how a user can interact with the user interface (e.g., narration audio tracks, non-narration audio tracks)). In some implementations, the multiple audio tracks are distinct from each other, wherein each corresponding audio track has a different amount of utterances (e.g., spoken utterances (e.g., words)) and different utterances (e.g., spoken utterances (e.g., words)). In some implementations, the computer system communicates with display generation components (e.g., display controller, touch-sensitive display system, television screen).

[0224] When transmitting a first audio (e.g., 606a) corresponding to a first audio track (e.g., including or represented therein) among a plurality of audio tracks to a first device (e.g., 606a) (e.g., headphones, one or more speakers) and a second device (e.g., 606b) (e.g., headphones, one or more speakers) (e.g., external electronic devices (e.g., outside the computer system)), the computer system receives (704) a request (e.g., 650e) via one or more input devices (e.g., 604) to transmit a second audio (e.g., 614f-614h) (e.g., an audio signal) corresponding to a second audio track among a plurality of audio tracks that is different from the first audio track. In some implementations, one or more audio tracks in the second set of audio tracks (e.g., a specific language track related to the media project (e.g., an English audio track, a Spanish audio track), an audio track with a specific level of detail (e.g., a descriptive audio track, a standard audio track), an audio track that identifies / does not identify one or more user interface elements and / or provides details about how the user can interact with the user interface (e.g., a narration audio track, a non-narration audio track)) correspond to the media project.

[0225] In response to (706) receiving a request to transmit (and when causing playback media items) a second audio corresponding to the second audio track (including therein) among a plurality of audio tracks to the second device, the computer system transmits (708) (e.g., indirectly (e.g., via an intermediate device and / or server) or directly) the second audio corresponding to the second audio track among the plurality of audio tracks (e.g., 614f-614h) to the second device (e.g., 606b).

[0226] In some implementations, as part of transmitting a second audio (e.g., 614f-614h) corresponding to a second audio track among a plurality of audio tracks to a second device (e.g., 606b, 606c), the computer system stops (710) (e.g., stops its transmission) transmitting a first audio (e.g., 610a-610k, 612a-612e) corresponding to a first audio track among a plurality of audio tracks to the second device. In some implementations, in response to receiving a request to transmit a second audio (e.g., including, or represented therein) corresponding to a second audio track among a plurality of audio tracks to the second device, the computer system stops transmitting the first audio corresponding to a first audio track among a plurality of audio tracks to the second device. Stopping the transmission of the first audio corresponding to a first audio track among a plurality of audio tracks to the second device while transmitting the second audio corresponding to a second audio track among a plurality of audio tracks improves the efficiency of the computer system because the computer system only needs to cause playback of one instance or media item, rather than causing playback of multiple instances of the media item to be adjusted for transmission to the device. Improving the efficiency of computer systems reduces power consumption and extends battery life by enabling users to use them more safely and efficiently.

[0227] In response to (706) receiving a request to transmit (and when causing playback of a media item) a second audio track (including, and represented therein) corresponding to a second audio track among a plurality of audio tracks to a second device, the computer system continues (712) to transmit to the first device a first audio track (e.g., 610f-610k) corresponding to a first audio track among the plurality of audio tracks (e.g., without transmitting the second audio track (e.g., the second audio track including the second audio track of the media item)). In some embodiments, the second audio track includes audio tracks included in the first audio track, and / or the second audio track includes audio tracks not included in the first audio track. In some embodiments, the first audio track and the second audio track are synchronized with the media item being played back. In some embodiments, audio tracks included in the first audio track and / or the second audio track (e.g., a speech track) correspond to the media item. In some embodiments, audio tracks included in the first audio track and / or the second audio track (e.g., a narration track) do not correspond to the media item. In some embodiments, transmitting audio to a device causes the device to output the audio track. Responding to a request to transmit a second audio track corresponding to a second audio track among multiple audio tracks to a second device, and continuing to transmit a first audio track corresponding to a first audio track among multiple audio tracks to a first device, improves the efficiency of the computer system because the computer system only needs to cause one instance of the playback media item, rather than causing multiple instances of the playback media item, to transmit different audio to different devices. Improving the efficiency of the computer system reduces power consumption and extends the battery life of the computer system by enabling users to use the computer system more safely and efficiently. Responding to a request to transmit a second audio track corresponding to a second audio track among multiple audio tracks to a second device, and continuing to transmit a first audio track corresponding to a first audio track among multiple audio tracks to a first device, provides enhanced control over transmitting specific audio to a specific device. Providing enhanced control over transmitting specific audio to a specific device makes the user-device interface more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the battery life of the device by enabling users to use the device more quickly and efficiently.

[0228] In some implementations, the second audio (e.g., 614f-614h) and the first audio (e.g., 610a-610k) have the same duration. In some implementations, when transmitting the second audio (e.g., 614f-614h) and the first audio (e.g., 610a-610k), both are synchronized with the media being played back (e.g., 670) (e.g., the first and second audio are played back at their respective positions corresponding to the corresponding positions on the media being played back). In some implementations, the media item is video (e.g., 670), and the first audio (e.g., 610a-610k) and the second audio (e.g., 614f-614h) are synchronized with the video.

[0229] In some implementations, the first audio track is generated from the media project (e.g., non-narration) (e.g., retrieved or calculated from data stored with the media project), while at least one audio track included in the second audio track is generated separately from the media project (e.g., narration).

[0230] In some embodiments, the second audio (e.g., 614f-614h) includes an audio track not included in the first and second audio tracks (e.g., narration). In some embodiments, the second audio (e.g., 614f-614h) does not include the first audio track (e.g., non-narration, when the first and second audio tracks are in two different languages ​​or have different levels of detail). In some embodiments, when the second audio (e.g., 614f-614h) does not include the first audio track, the first audio (e.g., 610a-610k) and the second audio (e.g., 614f-614h) do not include an audio track generated separately from the media project (e.g., represented by 670).

[0231] In some embodiments, the computer system communicates with a first device (e.g., 606a) via a first connection (e.g., 608a) (e.g., Bluetooth connection and / or pairing (e.g., direct or indirect pairing via an intermediary device, such as a cellular phone)). In some embodiments, the computer system communicates with a second device (e.g., 606b, 606c) via a second connection (e.g., 608b) different from (e.g., independent of) the first connection (e.g., 608a) (e.g., Bluetooth connection and / or pairing (e.g., direct or indirect pairing via an intermediary device, such as a cellular phone)). In some embodiments, the first and second devices are connected to the computer system independently.

[0232] In some embodiments, the first device (e.g., 606a) is a first type of device (e.g., a pair of headphones (e.g., in-ear headphones, ear-hook headphones, wired headphones, wireless and / or Bluetooth headphones)). In some embodiments, the second device (e.g., 606c) is a second type of electronic device, different from the first type of device (e.g., a set of smart speakers (e.g., wireless and / or Bluetooth speakers)). In some embodiments, the first device has one or more features (e.g., software features), capabilities, and / or hardware components that the second device does not possess.

[0233] In some embodiments, as part of receiving a request (e.g., 650e) to transmit second audio (e.g., 612e-612h) corresponding to a second audio track among a plurality of audio tracks to a second device (e.g., 606b), the computer system receives (e.g., detects) a selection of a language option (e.g., 630a-630b) via one or more input devices (e.g., 604). This selection is a selectable user interface object (e.g., corresponding to the selection of a language track) (e.g., English audio option, Spanish audio option)). In some embodiments, the language option is displayed by the computer system (or caused to be displayed on a monitor connected to the computer system). In some embodiments, the second audio track (e.g., 612e-612h) includes audio content corresponding to the language of the language option (e.g., 630a-630b) (e.g., the second audio track is a language track corresponding to the language option). In some embodiments, the first audio track includes first audio content corresponding to the language of the first language option (e.g., English), and the second track includes second audio content corresponding to the language of the second language option (e.g., Spanish). In some implementations, the first language option is different from (or represents a different language) (e.g., visually different from) the second language option. In some implementations, the first language is different from the second language. Transmitting the second audio, including audio content corresponding to the language option, after receiving a request to transmit the second audio corresponding to the second audio track among multiple audio tracks to the second device, includes receiving the selection of the language option via one or more input devices. This improves the efficiency of the computer system because the computer system only needs to cause playback of one instance or media item, rather than multiple instances of the media item, to transmit the audio corresponding to the language option to the device. Improved computer system efficiency reduces power consumption and extends battery life by enabling users to use the computer system more safely and efficiently. Transmitting the second audio, including audio content corresponding to the language option, after receiving a request to transmit the second audio corresponding to the second audio track among multiple audio tracks to the second device, includes receiving the selection of the language option via one or more input devices. This provides enhanced control over transmitting specific audio to a specific device. Providing enhanced control over the transmission of specific audio to specific devices makes user-device interfaces more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with devices), which in turn reduces power consumption and extends device battery life by enabling users to use devices more quickly and efficiently.

[0234] In some embodiments, as part of receiving a request to transmit second audio corresponding to a second audio track among a plurality of audio tracks to a second device (e.g., 608b, 608c), the computer system receives (e.g., detects) a selection of a detail level option (e.g., 632a-632b) via one or more input devices (e.g., 604). This selection is made via a selectable user interface object (e.g., corresponding to the selection of the detail level option, e.g., standard audio option, descriptive audio option)). In some embodiments, the detail level option is displayed by the computer system (or caused to be displayed on a monitor connected to the computer system). In some embodiments, the second audio track has audio content corresponding to a quantity of the detail level option (e.g., 632a-632b). In some embodiments, the first audio track includes a first quantity of audio corresponding to a first detail level option (e.g., standard audio). In some embodiments, the second audio track includes a second quantity of audio corresponding to a second detail level option (e.g., descriptive audio). In some implementations, the first amount of audio (e.g., the amount of audio during a certain time frame (e.g., per second, per minute, 30 minutes, etc.)) is different from (e.g., greater than (or less than)) the second amount of audio. In some implementations, the first level of detail option is different from (e.g., visually different from) the second level of detail option. Transmitting the second audio of the second track, which includes audio content with the amount corresponding to the level of detail option, after receiving a request to transmit the second audio corresponding to the second audio track among multiple audio tracks to the second device, including the detection of the level of detail selection via one or more input devices, improves the efficiency of the computer system because the computer system only needs to cause playback of one instance or media item, rather than multiple instances of the media item, to transmit the audio including the amount of audio content corresponding to the level of detail option. Improving the efficiency of the computer system reduces power consumption and extends the battery life of the computer system by enabling users to use the computer system more safely and efficiently. Transmitting the second audio of the second track, which includes audio content with the amount corresponding to the level of detail option, after receiving a request to transmit the second audio corresponding to the second audio track among multiple audio tracks to the second device, including the detection of the level of detail selection via one or more input devices, provides enhanced control over transmitting specific audio to a specific device. Providing enhanced control over the transmission of specific audio to specific devices makes user-device interfaces more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with devices), which in turn reduces power consumption and extends device battery life by enabling users to use devices more quickly and efficiently.

[0235] In some embodiments, as part of receiving a request to transmit second audio corresponding to a second audio track among a plurality of audio tracks to a second device, the computer system receives (e.g., detects) a selection of an external audio track option (e.g., 634a-634b) via one or more input devices (e.g., a selectable user interface object (e.g., corresponding to a selection of an external audio track option (e.g., a narration option))). In some embodiments, the language option is displayed by the computer system (or caused to be displayed on a monitor connected to the computer system). In some embodiments, the second audio includes audio content not included in the plurality of audio tracks of the media project (e.g., represented by 670) (e.g., generated separately from the media project (or including a third audio track generated separately from the media project)) (e.g., narration content corresponding to the user interface (e.g., audio content with audio instructions on how to interact with the user interface)). In some embodiments, the first audio track does not include audio content or narration content not included in the plurality of audio tracks of the media project. Upon receiving a request to transmit a second audio corresponding to a second audio track among multiple audio tracks to a second device, transmitting the second audio, which includes audio content not included in the multiple audio tracks of a media item, including receiving the selection of external audio track options via one or more input devices, improves the efficiency of the computer system because the computer system only needs to cause playback of one instance or media item, rather than multiple instances of the media item, to transmit the audio, including the audio content corresponding to the external audio track options. This improved efficiency reduces power consumption and extends the battery life of the computer system by enabling users to use it more safely and efficiently. Transmitting the second audio, which includes audio content not included in the multiple audio tracks of a media item, after receiving a request to transmit the second audio corresponding to a second audio track among multiple audio tracks to a second device, including receiving the selection of external audio track options via one or more input devices, provides enhanced control over transmitting specific audio to a specific device. Providing enhanced control over transmitting specific audio to a specific device makes the user-device interface more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0236] In some embodiments, when transmitting first audio (e.g., 610a-610k) corresponding to a first audio track among a plurality of audio tracks to a first device (e.g., 606a), the computer system receives (714) a request to connect (e.g., via wireless and / or Bluetooth connection) (e.g., directly or indirectly) to a third device (e.g., 606c) (e.g., a set of headphones, a set of smart speakers) (e.g., a third device different from the first and second devices). In some embodiments, in response to receiving the request to connect to the third device (e.g., 606c), the computer system transmits (716) a third audio (e.g., 616k) including a third audio track among a plurality of audio tracks (e.g., the third audio track is different from the first and second audio tracks). In some implementations, the third audio track is selected based on the historical usage of a third device (e.g., 606c) (e.g., the historical usage of the third device when it is connected to a computer system) (e.g., the historical usage associated with a profile also associated with the third device) (e.g., the historical usage of the third device when the computer system is connected to the third device and when the computer system (e.g., the same computer system or a different computer system) is playing back a media project, the historical usage of the third device when the computer system is connected to the third device and when the computer system is playing back a media project of a specific type (e.g., video media, sports media), the historical usage of the third device when the computer system is connected to the third device and when the computer system is playing back media at a specific time (e.g., 3:00 PM, 4:00 PM) or during a specific time frame (e.g., after 8:00 PM), and / or the historical usage of the third device when the computer system is connected to the third device and when the computer system is playing back a specific media project) (e.g., automatically selected without user input). In some implementations, the third audio track is different from the first and second audio tracks. Transmitting a third audio track, including the third audio track from multiple tracks, based on the historical usage selection of the third device (e.g., automatic selection without user intervention), reduces the amount of user input required to reset the device to the configuration each device was in during a previous session. Reducing the amount of user input enhances device operability and makes the user-device interface more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0237] In some implementations, the computer system detects the end of the current session of media playback when transmitting a first audio track (e.g., 610a-610k) corresponding to a first audio track among a plurality of audio tracks to a first device (e.g., 606a) and a second audio track (e.g., 614f-614h) corresponding to a second audio track among a plurality of audio tracks to a second device (e.g., 606b). In some implementations, the computer system detecting the end of the current session of media playback includes: the computer system detecting a request to stop media playback; the computer system detecting a request to shut down the computer system; the computer system detecting a request to turn the computer system back on after it has been shut down; and the computer system detecting a request that playback of a media item has been paused or stopped for a predetermined period of time. In some implementations, after the computer system detects (and in some implementations, in response to the computer system detecting) the end of the current session of media playback, the computer system stops transmitting first audio (e.g., 610a-610k) to a first device (e.g., 606a) (e.g., or stops transmitting all audio to the first device after detecting the end of the current session of media playback), and the computer system stops transmitting second audio (e.g., 614f-614h) to a second device (e.g., 606b) (e.g., stops transmitting all audio to the second device after detecting the end of the current session of media playback). In some implementations, after the computer system stops transmitting the first audio (e.g., 610a-610k) to the first device (e.g., 606a) and stops transmitting the second audio (e.g., 614f-614h) to the second device (e.g., 606b), the computer system detects the start of a new session of media playback. In some implementations, detecting the start of a new session for media playback includes detecting a request to initiate playback of a media item and a request to resume playback of a media item. In some implementations, in response to the computer system detecting the start of a new session for media playback, the computer system transmits to a first device (e.g., 606a) a fourth audio that includes one or more characteristics of the first audio (e.g., 610a-610k) (e.g., language type (e.g., the audio is English, Spanish, or German), level of detail (e.g., standard audio, descriptive audio), and whether or not external audio unrelated to the media (e.g., narration audio) (e.g., one or more audio configurations).In some embodiments, in response to the computer system detecting the start of a new media playback session, the computer system transmits a fifth audio to a second device (e.g., 606b) that includes one or more characteristics of the second audio (e.g., 614f-614h) (e.g., language type (e.g., audio is English, Spanish, German), level of detail (e.g., standard audio, descriptive audio), and includes or excludes external audio unrelated to the media (e.g., narration audio) (e.g., one or more audio configurations). In some embodiments, one or more characteristics of the first audio (e.g., 610a-610k) differ from one or more characteristics of the second audio (e.g., 614f-614h). In some embodiments, the new media playback session includes the computer system playing a second media item that is different from the media item. In some embodiments, in response to the computer system detecting the start of a new media playback session, the computer system plays the second media item (or media item). In some embodiments, the computer system transmits a fourth audio to the first device and a fifth audio to the second device while playing the second media item (or the first media item). In some implementations, the computer system transmits a fourth audio signal to the first device but not a fifth audio signal. In some implementations, the computer system transmits a fifth audio signal to the second device but not the fourth audio signal. In some implementations, when a new media playback session includes a media item being played, the fourth audio signal is the same as the first audio signal, and the fifth audio signal is the same as the second audio signal. Transmitting a fourth audio signal, including one or more characteristics of the first audio signal, to the first device and a fifth audio signal, including one or more characteristics of the second audio signal, to the second device in response to detecting the start of a new media playback session reduces the amount of user input required for a user to reset the device to the configuration each device was in during a previous session. Reducing the amount of user input required to perform operations enhances device operability and makes the user-device interface more efficient (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0238] In some embodiments, when causing playback of media corresponding to multiple audio tracks, the computer system causes (e.g., sends instructions to display one or more input devices) to display a user interface via one or more input devices (e.g., 604) (e.g., on which, for example, a display on an external remote control). This user interface includes multiple selectable options (e.g., one or more language options, one or more detail level options, one or more external audio track options) for configuring the audio to be played back on a second device (e.g., 608b, 608c). In some embodiments, a request to transmit a second audio (e.g., 614f-614h) corresponding to a second audio track among the multiple audio tracks to the second device (e.g., 608b, 608c) includes receiving a selection (e.g., one or more taps on one or more input devices) of one or more selectable options among the multiple selectable options (e.g., 630a-630c, 632a-632b, 634a-634b, 636a-636b). This provides the user with additional control options by displaying a user interface via one or more input devices, including multiple selectable options for configuring the audio to be played back on a second device, without confusing the user interface that includes a representation of the media being played back. Providing additional control options without confusing the user interface makes the user-system interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0239] For purposes of explanation, the foregoing description has been given by reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible based on the teachings above. These embodiments were chosen and described in order to best explain the principles of these techniques and their practical applications. Others skilled in the art will thus be able to best utilize these techniques and the various embodiments with various modifications suitable for the particular intended use.

[0240] While this disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of this disclosure and examples as defined by the claims.

[0241] As described above, one aspect of the present invention involves collecting and using data from various sources to improve the audio delivery of media content (e.g., or any other content that the user may be interested in) to the user. This disclosure contemplates that, in some instances, this collected data may include personal information data that uniquely identifies or can be used to contact or locate specific individuals. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0242] This disclosure recognizes that the use of such personal information data in the techniques of this invention can benefit users. For example, personal information data can be used to transmit audio of media content in a specific format that is of particular interest to the user. Therefore, the use of such personal information data enables users to exercise planned control over the media content. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into a user's overall health status or as positive feedback for individuals using technology to pursue health goals.

[0243] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as data collection and / or use change. Personal information from users should be collected for the entity's lawful and reasonable purposes and not shared or sold outside of these lawful uses. Furthermore, such collection / sharing should be conducted only after obtaining informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and processes. Furthermore, such entities may be subject to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Additionally, policies and practices should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including specific considerations regarding jurisdiction. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0244] Regardless of the foregoing, this disclosure also contemplates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure contemplates providing hardware and / or software components to prevent or block access to such personal information data. For example, with respect to media content providers, the technology of this invention can be configured to allow users to opt-in or opt-out at any time during or after registration for the service to participate in the collection of personal information data. As another example, users can choose not to provide audio configuration data and / or experience data for the targeted delivery of media content (including the audio content of the media). Furthermore, users can choose to limit the duration for which audio configuration data and / or experience data are maintained for the targeted delivery of media content or to completely prohibit the development of media content profiles. In addition to providing opt-in and opt-out options, this disclosure also envisions providing notifications related to access to or use of personal information. For example, users can be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0245] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0246] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be rendered inoperable due to the absence of all or part of such personal information data. For example, preferences can be inferred from non-personal information data or an absolute minimum of personal information (such as content requested by a user's associated device, other non-personal information or publicly available information available to media content providers) to select and deliver audio content for a specific media project to a user.

Claims

1. A method for causing a media playback project, comprising: At the computer system that communicates with the display generation component and one or more input devices: When causing playback of a media item corresponding to multiple audio tracks and simultaneously displaying a user interface on the display generation component, wherein the user interface includes a representation of a first device, a representation of a second device, multiple selectable options for configuring audio to be played back on the first device, and multiple selectable options for configuring audio to be played back on the second device, and transmitting a first audio corresponding to a first audio track among the multiple audio tracks to the first device and the second device; When transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the first device and the second device, and simultaneously displaying the user interface on the display generation component, wherein the user interface includes the representation of the first device, the representation of the second device, the plurality of selectable options for configuring audio to be played back on the first device, and the plurality of selectable options for configuring audio to be played back on the second device, and receiving a request via the one or more input devices to transmit the second audio corresponding to the second audio track among the plurality of audio tracks that is different from the first audio track; as well as In response to receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks: The request to determine transmission includes detecting the selection of one of the plurality of selectable options for configuring the audio to be played back on the first device: A first indicator is displayed on the user interface, wherein the first indicator indicates that the second audio corresponding to the second audio track is configured to be transmitted to the first device; Transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device; as well as Continue transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device; When transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device and transmitting the second audio corresponding to the second audio track among the plurality of audio tracks to the first device, the end of the current session of media playback is detected; After detecting the end of the current session of media playback, stop transmitting the first audio to the second device and stop transmitting the second audio to the first device; After stopping the transmission of the first audio to the second device and stopping the transmission of the second audio to the first device, the start of a new session for media playback is detected; as well as In response to the detection of media playback and the start of the new session: Transmit a fourth audio to the second device, including one or more characteristics of the first audio, wherein the one or more characteristics include language type, level of detail, or whether external audio not associated with the media item is included; as well as A fifth audio signal, comprising one or more characteristics of the second audio signal, is transmitted to the first device, wherein the one or more characteristics of the first audio signal are different from the one or more characteristics of the second audio signal.

2. The method according to claim 1, wherein: The computer system communicates with the first device via a first connection; and The computer system communicates with the second device via a second connection different from the first connection.

3. The method according to any one of claims 1 to 2, wherein: The first device is a device of the first type; and The second device is a second type of electronic device that is different from the first type of device.

4. The method according to any one of claims 1 to 2, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of a language option via the one or more input devices; and The second audio track includes audio content corresponding to the language of the language option.

5. The method according to any one of claims 1 to 2, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of a detail level option via the one or more input devices; and The second audio track has an amount of audio content corresponding to the level of detail option.

6. The method according to any one of claims 1 to 2, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of an external audio track option via the one or more input devices; and The second audio includes audio content not included in the plurality of audio tracks of the media project.

7. The method according to any one of claims 1 to 2, further comprising: While transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device, a request to connect to the third device is received. as well as In response to receiving the request to connect to the third device, a third audio track, comprising a third audio track from the plurality of audio tracks, is transmitted, wherein the third audio track is selected based on the historical usage of the third device.

8. The method according to any one of claims 1 to 2, further comprising: When causing playback of the media corresponding to the plurality of audio tracks, a user interface is displayed via the one or more input devices, the user interface including a plurality of selectable options for configuring audio to be played back on the first device, wherein the request to transmit second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: detecting selection of one or more of the plurality of selectable options.

9. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system, wherein the computer system is configured to communicate with a display generation component and one or more input devices, the one or more programs including instructions for performing the following operations: When causing playback of a media item corresponding to multiple audio tracks and simultaneously displaying a user interface on the display generation component, wherein the user interface includes a representation of a first device, a representation of a second device, multiple selectable options for configuring audio to be played back on the first device, and multiple selectable options for configuring audio to be played back on the second device, and transmitting a first audio corresponding to a first audio track among the multiple audio tracks to the first device and the second device; When transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the first device and the second device, and simultaneously displaying the user interface on the display generation component, wherein the user interface includes the representation of the first device, the representation of the second device, the plurality of selectable options for configuring audio to be played back on the first device, and the plurality of selectable options for configuring audio to be played back on the second device, and receiving a request via the one or more input devices to transmit the second audio corresponding to the second audio track among the plurality of audio tracks that is different from the first audio track; as well as In response to receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks: The request to determine transmission includes detecting the selection of one of the plurality of selectable options for configuring the audio to be played back on the first device: A first indicator is displayed on the user interface, wherein the first indicator indicates that the second audio corresponding to the second audio track is configured to be transmitted to the first device; Transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device; as well as Continue transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device; When transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device and transmitting the second audio corresponding to the second audio track among the plurality of audio tracks to the first device, the end of the current session of media playback is detected; After detecting the end of the current session of media playback, stop transmitting the first audio to the second device and stop transmitting the second audio to the first device; After stopping the transmission of the first audio to the second device and stopping the transmission of the second audio to the first device, the start of a new session for media playback is detected; as well as In response to the detection of media playback and the start of the new session: Transmit a fourth audio to the second device, including one or more characteristics of the first audio, wherein the one or more characteristics include language type, level of detail, or whether external audio not associated with the media item is included; as well as A fifth audio signal, comprising one or more characteristics of the second audio signal, is transmitted to the first device, wherein the one or more characteristics of the first audio signal are different from the one or more characteristics of the second audio signal.

10. The computer-readable storage medium according to claim 9, wherein: The computer system communicates with the first device via a first connection; and The computer system communicates with the second device via a second connection different from the first connection.

11. The computer-readable storage medium according to any one of claims 9 to 10, wherein: The first device is a device of the first type; and The second device is a second type of electronic device that is different from the first type of device.

12. The computer-readable storage medium according to any one of claims 9 to 10, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of a language option via the one or more input devices; and The second audio track includes audio content corresponding to the language of the language option.

13. The computer-readable storage medium according to any one of claims 9 to 10, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of a detail level option via the one or more input devices; and The second audio track has an amount of audio content corresponding to the level of detail option.

14. The computer-readable storage medium according to any one of claims 9 to 10, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of an external audio track option via the one or more input devices; and The second audio includes audio content not included in the plurality of audio tracks of the media project.

15. The computer-readable storage medium according to any one of claims 9 to 10, wherein the one or more programs further include instructions for: While transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device, a request to connect to the third device is received; and In response to receiving the request to connect to the third device, a third audio track, comprising a third audio track from the plurality of audio tracks, is transmitted, wherein the third audio track is selected based on the historical usage of the third device.

16. The computer-readable storage medium according to any one of claims 9 to 10, wherein the one or more programs further include instructions for: When causing playback of the media corresponding to the plurality of audio tracks, a user interface is displayed via the one or more input devices, the user interface including a plurality of selectable options for configuring audio to be played back on the first device, wherein the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: Detect the selection of one or more of the plurality of selectable options.

17. A computer system for causing playback of media projects, comprising: One or more processors, wherein the computer system is configured to communicate with a display generation component and one or more input devices; and The memory stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: When causing playback of a media item corresponding to multiple audio tracks and simultaneously displaying a user interface on the display generation component, wherein the user interface includes a representation of a first device, a representation of a second device, multiple selectable options for configuring audio to be played back on the first device, and multiple selectable options for configuring audio to be played back on the second device, and transmitting a first audio corresponding to a first audio track among the multiple audio tracks to the first device and the second device; When transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the first device and the second device, and simultaneously displaying the user interface on the display generation component, wherein the user interface includes the representation of the first device, the representation of the second device, the plurality of selectable options for configuring audio to be played back on the first device, and the plurality of selectable options for configuring audio to be played back on the second device, and receiving a request via the one or more input devices to transmit the second audio corresponding to the second audio track among the plurality of audio tracks that is different from the first audio track; as well as In response to receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks: The request to determine transmission includes detecting the selection of one of the plurality of selectable options for configuring the audio to be played back on the first device: A first indicator is displayed on the user interface, wherein the first indicator indicates that the second audio corresponding to the second audio track is configured to be transmitted to the first device; Transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device; as well as Continue transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device; When transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device and transmitting the second audio corresponding to the second audio track among the plurality of audio tracks to the first device, the end of the current session of media playback is detected; After detecting the end of the current session of media playback, stop transmitting the first audio to the second device and stop transmitting the second audio to the first device; After stopping the transmission of the first audio to the second device and stopping the transmission of the second audio to the first device, the start of a new session for media playback is detected; as well as In response to the detection of media playback and the start of the new session: Transmit a fourth audio to the second device, including one or more characteristics of the first audio, wherein the one or more characteristics include language type, level of detail, or whether external audio not associated with the media item is included; as well as A fifth audio signal, comprising one or more characteristics of the second audio signal, is transmitted to the first device, wherein the one or more characteristics of the first audio signal are different from the one or more characteristics of the second audio signal.

18. The computer system according to claim 17, wherein: The computer system communicates with the first device via a first connection; and The computer system communicates with the second device via a second connection different from the first connection.

19. The computer system according to any one of claims 17 to 18, wherein: The first device is a device of the first type; and The second device is a second type of electronic device that is different from the first type of device.

20. The computer system according to any one of claims 17 to 18, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of a language option via the one or more input devices; and The second audio track includes audio content corresponding to the language of the language option.

21. The computer system according to any one of claims 17 to 18, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of a detail level option via the one or more input devices; and The second audio track has an amount of audio content corresponding to the level of detail option.

22. The computer system according to any one of claims 17 to 18, wherein: Receiving the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: receiving a selection of an external audio track option via the one or more input devices; and The second audio includes audio content not included in the plurality of audio tracks of the media project.

23. The computer system according to any one of claims 17 to 18, wherein the one or more programs further include instructions for performing the following operations: While transmitting the first audio corresponding to the first audio track among the plurality of audio tracks to the second device, a request to connect to the third device is received; and In response to receiving the request to connect to the third device, a third audio track, comprising a third audio track from the plurality of audio tracks, is transmitted, wherein the third audio track is selected based on the historical usage of the third device.

24. The computer system according to any one of claims 17 to 18, wherein the one or more programs further include instructions for performing the following operations: When causing playback of the media corresponding to the plurality of audio tracks, a user interface is displayed via the one or more input devices, the user interface including a plurality of selectable options for configuring audio to be played back on the first device, wherein the request to transmit the second audio corresponding to the second audio track among the plurality of audio tracks to the first device includes: Detect the selection of one or more of the plurality of selectable options.

25. A computer system, comprising: Apparatus for performing the method according to any one of claims 1 to 2.

26. A computer program product comprising: One or more programs configured to be executed by one or more processors of a computer system, the one or more programs including instructions for performing the method according to any one of claims 1 to 2.

Images