Multi-function device control of another electronic device

Abstract

Embodiments of the present disclosure relate to multifunction device control of another electronic device. The present disclosure provides one or more input devices, and some embodiments described in the present disclosure relate to one or more input devices that simulate dedicated remote control functions for navigating and playing content items available on other electronic devices, and one or more operations related to the above that are optionally performed by the input device and other electronic devices. Some embodiments described in the present disclosure relate to one or more multifunction devices via which keyboard input is provided to an electronic device, and one or more operations related to the above that are optionally performed by the multifunction device and the electronic device. Some embodiments described in the present disclosure relate to one or more multifunction devices via which control and / or navigation input is provided to an electronic device, and one or more operations related to the above that are optionally performed by the multifunction device and the electronic device.

Description

[0001] This application is a divisional application of the invention patent application with an international application date of March 27, 2017, which entered the Chinese national phase on August 31, 2018, with Chinese national application number 201780014752.5 and invention name “Multi-function device control of another electronic device”. Technical Field

[0002] The present invention generally relates to using multifunction devices to control electronic devices, and user interaction with such devices. Background Art

[0003] In recent years, users' interactions with electronic devices have increased significantly. These devices can be devices such as computers, tablets, televisions, multimedia devices, mobile devices, etc.

[0004] In some cases, such devices have access to content (e.g., music, movies, etc.), and user interaction with such devices requires the use of a multifunction device to provide input to the device. Enhancing these interactions can improve the user's experience with the device and reduce user interaction time, which is particularly important when the input device is battery-powered. Summary of the Invention

[0005] Some embodiments described in the present disclosure relate to one or more input devices that simulate dedicated remote control functions for navigating and playing content items available on other electronic devices, and one or more operations related to the above that are optionally performed by the input devices and other electronic devices. Some embodiments described in the present disclosure relate to one or more multifunction devices via which keyboard input is provided to electronic devices, and one or more operations related to the above that are optionally performed by the multifunction devices and electronic devices. Some embodiments described in the present disclosure relate to one or more multifunction devices via which control and / or navigation input is provided to electronic devices, and one or more operations related to the above that are optionally performed by the multifunction devices and electronic devices. A comprehensive description of the embodiments is provided in the drawings and detailed description, and it should be understood that the invention content provided above does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

[0007] Figure 1A is a block diagram illustrating a multifunction device with a touch-sensitive display according to some embodiments of the present disclosure.

[0008] Figure 1Bis a block diagram illustrating exemplary components for event processing according to some embodiments of the present disclosure.

[0009] Figure 2 A multifunction device with a touch screen according to some embodiments of the present disclosure is shown.

[0010] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments of the present disclosure.

[0011] Figure 4 An exemplary user interface for a multifunction device having a touch-sensitive surface that is separate from the display is shown in accordance with some embodiments of the present disclosure.

[0012] Figures 5A to 5B A block diagram illustrating an exemplary architecture for a device according to some embodiments of the present disclosure is shown.

[0013] Figures 6A to 6Q An exemplary manner of simulating button click functionality on a device having a touch-sensitive surface but without button click functionality is shown according to some embodiments of the present disclosure.

[0014] 7A to 7E is a flowchart illustrating a method of simulating a button click function on a device having a touch-sensitive surface but not having a button click function according to some embodiments of the present disclosure.

[0015] Figures 8A to 8R Exemplary ways are shown in which an electronic device reduces unintentional recognition of click or selection inputs when a user is providing moving touch inputs on a touch-sensitive surface according to some embodiments of the present disclosure.

[0016] Figures 9A to 9G is a flowchart illustrating a method for reducing unintentional recognition of click or selection inputs when a user is providing moving touch input on a touch-sensitive surface according to some embodiments of the present disclosure.

[0017] Figures 10A to 10N Illustrated are exemplary ways in which a user may interact with an electronic device using a multifunction device that displays various user interfaces for controlling and interacting with the electronic device according to some embodiments of the present disclosure.

[0018] Figures 11A to 11J is a flow chart illustrating a method of interacting with an electronic device using a multifunction device that displays various user interfaces for controlling and interacting with the electronic device according to some embodiments of the present disclosure.

[0019] Figures 12A to 12RRIllustrated are exemplary ways in which a need for text input to an electronic device is indicated on a multifunction device according to some embodiments of the present disclosure.

[0020] Figures 13A to 13K is a flow chart illustrating a method of indicating a need for text input to an electronic device on a multifunction device according to some embodiments of the present disclosure.

[0021] Figures 14A to 14GG An exemplary manner is shown in which a multifunction device selects a primary touch navigation area on its touch-sensitive surface that behaves similarly to the touch-sensitive surface of a dedicated remote control according to some embodiments of the present disclosure.

[0022] Figures 15A to 15H is a flowchart illustrating a method of selecting a primary touch navigation area on a touch-sensitive surface of an electronic device that behaves similarly to the touch-sensitive surface of a dedicated remote control according to some embodiments of the present disclosure.

[0023] 16A to 16T An exemplary manner is shown in which a multifunction device selects a primary touch navigation area on its touch-sensitive surface based on movement of contact when contact is first detected by the multifunction device (e.g., when contact lands on the touch-sensitive surface) according to some embodiments of the present disclosure.

[0024] Figures 17A to 17G is a flowchart illustrating a method for selecting a primary touch navigation area on a touch-sensitive surface of an electronic device based on movement of a contact when the contact is first detected by the electronic device (e.g., when the contact lands on the touch-sensitive surface) according to some embodiments of the present disclosure.

[0025] Figures 18A to 18II An exemplary manner in which a multifunction device arranges a control panel area and a touch navigation area in a user interface of the multifunction device according to some embodiments of the present disclosure is shown.

[0026] Figures 19A to 19H is a flowchart illustrating a method of arranging a control panel area and a touch navigation area in a user interface of an electronic device according to some embodiments of the present disclosure.

[0027] Figures 20 to 26 is a functional block diagram of an electronic device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0028] In the following description of the embodiments, reference will be made to the accompanying drawings, which form a part of the following description and show, by way of example, specific embodiments that may be implemented. It should be understood that other embodiments may be optionally used and structural changes may be optionally made without departing from the scope of the disclosed embodiments. In addition, although the following description uses the terms "first," "second," etc. to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, a first touch may be named a second touch and similarly, a second touch may be named a first touch without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0029] The terms used in the description of the various described embodiments herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various described embodiments and in the appended claims, the singular forms "a", "an" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the items listed in association. It will also be understood that the terms "comprises" and / or "comprising" when used in this manual specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their groupings.

[0030] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that," or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining," or "in response to determining," or "upon detecting [stated condition or event]," or "in response to detecting [stated condition or event]," depending on the context.

[0031] Exemplary devices

[0032] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod equipment, and Device. Other portable electronic devices are optionally used, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer or television with a touch-sensitive surface (e.g., touch screen displays and / or touchpads). In some embodiments, the device does not have a touch screen display and / or touchpad, but is capable of outputting display information for display on a separate display device (such as a user interface of the present disclosure), and is capable of receiving input information from a separate input device with one or more input mechanisms (such as one or more buttons, touch screen displays and / or touchpads). In some embodiments, the device has a display, but is capable of receiving input information from a separate input device with one or more input mechanisms (such as one or more buttons, touch screen displays and / or touchpads).

[0033] In the following discussion, an electronic device is described that includes a display and a touch-sensitive surface. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed among two or more devices. Therefore, as used in this disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device for display on a separate display device (touch-sensitive or non-touch-sensitive). Similarly, as used in this disclosure, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device) is optionally used to describe input received on a separate input device from which the electronic device receives input information.

[0034] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a rendering application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application.

[0035] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0036] Attention is now directed to embodiments of a portable or non-portable device having a touch-sensitive display, although the device need not include a touch-sensitive display or a general display, as described above. Figure 1A 1 is a block diagram illustrating a portable or non-portable multifunction device 100 with a touch-sensitive display 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a touch-sensitive display system. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contacts on the device 100 (e.g., a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0037] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, the force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functionality that would not otherwise be accessible to the user on a smaller device with limited real estate, which is used to display an enable indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0038] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "up click," "down click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate the described sensory perception for a typical (or average) user.

[0039] It should be understood that device 100 is only one example of a portable or non-portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the figure are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits. Figure 1A The various components shown in are, optionally, implemented on two or more devices; for example, a display and audio circuitry on a display device, a touch-sensitive surface on an input device, and the remaining components on device 100. In such embodiments, device 100 optionally communicates with the display device and / or input device to facilitate operation of the system as described in this disclosure, and the various display and / or input-related components described herein remain in device 100 or are optionally included in the display and / or input device as appropriate.

[0040] Memory 102 optionally includes high-speed random access memory and optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0041] Peripherals interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0042] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0043] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with communication networks and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs). RF circuitry 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via a short-range communication radio. The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11d), IEEE 802.11e, IEEE 802.11f, IEEE 802.11g, IEEE 802.11g), IEEE 802.11f, IEEE 802.11g ... 11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0044] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0045] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, and the like. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in ).

[0046] A quick press of the push button optionally unlocks the touch screen 112 or optionally begins the process of unlocking the device using gestures on the touch screen, as described in U.S. patent application Ser. No. 11 / 322,549, filed Dec. 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," (U.S. Pat. No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0047] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. As described above, the touch-sensitive operation and display operation of the touch-sensitive display 112 are optionally separated from each other so that the display device is used for display purposes, and the touch-sensitive surface (whether it is a display or not) is used for input detection purposes, and the components and functions described are modified accordingly. However, for the sake of brevity, the following description is provided with reference to the touch-sensitive display. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some visual outputs or all of the visual outputs correspond to user interface objects.

[0048] The touch screen 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from the user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact on the touch screen 112 (and any movement or interruption of that contact) and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0049] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In one exemplary embodiment, projected mutual capacitance sensing technology is used, such as the ATmega 200 from Apple Inc. (Cupertino, California). iPod and Discovery technology.

[0050] In some embodiments, the touch-sensitive display of the touch screen 112 is optionally similar to the multi-touch-sensitive trackpad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman) and / or U.S. Patent Publication 2402 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, whereas a touch-sensitive trackpad does not provide visual output.

[0051] In some embodiments, the touch-sensitive display of the touch screen 112 is as described in the following applications: (1) U.S. patent application No. 11 / 381,313, filed on May 2, 2006, entitled "Multipoint Touch Surface Controller"; (2) U.S. patent application No. 10 / 840,862, filed on May 6, 2004, entitled "Multipoint Touchscreen"; (3) U.S. patent application No. 10 / 903,964, filed on July 30, 2004, entitled "Gestures For Touch Sensitive Input Devices"; (4) U.S. patent application No. 11 / 048,264, filed on January 31, 2005, entitled "Gestures For Touch Sensitive Input Devices"; (5) U.S. patent application No. 11 / 048,264, filed on January 31, 2005, entitled "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices”; (6) U.S. patent application Ser. No. 11 / 228,758, filed Sep. 16, 2005, entitled “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. patent application Ser. No. 11 / 228,700, filed Sep. 16, 2005, entitled “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. patent application Ser. No. 11 / 228,737, filed Sep. 16, 2005, entitled “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. patent application Ser. No. 11 / 367,749, filed Mar. 3, 2006, entitled “Multi-Functional Hand-Held Device”. All of these applications are incorporated herein by reference in their entirety.

[0052] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or appendage, such as a stylus, a finger, or the like, to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the user's desired action.

[0053] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0054] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable or non-portable device.

[0055] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor is shown coupled to the optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, facing away from the touch screen display 112 on the front of the device, enabling the touch screen display to be used as a viewfinder for still and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing the user to optionally capture an image of the user while viewing other video conference participants on the touch screen display for video conferencing. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), allowing a single optical sensor 164 to be used with the touch screen display for both video conferencing and still and / or video image acquisition.

[0056] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100 opposite touch screen display 112 located on the front of device 100.

[0057] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent applications: 11 / 241,839, entitled “Proximity Detector In Handheld Device”; 11 / 240,788, entitled “Proximity Detector In Handheld Device”; 11 / 620,702, entitled “Using Ambient Light Sensor To Augment Proximity Sensor Output”; 11 / 586,862, entitled “Automated Response To And Sensing Of User Activity In Portable Devices”; and 11 / 638,251, entitled “Methods And Systems For Automatic Configuration Of Peripherals,” which are hereby incorporated by reference in their entireties. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0058] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1A A tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile outputs on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile outputs on the device 100 that can be felt by the user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0059] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripherals interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally implements as described in the following U.S. Patent Publication: U.S. Patent Publication 20050230059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 24060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated by reference in their entireties. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0060] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state indicating which applications, if any, are currently active; a display state indicating what applications, views, or other information occupy various areas of the touch screen display 112; a sensor state including information obtained from the device's various sensors and input control devices 116; and position information regarding the device's position and / or posture.

[0061] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware components and software components.

[0062] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are suitable for coupling directly to other devices or indirectly through a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used on iPod (trademark of Apple Inc.) devices.

[0063] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a trackpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger up event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a sequence of contact data. These operations are optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touchpad.

[0064] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0065] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0066] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0067] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program or the like, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0068] Haptic feedback module 133 includes various software components for generating instructions used by one or more tactile output generators 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0069] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0070] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as a weather widget, local pages widget, and map / navigation widget).

[0071] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0072] Contacts module 137 (sometimes called address book or contact list);

[0073] Telephone module 138;

[0074] Video conferencing module 139;

[0075] Email client module 140;

[0076] Instant messaging (IM) module 141;

[0077] Fitness support module 142;

[0078] A camera module 143 for still and / or video images;

[0079] Image management module 144;

[0080] Video player module;

[0081] Music player module;

[0082] Browser module 147;

[0083] Calendar module 148;

[0084] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, and user-created widgets 149-6;

[0085] A widget creator module 150 for forming user-created widgets 149-6;

[0086] Search module 151;

[0087] Video and music player module 152, which combines the video player module and the music player module;

[0088] Notepad module 153;

[0089] Map module 154; and / or

[0090] Online video module 155.

[0091] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0092] In combination with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contacts module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating telephone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0093] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contacts module 137, modify an entered phone number, dial the corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0094] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the touch / motion module 130, the graphics module 132, the text input module 134, the contacts module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0095] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0096] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messaging, or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages transmitted using SMS or MMS) and Internet-based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

[0097] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing, and transmitting fitness data.

[0098] In combination with the touch screen 112, the display controller 156, one or more optical sensors 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, and the image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0099] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0100] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or portions thereof and attachments and other files linked to web pages) in accordance with user instructions.

[0101] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0102] In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., the Yahoo! desktop widget).

[0103] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop widget creator module 150 is optionally used by a user to create a desktop widget (e.g., to transfer a user-specified portion of a web page into a desktop widget).

[0104] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0105] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0106] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0107] In combination with the RF circuitry 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0108] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send links to specific online videos. Additional descriptions of online video applications can be found in U.S. provisional patent application 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. patent application 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0109] Each of the above modules and applications corresponds to an executable instruction set for performing one or more of the above functions and methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as separate software programs, processes, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0110] In some embodiments, device 100 is a device that performs operations on a predefined set of functions on it solely through a touch screen and / or a touch pad. By using the touch screen and / or the touch pad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0111] A predefined set of functions that are exclusively performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, the trackpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0112] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0113] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0114] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by the application 136-1 or ready to be displayed by the application, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user.

[0115] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (via audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0116] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0117] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0118] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0119] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0120] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0121] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0122] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver 182.

[0123] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0124] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0125] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0126] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0127] The event comparator 184 compares the event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, the sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, a drag includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0128] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler that is associated with the sub-event and the object that triggered the hit test.

[0129] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0130] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0131] In some embodiments, the corresponding event recognizers 180 include metadata 183 having configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0132] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag receives the flag and performs a predefined process.

[0133] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0134] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0135] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module or application view 191 of the corresponding application 136-1. In other embodiments, they are included in two or more software modules.

[0136] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a trackpad, such as taps, drags, scrolls, etc.; stylus input; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0137] Figure 2 A portable or non-portable multifunction device 100 with a touch screen 112 according to some embodiments is shown. As described above, the multifunction device 100 is described as having various illustrated structures (such as the touch screen 112, the speaker 111, the accelerometer 168, the microphone 113, etc.); however, it should be understood that these structures may optionally reside on separate devices. For example, display-related structures (e.g., a display, a speaker, etc.) and / or functions may optionally reside on a separate display device, input-related structures (e.g., a touch-sensitive surface, a microphone, an accelerometer, etc.) and / or functions may optionally reside on a separate input device, and the remaining structures and / or functions may optionally reside on the multifunction device 100.

[0138] The touch screen 112 optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and in other embodiments described below, a user can select one or more of the graphics by, for example, making a gesture on the graphics using one or more fingers 202 (not drawn to scale in the accompanying drawings) or using one or more styluses 203 (not drawn to scale in the accompanying drawings). In some embodiments, selection of one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some embodiments or in some cases, accidentally making contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0139] The device 100 optionally also includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0140] In one embodiment, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on and off and locking the device, volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and an external docking / charging port 124. The push button 206 is optionally used to turn the device on and off by pressing and holding the button for a predefined time interval; to lock the device by pressing and releasing the button before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions via the microphone 113. The device 100 also optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0141] Figure 33 is a block diagram of an exemplary multi-function device with a display and a touch-sensitive surface according to some embodiments. As described above, the device 300 does not have to include a display and a touch-sensitive surface. Instead, in some embodiments, it optionally communicates with displays and touch-sensitive surfaces on other devices. In addition, the device 300 does not have to be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device (such as a television or set-top box), a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home or industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes called chipsets) that interconnect system components and control communication between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and touchpad 355, a tactile output generator 357 for generating tactile output on device 300 (e.g., similar to the above referenced embodiments). Figure 1A The tactile output generator 167 described above), sensor 359 (e.g., optical sensor, acceleration sensor, proximity sensor, touch sensor and / or contact intensity sensor (similar to the above reference Figure 1A The contact force sensor 165 is described. The memory 370 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores programs, modules, and data structures, or a subset thereof, similar to those of the portable or non-portable multifunction device 100 ( Figure 1A ) stored in the memory 102 of the portable or non-portable multifunction device 100. In addition, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable or non-portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable or non-portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0142] Figure 3 Each of the above-mentioned elements is optionally stored in one or more of the memory devices mentioned above. Each module in the above-mentioned identified module corresponds to a group of instructions for performing the functions described above. The above-mentioned identified module or program (that is, instruction set) need not be implemented as independent software program, process or module, so the various subgroups of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subgroup of the above-mentioned modules and data structure. In addition, memory 370 optionally stores other modules and data structure not described above.

[0143] Figure 4 A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, touch screen display 112) that is separate from a display 450 (eg, touch screen display 112). Figure 3 tablet or touchpad 355) of the device (e.g., Figure 3 Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 357) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 359 for generating tactile output for a user of device 300.

[0144] Although some of the examples below will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4 In some embodiments, the touch-sensitive surface (e.g., Figure 4 451) with a principal axis (e.g., Figure 4 452 in ), which corresponds to the main axis on the display (e.g., 450) (e.g., Figure 4 According to these embodiments, the device detects contact with the touch-sensitive surface 451 (e.g., Figure 4 460 and 462 in FIG), the location of the contact corresponds to a corresponding location on the display (e.g., Figure 4 , 460 corresponds to 468 and 462 corresponds to 470). Thus, when a touch-sensitive surface (e.g., Figure 4 451) and the display of the multi-function device ( Figure 4 When the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is separated, the device is used to operate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0145] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting the contact, followed by ceasing to detect the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or that a mouse and finger contact are optionally used simultaneously.

[0146] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor rests over a particular user interface element (e.g., a button, window, slider, or other user interface element), a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4 In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, a particular user interface element is adjusted according to the detected input. In the case that the touch-sensitive surface 451 in the display includes a touch-screen display (e.g., a touch-sensitive surface 451 in the display), a specific user interface element is adjusted according to the detected input. Figure 1AIn some implementations of a touch-sensitive display system 112 in a touch-sensitive display, a contact detected on the touch screen acts as a "focus selector" such that when input (e.g., a press input by a contact) is detected on the touch screen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move focus from one button to another); in these implementations, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) that is controlled by the user to convey the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0147] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0148] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting an increase in the intensity of the contact (or multiple contacts) to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting an increase in the intensity of the corresponding contact to above the press input intensity threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below the press input intensity threshold, and the corresponding operation is performed in response to detecting a subsequent decrease in the intensity of the corresponding contact to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0149] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).

[0150] For ease of explanation, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting any of the following: contact intensity increasing above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing below the press input intensity threshold, and / or contact intensity decreasing below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0151] Figure 5A A block diagram illustrating an exemplary architecture for a device 500 according to some embodiments of the present disclosure is shown. Figure 5A In an embodiment, media content or other content is optionally received by device 500 via network interface 502, which is optionally a wireless connection or a wired connection. One or more processors 504 optionally execute any number of programs stored in memory 506 or storage devices, the programs optionally including instructions for performing one or more of the methods and / or processes described herein (e.g., method 700).

[0152] In some embodiments, the display controller 508 causes the various user interfaces of the present disclosure to be displayed on the display 514. Additionally, input to the device 500 is optionally provided by a remote control 510 via a remote control interface 512, which is optionally a wireless or wired connection. In some embodiments, input to the device 500 is provided by a multifunction device 511 (e.g., a smartphone) on which a remote control application is running that configures the multifunction device to simulate remote control functionality, as will be described in more detail below. In some embodiments, the multifunction device 511 corresponds to Figure 1A and Figure 2 Device 100 and Figure 3 One or more devices in the device 300. It should be understood that Figure 5A The embodiments are not meant to limit the features of the device of the present disclosure, and other components that facilitate other features described in the present disclosure are optionally included. Figure 5A In some embodiments, the device 500 optionally corresponds to Figure 1A and Figure 2 The multifunction device 100 and Figure 3 One or more devices in the device 300; the network interface 502 optionally corresponds to Figure 1A and Figure 2 RF circuit 108, external port 124 and peripheral device interface 118 in the Figure 3 One or more of the network communication interfaces 360 in; the processor 504 optionally corresponds to one or more of the following: Figure 1A The processor 120 and Figure 3 The CPU 310 in the display controller 508 optionally corresponds to one or more of the following: Figure 1A The display controller 156 and Figure 3 The I / O interface 330 in the embodiment; the memory 506 optionally corresponds to one or more of the following: Figure 1A The memory 102 and Figure 3 The memory 370 in the remote control interface 512 optionally corresponds to one or more of the following: Figure 1A The peripheral device interface 118 and the I / O subsystem 106 (and / or its components) in Figure 3 The I / O interface 330 in the remote control 512 optionally corresponds to and / or includes one or more of the following: Figure 1A Speaker 111, touch-sensitive display system 112, microphone 113, optical sensor 164, contact intensity sensor 165, tactile output generator 167, other input control devices 116, accelerometer 168, proximity sensor 166 and I / O subsystem 106, and Figure 3The keyboard / mouse 350, the touchpad 355, the tactile output generator 357 and the contact intensity sensor 359 in Figure 4 and display 514 optionally corresponds to one or more of: Figure 1A and 2 one or more touch-sensitive display systems 112, and Figure 3 Display 340 in.

[0153] Figure 5B 1 shows an exemplary structure of a remote control 510 according to some embodiments of the present disclosure. In some embodiments, the remote control 510 optionally corresponds to Figure 1A and Figure 2 The multifunction device 100 and Figure 3 The remote control 510 optionally includes a touch-sensitive surface 451. In some embodiments, the touch-sensitive surface 451 is edge-to-edge (e.g., it extends to the edge of the remote control 510 so that there is little or no surface of the remote control 510 between the touch-sensitive surface 451 and one or more edges of the remote control 510, such as Figure 5BAs previously described in this disclosure, touch-sensitive surface 451 is optionally capable of sensing contact and contact intensity (e.g., clicking on touch-sensitive surface 451). In addition, touch-sensitive surface 451 optionally includes a mechanical actuator for providing physical button click functionality (e.g., touch-sensitive surface 451 is "clickable" to provide corresponding input to device 500). Remote control 510 also optionally includes buttons 516, 518, 520, 522, 524, and 526. Buttons 516, 518, 520, 522, 524, and 526 are optionally mechanical buttons or mechanical button alternatives that are capable of sensing contact or pressing with such buttons to initiate corresponding actions, for example, on device 500. In some embodiments, user selection of "Menu" button 516 navigates device 500 backward in a currently executed application or currently displayed user interface (e.g., backward to a user interface displayed before the currently displayed user interface), or navigates device 500 to a user interface that is one level higher than the currently displayed user interface. In some embodiments, user selection of the “home” button 518 navigates the device 500 to a main menu, home menu, or root menu user interface (e.g., to the home screen of the device 500, which optionally includes one or more applications accessible on the device 500) from any user interface displayed on the device 500. In some embodiments, user selection of the “play / pause” button 520 toggles back and forth between playing and pausing a currently playing content item on the device 500 (e.g., optionally pausing the content item if the content item is playing on the device 500 when the “play / pause” button 520 is selected, and optionally playing the content item if the content item is paused on the device 500 when the “play / pause” button 520 is selected). In some embodiments, user selection of the “+” 522 or “-” 524 button increases or decreases the volume of audio reproduced by the device 500 (e.g., the volume of the content item currently playing on the device 500), respectively. In some embodiments, user selection of the "audio input" button 526 allows the user to provide audio input (e.g., voice input) to the device 500, optionally to a voice assistant on the device. In some embodiments, the remote control 510 includes a microphone via which the user provides audio input to the device 500 upon selection of the "audio input" button 526. In some embodiments, the remote control 510 includes one or more accelerometers for detecting information about the motion of the remote control.

[0154] User interface and associated processes

[0155] Simulate click

[0156] Users interact with electronic devices in many different ways, including interacting with content (e.g., music, movies, etc.) available (e.g., stored or otherwise accessible) on the electronic device. In some cases, users may use a dedicated remote control with button click functionality (such as a Figures 5A to 5B However, in some cases, a user may desire to use a multifunction device (such as a touch-sensitive surface) that does not have button click functionality. Figure 5A The embodiment described below provides a method for simulating button click functions on a device with a touch-sensitive surface, thereby enhancing user interaction with the electronic device. Enhanced interaction with the device reduces the amount of time required for the user to perform an action, thereby reducing the power consumption of the device and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, the person is optionally referred to as a user of the device.

[0157] Figures 6A to 6Q The embodiments in these figures are used to illustrate the process described below, including reference to 7A to 7E Describe the process.

[0158] Figure 6A An exemplary display 514 is shown. Display 514 optionally displays one or more user interfaces including various content. Figure 6A In the example shown, the display 514 displays a message on the electronic device (e.g., Figure 5A 514 ). The content application displays a user interface 602 that includes a plurality of selectable user interface objects 606-1, 606-2, 606-3, and 606-4. One or more of the user interface objects 606, when selected, optionally causes corresponding content (e.g., a movie, a song, a television program, a game, a menu for an application or a menu for navigating to media content, etc.) to be displayed on the display 514. Specifically, object 606-1 corresponds to content item A, object 606-2 corresponds to content item B, object 606-3 corresponds to content item C, and object 606-4 corresponds to content item D, and selection of one of the objects 606 causes playback of the corresponding content item A, B, C, or D on the display 514. By placing the current focus indicator— Figure 6A606-2—moves to the desired user interface object 606 and detects a dedicated remote control (e.g., Figure 5B 500 ). In some cases, however, a user may desire to provide selections and other inputs to the electronic device 500 using a device other than a dedicated remote control; for example, a multifunction device (e.g., a mobile phone, a media playback device, or a wearable device) that is configured to operate in a manner similar to a dedicated remote control. Such devices optionally do not include a touch-sensitive surface with mechanical click or contact intensity detection capabilities, as previously described. The touch-sensitive surface 604 optionally corresponds to such a device (e.g., the touch-sensitive surface 604 is optionally included in a multifunction device that is configured to simulate the functionality of a dedicated remote control in the electronic device 500). In these cases, it is beneficial to simulate click or selection input functionality on the touch-sensitive surface 604 to enhance the interaction between the touch-sensitive surface 604 and the electronic device 500. The device in which the touch-sensitive surface 604 is included optionally corresponds to Figure 1A Device 100, Figure 2 Device 100, Figure 3 Device 300 and Figure 5A For ease of description, actions optionally taken by a device including a touch-sensitive surface 604 therein (e.g., transmitting commands to the electronic device 500, processing touch input, recognizing a contact as a specific input, tracking various characteristics of a contact, etc.) will be described as being taken by the touch-sensitive surface 604, although it will be understood that in some embodiments, the device rather than the touch-sensitive surface 604 takes these actions.

[0159] As mentioned above, in Figure 6A 6 , object 606-2 has current focus. When object 606-2 has current focus, a landing of contact 608 on touch-sensitive surface 604 is detected. Due to the landing of contact 608, touch-sensitive surface 604 optionally sends information 620 about the location of contact 608 on the touch-sensitive surface and / or a landing event 622 to electronic device 500 to allow the electronic device to respond accordingly.

[0160] Additionally, as a result of the landing of contact 608, touch-sensitive surface 604, or a device including touch-sensitive surface 604, optionally begins tracking the movement of contact 608 and the duration of contact 608 on touch-sensitive surface 604 (e.g., the length of time between the landing and lift-off of contact 608, in which case the touch-sensitive surface 604 may be used to track the movement of contact 608 and the duration of contact 608 on touch-sensitive surface 604). Figure 6A Specifically, if the touch-sensitive surface 604 is within the time threshold (between Figure 6A) during which movement of contact 608 is detected to be greater than a movement threshold (shown as time threshold 612 in FIG. Figure 6A 614), contact 608 and its movement are optionally recognized as movement input. On the other hand, if touch-sensitive surface 604 detects movement of contact 608 less than movement threshold 614 during time threshold 612 and lift-off of contact 608 within time threshold 612, touch-sensitive surface 604 optionally recognizes contact 608 as a click or selection input. In this way, touch-sensitive surface 604 can simulate the button click function of a dedicated remote control, for example. The above-described and other behaviors will be described in more detail below.

[0161] exist Figure 6B In, when Figure 6A After contact 608 is detected landing, contact 608 has moved less than a movement threshold 614. Some amount of time T1 less than a time threshold 612 has passed since landing of contact 608, as shown by duration bar 610. In some embodiments, while contact 608 is landing on touch-sensitive surface 604, touch-sensitive surface 604 continuously transmits information 620 about the location of contact 608 to electronic device 500, as shown in FIG. Figure 6B shown.

[0162] exist Figure 6C , the device has detected the lift-off of contact 608 from touch-sensitive surface 604 after movement less than a movement threshold 614. After time T1, within a time threshold 612 of the detection of the touchdown of contact 608, the lift-off of contact 608 is detected at time T2, as shown by duration bar 610. In response to detecting the lift-off of contact 608, touch-sensitive surface 604 optionally transmits a lift-off event 624 to electronic device 500 to allow the electronic device to respond accordingly. Because the lift-off of contact 608 was detected within the threshold time 612 of the touchdown of contact 608, and because contact 608 moved less than the movement threshold 614 during this time, touch-sensitive surface 604 optionally recognizes the touch input including contact 608 as a click or selection input. As a result, touch-sensitive surface 604 transmits a simulated button press event 626 to electronic device 500, followed by a simulated button release event 628. Additionally, in some embodiments, when a touch input including contact 608 is recognized as a click or selection input, the touch-sensitive surface 604 provides a tactile output to the user (e.g., by a Figure 6CThe touch-sensitive surface 604 may be a surface that receives a touch 608 and a selection input. ...

[0163] exist Figure 6D Because when contact 608 in Figure 6C When the click or selection input is recognized in the figure, the object 606 - 2 corresponding to the content item B has the current focus, so the electronic device 500 displays the content item B on the display 514 .

[0164] Figures 6E to 6G 6 shows a scenario where contact 608 moves more than a movement threshold 614 within a time threshold 612. Specifically, Figure 6E , the landing of contact 608 is detected (e.g., as shown in FIG. Figure 6A described). Figure 6F , contact 608 has moved more than movement threshold 614 in an amount of time (e.g., T3, as shown by duration bar 610) that is less than time threshold 612. As a result, touch-sensitive surface 604 optionally does not recognize contact 608 as a click or selection input (e.g., as shown by Figures 6A to 6C Thus, touch-sensitive surface 604 optionally initiates an operation to display a change in the appearance of object 606-2 (the object with current focus) on display 514 to indicate that continued movement of contact 608 will result in a change of focus to a different object on display 514. Figure 6F In the example of , because contact 608 is moving to the left, the appearance of object 606-2 is changed to show a skew to the left to indicate that continued movement of contact 608 will cause the current focus to change to object 606-1. In some embodiments, object 606-2 is optionally skewed or tilted up or down based on the upward or downward movement of contact 608 detected on touch-sensitive surface 604 (in a manner similar to how it skews or tilts to the right or left based on the right or left movement of contact 608 detected on touch-sensitive surface 604). Figure 6F , further movement of contact 608 to the left optionally causes object 606-2 to lose current focus, and object 606-1 to receive current focus, as shown in FIG. Figure 6G shown.

[0165] exist Figure 6G, continued movement of contact 608 to the left is detected between time T3 and time T4, and the current focus is changed to object 606-1 based on the detected continued movement. Because the current focus has moved from object 606-2 to object 606-1, the appearance of object 606-2 is optionally restored to its original appearance. Figure 6E As previously mentioned, when contact 608 lands on touch-sensitive surface 604, touch-sensitive surface 604 optionally continuously transmits information 620 (e.g., Figures 6E to 6G shown).

[0166] Figures 6H to 6L 6. A scenario is shown in which contact 608 moves less than a movement threshold 614 and lift-off of contact 608 is detected after a time threshold 612, simulating a button press followed by a button release on a dedicated remote control. Figure 6H , the landing of contact 608 is detected (e.g., as shown in FIG. Figure 6A described). Figure 6I , contact 608 has moved less than movement threshold 614 in an amount of time (e.g., T5, as shown by duration bar 610) that is less than time threshold 612. Figure 6J , contact 608 has continued to move less than movement threshold 614 and remains in contact with touch-sensitive surface 604 (e.g., has not been lifted from touch-sensitive surface 604) when time threshold 612 ends (as shown by duration bar 610). As a result, at the end of time threshold 612, touch-sensitive surface 604 optionally recognizes contact 608 as a button press input and transmits a simulated button press event 626 to electronic device 500. In response to receiving button press event 626, electronic device 500 optionally changes the appearance of object 606-2 (the object with current focus) to indicate that lifting of contact 608 will cause content item B—associated with object 606-2—to be shown on display 514. Specifically, in response to button press event 626, object 606-2 is optionally “pressed back” into user interface 602 and is therefore shown at a slightly smaller size than other objects 606 on display 514, as shown in FIG. Figure 6J In addition, in some embodiments, when contact 608 is recognized as a button press input, touch-sensitive surface 604 provides tactile output to the user (e.g., by Figure 6J ) to indicate that the user's input is recognized as a button press input.

[0167] exist Figure 6K, contact 608 has been lifted off of touch-sensitive surface 604 after time threshold 612 (e.g., T6, as shown by duration bar 610). In response to detecting the lift-off of contact 608, touch-sensitive surface 604 optionally transmits a lift-off event 624 to electronic device 500 to allow the electronic device to respond accordingly. Additionally, upon detecting the lift-off of contact 608, touch-sensitive surface 604 transmits a simulated button release event 628 to electronic device 500 and optionally provides a second tactile output to the user (e.g., by a Figure 6K The vibration represented by the zigzag pattern on the touch-sensitive surface 604 in FIG. 5 is used to indicate that the lift-off of the contact 608 is recognized as a button release input. The appearance of the object 606-2 on the display 514 is also restored to the appearance it had at the time. Figures 6H to 6I , because contact 608 has been lifted from the touch-sensitive surface (eg, the simulated button press has been released), and object 606 - 2 is no longer “pressed back” into user interface 602 .

[0168] exist Figure 6L Because when contact 608 is recognized as a button press input (in Figure 6J ), followed by a button release input (in Figure 6K ), object 606-2 corresponding to content item B has the current focus, so the electronic device 500 displays content item B on the display 514. As previously mentioned, when contact 608 lands on the touch-sensitive surface 604, the touch-sensitive surface 604 optionally continuously transmits information 620 about the location of contact 608 to the electronic device 500 (e.g., Figures 6H to 6J shown).

[0169] Figures 6M to 6N 6 shows a scenario in which contact 608 has moved less than movement threshold 614 during time threshold 612, and thus has been recognized as a button press input, and has moved after being so recognized. Figure 6M , contact 608 has been recognized as a button press input at time threshold 612 (e.g., as shown in FIG. Figures 6H to 6J described). Figure 6N In some embodiments, the movement of contact 608 after being recognized as a button press input is not recognized as a movement input and therefore does not cause a change in the appearance of object 606-2 (e.g., the object with current focus) that the movement of contact 608 before being recognized as a button press input may have caused (e.g., as shown in FIG. Figures 6E to 6F described above).

[0170] Figures 6O to 6Q612 , and thus is recognized as a button press input, and has continued to move less than movement threshold 614 while remaining landed on touch-sensitive surface 604 for a second time threshold 618 that is longer than time threshold 612. Figure 6O , contact 608 has been recognized as a button press input at time threshold 612 (e.g., as shown in FIG. Figures 6H to 6J described). Figure 6P , contact 608 has remained landed on touch-sensitive surface 604 through time threshold 618, which is longer than time threshold 612. Additionally, contact 608 has moved less than movement threshold 614 during time threshold 618. As a result, contact 608 is optionally recognized as a long press input that causes electronic device 500 to enter object rearrangement mode, in which object 606 may be rearranged in response to movement detected on touch-sensitive surface 604. In some embodiments, when object rearrangement mode is entered, the appearance of object 606-2 (the object with current focus) is optionally changed to indicate that subsequent movement of contact 608 will result in movement of object 606-2 within the arrangement of object 606 in user interface 602. Figure 6P In the example of , object 606-2 is enlarged relative to other objects 606 to indicate that subsequent movement of contact 608 will result in movement of object 606-2. Alternatively or in addition, the object optionally also moves slightly (e.g., wiggles or shakes) to indicate that it can move within the multiple objects.

[0171] exist Figure 6Q , after being recognized as a long press input (e.g., between time threshold 618 and time T8), contact 608 has moved to the right. As a result, object 606-2 has moved to the right into objects 606 in accordance with the movement of contact 608, and specifically, has replaced object 606-3, which has moved to replace object 606-2's original position in the arrangement of objects 606. Additional movement of contact 608 on the touch-sensitive surface optionally causes further movement of object 606-2 in the arrangement of objects 606 in accordance with the additional movement of contact 608.

[0172] 7A to 7E 7 is a flow chart illustrating a method 700 for simulating a button click function on a device having a touch-sensitive surface but not having a button click function according to some embodiments of the present disclosure. Method 700 is optionally performed on an electronic device (such as device 100, device 300, or device 500), as described above in conjunction with Figure 1A to Figure 1B 、 Figures 2 to 3 and Figures 5A to 5BSome operations in method 700 are optionally combined, and / or the order of some operations is optionally changed.

[0173] As described below, method 700 provides a way to simulate button click functionality on a device with a touch-sensitive surface that does not have button click functionality. This method reduces the cognitive burden on the user when interacting with the device user interface of the present disclosure, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charges.

[0174] In some embodiments, when a corresponding object of multiple selectable user interface objects in a user interface displayed on a display (e.g., a television connected to a set-top box) has focus (e.g., a representation of a content item available on the set-top box), an electronic device (e.g., a mobile phone, a remote control, a media playback device, a set-top box connected to the television, such as device 100, 300, or 500 or remote control 510) detects (702) touch input on a touch-sensitive surface (e.g., a touchpad, a touch screen) of an input device (e.g., a remote control, a mobile phone, or a media playback device configured to control the user interface controlling the set-top box), wherein detecting the touch input includes detecting landing of a contact on the touch-sensitive surface, such as on Figure 6A In some embodiments, after detecting the landing of the contact (704): based on determining (e.g., determined on a mobile phone, remote control, media playback device, set-top box connected to a television) that the touch input includes the landing of the contact followed by the lifting of the contact within a first time threshold (e.g., 20ms, 50ms, 80ms, 100ms, 150ms, before the command corresponding to the touch input is transmitted to the set-top box connected to the display), and the movement of the contact is less than a threshold movement amount (e.g., 0.5mm, 1mm, or 2mm; the contact lands on and lifts off the touch-sensitive surface without substantially moving (e.g., moving less than one or two pixels)), the electronic device initiates (706) an operation to display content associated with the corresponding object on the display, such as in Figures 6B to 6D In some embodiments, after detecting landing of the contact, based on determining that the touch input includes landing of the contact followed by movement of the contact greater than a threshold movement amount within a first time threshold (e.g., the contact lands on the touch-sensitive surface and moves substantially), the electronic device initiates (708) an operation to display a change in the appearance of the corresponding object on the display to indicate that continued movement of the contact will result in a change of focus to a different one of the multiple selectable user interface objects in the user interface displayed by the display, such as in Figures 6E to 6FIn some embodiments, the appearance of the corresponding object, such as its shading, color, positioning, etc., changes as the contact in the touch input moves.

[0175] In some embodiments, based on determining that the touch input includes a landing of the contact followed by a movement of the contact greater than a threshold movement amount within a first time threshold, the electronic device forgoes initiating (710) an operation to display content associated with the corresponding object when the contact is lifted from the touch-sensitive surface, such as in Figures 6E to 6F For example, if a contact moves substantially after landing on the touch-sensitive surface, the contact is optionally not recognized as a "click" or selection input, but rather as a movement input. Thus, the touch input does not select the corresponding object with the current focus.

[0176] In some embodiments, after detecting landing of the contact, based on determining that the touch input includes landing of the contact followed by lifting of the contact after a first time threshold, and the movement of the contact during the first time threshold is less than a threshold movement amount (e.g., 0.5 mm, 1 mm, or 2 mm; during the first time threshold, the contact lands on and lifts off the touch-sensitive surface without substantially moving (e.g., moving less than one or two pixels). For example, an input corresponding to a button press is detected for a time period that is shorter than a time period for detecting a long button press input), the electronic device initiates (712) an operation to display a change in the appearance of the corresponding object on the display to indicate that lifting of the contact will cause content associated with the corresponding object to be displayed on the display, such as in Figures 6H to 6K For example, if the contact remains down longer than a first time threshold, the electronic device optionally generates a simulated button press event at the end of the first time threshold, such as at Figure 6J The lifting of the contact after the first time threshold optionally causes the electronic device to generate a simulated button release event when the lifting of the contact is detected, such as in Figure 6K If the touchdown is maintained longer than the first time threshold but shorter than the second time threshold, the simulated button press event optionally causes the corresponding object to be pushed back into the user interface to indicate that liftoff of the contact will result in selection of the corresponding object and, therefore, playback of content associated with the corresponding object.

[0177] In some embodiments, after detecting the landing of the contact, based on determining that the touch input includes the landing of the contact, followed by the lifting of the contact after a first time threshold, and the movement of the contact during the first time threshold is less than a threshold movement amount, the electronic device detects (714) the movement of the contact after the first time threshold without initiating an operation to display a change in the appearance of the corresponding object on the display based on the detected movement of the contact after the first time threshold, such as in Figures 6M to 6N For example, once a touch input is recognized as corresponding to a simulated button press event because it has been substantially stationary for a first time threshold, subsequent movement of the contact is optionally not recognized as corresponding to an input for moving the current focus in the user interface. In this way, the appearance of the corresponding object in the user interface is optionally not changed to indicate that the current focus will change as the contact continues to move.

[0178] In some embodiments, after detecting landing of the contact, based on determining that the touch input includes landing of the contact, followed by lifting of the contact after a second time threshold that is longer than the first time threshold (for example, input corresponding to a button press is detected for a time period longer than the time period for detecting a long button press input), and the movement of the contact during the second time threshold is less than a threshold movement amount, the electronic device initiates (716) an operation to display a change in the appearance of the corresponding object on the display to indicate that subsequent movement of the contact will cause movement of the corresponding object within the arrangement of the multiple selectable user interface objects, such as in Figures 6O to 6P In some embodiments, subsequent movement of the contact then optionally moves the corresponding object relative to other objects in the user interface in accordance with the movement of the contact, such as in Figure 6Q Initiation of the mode is optionally indicated by changing the appearance of the corresponding object, such as causing the corresponding object to twist or jiggle in place.

[0179] In some embodiments, where it is determined that the touch input includes a landing of the contact, followed by a lift-off of the contact after a second time threshold, and the movement of the contact during the second time threshold is less than a threshold movement amount (718), after the second time threshold (720): the electronic device detects (722) a subsequent movement of the contact (e.g., detecting the movement of the contact after the touch input is recognized as corresponding to an input for moving the corresponding object in the user interface), and initiates (724) an operation to move the corresponding object within the arrangement of the multiple selectable user interface objects based on the detected subsequent movement of the contact, such as in Figures 6P to 6Q middle.

[0180] In some embodiments, the electronic device includes (726) an input device and a touch-sensitive surface (e.g., the electronic device is a mobile phone with a touch screen that is configured as an input device (e.g., a remote control) for a second electronic device such as a set-top box connected to a television. In some embodiments, initiating an operation to display content associated with the corresponding object includes transmitting (728) a corresponding first event (e.g., a remote control command, such as a button press event, a button release event) by the electronic device to a second electronic device different from the electronic device (e.g., a set-top box connected to the television) to display the content associated with the corresponding object on the display, such as on the screen. Figure 6C In the embodiment of the present invention, the electronic device processes the touch input and recognizes it as a selection input, and after processing the touch input, transmits a command corresponding to the selection input (e.g., a button press event and a button release event) to the second electronic device, and initiating an operation to display a change in the appearance of the corresponding object includes transmitting (730) a corresponding second event (e.g., a remote control command, such as one or more contact movement events) to the second electronic device to display a change in the appearance of the corresponding object, such as in the embodiment of the present invention. Figure 6F In some embodiments, the electronic device comprises a mobile phone.

[0181] In some embodiments, after detecting the landing of the contact, the electronic device continuously transmits (734) information about the location of the contact on the touch-sensitive surface of the electronic device to the second electronic device, such as in Figures 6A to 6Q For example, the electronic device optionally transmits a contact location command to the second electronic device regardless of which operation the electronic device initiates based on the characteristics of the touch input. In this way, the second electronic device optionally always has information about the location of the contact on the touch-sensitive surface and responds appropriately.

[0182] In some embodiments, in response to detecting the landing of the contact, the electronic device transmits (736) a simulated landing event to the second electronic device, such as in Figure 6A For example, in response to detecting the contact, the electronic device optionally sends information to the second electronic device indicating that contact has been detected on the touch-sensitive surface.

[0183] In some embodiments, based on determining that the touch input includes landing of a contact followed by lifting of the contact within a first time threshold (e.g., 20ms, 50ms, 80ms, 100ms, 150ms), and the movement of the contact is less than a threshold movement amount (e.g., 0.5mm, 1mm, or 2mm; within the first time threshold, the contact lands on and lifts off the touch-sensitive surface without substantially moving (e.g., moving less than one or two pixels)), the electronic device transmits (738) a simulated button press event to a second electronic device, followed by a simulated button release event, such as in Figure 6C In some embodiments, the simulated button press event is the same as the button press event sent to the second electronic device when a physical button of the dedicated remote control device is pressed, and based on the button press and subsequent button release of the physical (simulated) button, the object with current focus in the user interface is optionally pushed down and popped out.

[0184] In some embodiments, after detecting the landing of the contact, based on determining that the touch input includes the landing of the contact, followed by the lifting of the contact after a first time threshold, and the movement of the contact during the first time threshold is less than a threshold movement amount (740): (for example, during the first time threshold, a long and substantially stationary contact is detected), in response to detecting the end of the first time threshold, the electronic device transmits (742) a simulated button press event to the second electronic device, such as in Figure 6J (e.g., the touch input is optionally recognized as corresponding to a button press at the end of the first time threshold. In some embodiments, the simulated button press event is the same as the button press event sent to the second electronic device when a physical button of the dedicated remote control device is pressed.) In some embodiments, in response to detecting the lift-off of the contact, the electronic device transmits (744) a simulated button release event to the second electronic device, such as in Figure 6K In some embodiments, the simulated button release event is the same as the button release event sent to the second electronic device when a physical button of the dedicated remote control device is released.

[0185] In some embodiments, the electronic device includes a multifunction device. In some embodiments, the multifunction device is a mobile phone (746) running a remote control application that is configured to perform multiple functions (such as phone functions, messaging functions, etc.) independent of controlling content displayed on a display (e.g., the electronic device is configured to run applications unrelated to the control functions of a set-top box), such as in Figures 10A to 10NIn one embodiment, the present invention relates to a remote control application (e.g., software on the multifunction device for configuring the multifunction device to operate as a remote control for a second electronic device, such as a set-top box), and the remote control application causes the electronic device to transmit (748) to the second electronic device an event including a corresponding first event and a corresponding second event, the transmitted event corresponding to an event transmitted to the second electronic device by a dedicated remote control device for the second electronic device, the dedicated remote control device having a touchpad including button click functionality. For example, the application optionally configures the multifunction device to operate in a manner similar to the dedicated remote control device and thereby transmits to the second electronic device remote control events corresponding to the remote control events that the dedicated remote control device would transmit to the second electronic device. The dedicated remote control device is optionally a remote control device having a physical actuator for allowing clicks on a button or surface of the remote control, or a remote control device having a tactile actuator and a pressure detector coupled to a surface of the remote control device (e.g., a touch-sensitive surface, a touch screen, etc.), the pressure detector being used to trigger the tactile actuator when contact is detected at one or more predefined pressures on the surface of the remote control device.

[0186] In some embodiments, after detecting the landing of the contact (750): based on determining that the touch input includes the landing of the contact, followed by the lifting of the contact within a first time threshold, and the movement of the contact is less than the threshold movement amount, in response to detecting the lifting of the contact, the electronic device initiates (752) an operation to provide tactile feedback at the input device, such as at Figure 6C In some embodiments, the electronic device initiates (754) an operation to provide first tactile feedback at the input device in response to detecting the end of the first time threshold, such as at the end of the first time threshold, based on determining that the touch input includes a touchdown followed by a liftoff of the contact after a first time threshold, and the movement of the contact during the first time threshold is less than a threshold movement amount (e.g., the contact is relatively long and has substantially no movement). Figure 6J and providing second tactile feedback at the input device in response to detecting lift-off of the contact, such as in Figure 6K In some embodiments, a simulated "release" of the touch-sensitive surface is optionally provided when the contact is lifted from the touch-sensitive surface (e.g., if the contact is a relatively long contact with substantially no movement, a simulated "click" of the touch-sensitive surface is optionally provided at the end of the first time threshold).

[0187] It should be understood that 7A to 7E The particular order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 900, 1100, 1300, 1500, 1700, and 1900) also apply in a similar manner to the details of the processes described above with respect to 7A to 7E Method 700 described herein. For example, the touch-sensitive surfaces, user interface objects, tactile outputs, software remote control applications, simulated buttons, simulated remote control trackpads, and / or touch inputs described above with reference to method 700 optionally have one or more of the features of the touch-sensitive surfaces, user interface objects, tactile outputs, software remote control applications, simulated buttons, simulated remote control trackpads, and / or touch inputs described herein with reference to other methods described herein (e.g., methods 900, 1100, 1300, 1500, 1700, and 1900). For the sake of brevity, these details are not repeated here.

[0188] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A 、 Figure 3 、 Figure 5A and Figure 20 In addition, the above reference 7A to 7E The described operations are optionally performed by Figure 1A to Figure 1B. For example, the detection operation 702 and the initiation operation 706 and the initiation operation 708 are optionally implemented by the event classifier 170, the event identifier 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 604, and the event distributor module 174 delivers the event information to the application 136-1. The corresponding event identifier 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know how to update the content displayed by the application based on the GUI updater 178. Figure 1A to Figure 1B The components depicted in the figure implement other processes.

[0189] Movement-dependent intensity threshold

[0190] Users interact with electronic devices in many different ways, including interacting with content (e.g., music, movies, etc.) available (e.g., stored or otherwise accessible) on the electronic device. In some cases, users may use a dedicated remote control with button click functionality and / or a multifunction device (such as a touch-sensitive surface with contact intensity detection capabilities) that includes a touch screen. Figures 5A to 5B The remote control 510 and Figure 5A 511 in the figure) to interact with the electronic device. When the intensity of the contact is higher than a predefined intensity threshold, a click or selection input is optionally detected at the touch-sensitive surface. However, in some cases, when providing mobile input, the user may unintentionally provide more force on the touch-sensitive surface than when providing static input, which may result in unintentional detection of click or selection input at the touch-sensitive surface. The embodiment described below provides a way in which the electronic device reduces the unintentional recognition of click or selection input when the user is providing mobile touch input on the touch-sensitive surface, thereby enhancing the interaction between the user and the electronic device. Enhancing the interaction with the device reduces the amount of time required for the user to perform an operation, thereby reducing the power consumption of the device and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, this person is optionally referred to as a user of the device.

[0191] Figures 8A to 8RFIGURES 10 and 11 illustrate exemplary ways in which an electronic device reduces unintentional recognition of click or selection inputs when a user is providing moving touch inputs on a touch-sensitive surface according to some embodiments of the present disclosure. The embodiments in these figures are used to illustrate the processes described below, including reference to FIGURES 10 and 12. Figures 9A to 9G Describe the process.

[0192] Figure 8A An exemplary display 514 is shown. Display 514 optionally displays one or more user interfaces including various content. Figure 8A In the example shown, the display 514 displays a message on the electronic device (e.g., Figure 5A 800) is an application running on an electronic device 500 of which the display 514 is a part or connected. The application displays the user interface 802. In some embodiments, the application is a content application (e.g., a content playback application) for displaying or playing content (e.g., a movie, a song, a television program, a game, a menu for an application or a menu for navigating to media content, etc.), as described in reference to FIG. Figures 6A to 6Q Providing selection input to the application (e.g., to display content on the display 514) is optionally accomplished by detecting a dedicated remote control (e.g., Figure 5B 500) (e.g., clicks on buttons on the remote control or clicks on a touch-sensitive surface of the remote control). In some cases, however, a user may desire to provide selections and other inputs to the electronic device 500 using a device other than a dedicated remote control; for example, a multifunction device (e.g., a mobile phone, a media playback device, or a wearable device) that is configured to operate in a manner similar to a dedicated remote control. Such devices optionally include a touch-sensitive surface with contact intensity detection capabilities. The touch-sensitive surface 805 optionally corresponds to such a device (e.g., the touch-sensitive surface 805 is optionally included in a multifunction device that is configured to emulate the dedicated remote control functionality in controlling the electronic device 500). Figures 6A to 6Q Using contact intensity to determine a click or selection input at a touch-sensitive surface, as will be described below, is advantageous compared to the simulated button click implementation described above because the click or selection input is optionally triggered as soon as the requisite contact intensity is reached—the device does not need to delay the click or selection input until a specific time threshold is reached, for example, as described in reference to FIG. Figures 6A to 6Q The device including the touch-sensitive surface 805 optionally corresponds to Figure 1A Device 100, Figure 2 Device 100, Figure 3 Device 300 and Figure 5AFor ease of description, actions optionally taken by a device including touch-sensitive surface 805 therein (e.g., transmitting commands to electronic device 500, processing touch input, recognizing contact as specific input, tracking various characteristics of contact, etc.) will be described as being taken by touch-sensitive surface 805, although it will be understood that in some embodiments, the device rather than touch-sensitive surface 805 takes these actions.

[0193] As previously described in this disclosure, a click or select input is optionally detected at touch-sensitive surface 805 when the intensity of the contact is above a predefined intensity threshold. However, as described above, in some cases, when providing a motion input, a user may inadvertently press harder on touch-sensitive surface 805 than when providing a stationary input. Furthermore, the user may not be aware that they are pressing harder. Therefore, to reduce unintentional recognition of a click or select input when the user is providing a moving touch input on touch-sensitive surface 805, the intensity required to trigger such a click or select input is optionally adjusted based on the motion detected on touch-sensitive surface 805, as will be described below.

[0194] Reference again Figure 8A , contact 807 is detected on touch-sensitive surface 805. As contact 807 lands, touch-sensitive surface 805 optionally detects the speed of contact 807 (shown in speed bar 804) and the intensity of contact 807 (shown in intensity bar 806). Figure 8A In some embodiments, contact 807 has an intensity less than intensity threshold 808 (e.g., an intensity corresponding to a finger resting on touch-sensitive surface 805). Additionally, in some embodiments, while contact 807 lands on touch-sensitive surface 805, touch-sensitive surface 805 continuously transmits information about the location of contact 807 to electronic device 500, and as contact 807 lands and lifts off from touch-sensitive surface 805, land events and lift-off events are transmitted to the electronic device, as described in reference to FIG. Figures 6A to 6Q As stated.

[0195] exist Figure 8B , contact 807 is moving at speed S1, and the intensity of contact 807 has increased above intensity threshold 808. As a result, touch-sensitive surface 805 has recognized contact 807 as a click or selection input, and has transmitted a selection event 810 to electronic device 500 to allow the electronic device to respond accordingly (e.g., as described in reference to FIG. Figures 6A to 6Q In some embodiments, selection event 810 corresponds to reference Figures 6A to 6Q The button press event 626 and / or release event 628 described above are also provided. In addition, in some embodiments, when contact 807 is recognized as a click or selection input, the touch-sensitive surface 805 provides a tactile output to the user (e.g., by a Figure 8BThe touch-sensitive surface 805 may be used to detect a touch input (e.g., a vibration represented by a sawtooth pattern on the touch-sensitive surface 805) to indicate that the user's input is recognized as a click or selection input. For ease of description, in the remainder of this disclosure, the touch-sensitive surface 805 will be described as recognizing contact 807 as a specific input (e.g., a click or selection input) rather than recognizing "touch input including contact 807" as a specific input. Furthermore, in some embodiments, the input is processed and analyzed by the electronic device 500 in addition to or alternatively to being processed and analyzed by the touch-sensitive surface 805.

[0196] Figure 8C A different scenario is shown, where contact 807 is not already Figure 8B As a result, the intensity required to generate the click or selection input (in Figure 8C ) is greater than the intensity required to generate a click or selection input when contact 807 is moving at speed S1 (shown as intensity threshold 812 in FIG. Figure 8C 808). As such, unintentional recognition of click or selection inputs is reduced when movement is detected on the touch-sensitive surface 805, as previously described. Figure 8C The contact 807 optionally has a Figure 8B However, due to the increased intensity threshold 812 for generating a click or selection input, Figure 8C Contact 807 in does not generate a click or selection input, and therefore the touch-sensitive surface 805 does not transmit a selection event to the electronic device 500.

[0197] Figures 8D to 8E Recognition of a click and hold input (e.g., corresponding to a substantially stationary contact 807 that has generated a click or select input) is shown. Figure 8D , contact 807 is moving at speed S1 and has a strength threshold 808 (e.g., a strength threshold corresponding to contact speed S1, as shown in FIG. Figure 8B As a result, the touch-sensitive surface 805 has recognized the contact 807 as a click or selection input and has transmitted a selection event 810 to the electronic device 500 to allow the electronic device to respond accordingly (e.g., as described in reference to FIG. Figures 6A to 6Q described above).

[0198] In some embodiments, after contact 807 is recognized as a click or selection input, the touch-sensitive surface 805 tracks the movement of contact 807 to determine whether contact 807 moves greater than a movement threshold 814, such as Figure 8E If the contact 807 moves less than the movement threshold 814 after being recognized as a click or selection input, as shown in FIG. Figure 8EAs shown, the touch-sensitive surface 805 transmits a click and hold event 816 to the electronic device 500 based on the characteristics of the detected contact 807 to allow the electronic device to respond accordingly (e.g., as shown in FIG. Figures 6A to 6Q described above).

[0199] Figures 8F to 8G Recognition of a click and drag input (e.g., corresponding to a substantially moving contact 807 that has generated a click or select input) is shown. Figure 8F , contact 807 is moving at speed S1 and has a strength threshold 808 (e.g., a strength threshold corresponding to contact speed S1, as shown in FIG. Figure 8B As a result, the touch-sensitive surface 805 has recognized the contact 807 as a click or selection input and has transmitted a selection event 810 to the electronic device 500 to allow the electronic device to respond accordingly (e.g., as described in reference to FIG. Figures 6A to 6Q described above).

[0200] exist Figure 8G , after contact 807 is recognized as a click or selection input, contact 807 has moved greater than a movement threshold 814. As a result, touch-sensitive surface 805 transmits a click and drag event 818 to electronic device 500 based on the characteristics of detected contact 807 to allow the electronic device to respond accordingly (e.g., as described in reference to FIG. Figures 6A to 6Q described above).

[0201] Figures 8H to 8I Recognition of a tap input (e.g., corresponding to a substantially stationary contact 807 that does not generate a click or select input) is shown. Figure 8H , contact 807 is moving at speed S2, so the intensity required to generate a click or selection input increases to intensity threshold 812, as shown in FIG. Figure 8C The contact 807 has a strength threshold 808 (e.g., a strength threshold corresponding to the contact speed S1, as shown in FIG. Figure 8B ) but does not meet the intensity threshold 812 (e.g., the intensity threshold corresponding to the contact speed S2, as shown in reference Figure 8C As a result, Figure 8H Contact 807 in does not generate a click or selection input, and therefore the touch-sensitive surface 805 does not transmit a selection event to the electronic device 500.

[0202] In some embodiments, after contact 807 moves at speed S2, touch-sensitive surface 805 tracks the movement of contact 807 to determine whether contact 807 moves greater than movement threshold 814, such as Figure 8I As shown. Figure 8I, after moving at speed S2, contact 807 has moved less than movement threshold 814, and thus touch-sensitive surface 805 transmits a tap event 820 to electronic device 500 based on the characteristics of detected contact 807 to allow the electronic device to respond accordingly (e.g., as described in reference to FIG. Figures 6A to 6Q described above).

[0203] Figures 8J to 8K Recognition of a swipe input (e.g., corresponding to a substantially moving contact 807 that does not generate a click or select input) is shown. Figure 8J , contact 807 is moving at speed S2, so the intensity required to generate a click or selection input increases to intensity threshold 812, as shown in FIG. Figure 8C The contact 807 has a strength threshold 808 (e.g., a strength threshold corresponding to the contact speed S1, as shown in FIG. Figure 8B ) but does not meet the intensity threshold 812 (e.g., the intensity threshold corresponding to the contact speed S2, as shown in reference Figure 8C As a result, Figure 8J Contact 807 in does not generate a click or selection input, and therefore the touch-sensitive surface 805 does not transmit a selection event to the electronic device 500.

[0204] exist Figure 8K , after contact 807 moves at speed S2, contact 807 has moved greater than movement threshold 814. As a result, touch-sensitive surface 805 transmits a swipe event 822 to electronic device 500 based on the characteristics of detected contact 807 to allow the electronic device to respond accordingly (e.g., as shown in FIG. Figures 6A to 6Q described above).

[0205] Figures 8L to 8M A further increase in the intensity threshold is shown due to faster movement of contact 807. Figure 8L In the example, contact 807 moves at a speed S2. As a result, the intensity required to generate a click or selection input (in Figure 8L ) is greater than the intensity required to generate a click or selection input when contact 807 is moving at speed S1 (shown as intensity threshold 812 in FIG. Figure 8L 808). However, unlike Figure 8C compared to, Figure 8L Contact 807 in has an intensity that exceeds intensity threshold 812. As a result, touch-sensitive surface 805 has recognized contact 807 as a click or selection input and has transmitted a selection event 810 to electronic device 500 to allow the electronic device to respond accordingly (e.g., as described in reference to FIG. Figures 6A to 6Q described above).

[0206] Figure 8M A different scenario is shown, where contact 807 is not already Figure 8LInstead of moving at speed S2, it is moving at speed S3 which is greater than speed S2. As a result, the intensity required to generate the click or selection input (in Figure 8M ) is greater than the intensity required to generate a click or selection input when contact 807 is moving at speed S2 (shown as intensity threshold 824 in FIG. Figure 8M , shown as intensity threshold 812). Figure 8M The contact 807 optionally has a Figure 8L However, due to the increased intensity threshold 824 for generating a click or selection input, Figure 8M Contact 807 in does not generate a click or selection input, and therefore the touch-sensitive surface 805 does not transmit a selection event to the electronic device 500.

[0207] Figures 8N to 8R A scenario is shown in which an increasing intensity threshold for generating a click or selection input is optionally maintained or reduced over time. Figures 8N to 8P In , two contacts are detected one after another, and whether the increased strength threshold is maintained depends on how long after the lift-off of the first contact is detected the touchdown of the second contact is detected. Figure 8N , contact A 807 is moving at speed S2, so the intensity required to generate a click or selection input increases to intensity threshold 812, as shown in FIG. Figure 8C Contact A 807 has a strength threshold 808 (eg, a strength threshold corresponding to contact velocity S1, as shown in FIG. 8 ). Figure 8B ) but does not meet the intensity threshold 812 (e.g., the intensity threshold corresponding to the contact speed S2, as shown in reference Figure 8C As a result, Figure 8N Contact A 807 in does not generate a click or selection input, and therefore the touch-sensitive surface 805 does not transmit a selection event to the electronic device 500.

[0208] exist Figure 8O , after detecting lift-off of contact A 807, touch-sensitive surface 805 detects the landing and movement of contact B 809. Contact B 809 is moving at speed S1, and contact B 809 optionally has the same intensity as contact A 807 (e.g., an intensity that satisfies intensity threshold 808 but not intensity threshold 812). In addition, the landing of contact B 809 is detected after time threshold 828 of the lift-off of contact A 807 (as shown in time bar 826). As a result, the intensity required to generate a click or selection input is reduced from 80. Figure 8N The intensity threshold 812 in (corresponding to speed S2) is reduced to Figure 8OThus, the touch-sensitive surface 805 has recognized contact B 809 as a click or selection input and has transmitted a selection event 810 to the electronic device 500 to allow the electronic device to respond accordingly (e.g., as shown in FIG. Figures 6A to 6Q described above).

[0209] Figure 8P A different scenario is shown, where contact B 809 is not already Figure 8O Instead of landing on touch-sensitive surface 805 longer than time threshold 828 after lift-off of contact A 807, landing within time threshold 828 after lift-off of contact A 807. As a result, the intensity required to generate a click or selection input (in Figure 8P shown as intensity threshold 812) remains constant due to Figure 8N The speed of contact A 807 is established by the increase level. Figure 8P Contact B 809 optionally has a Figure 8O However, due to the increased intensity threshold 812 for holding to generate a click or selection input, Figure 8P Contact B 809 in does not generate a click or selection input, and therefore the touch-sensitive surface 805 does not transmit a selection event to the electronic device 500.

[0210] exist Figures 8Q to 8R , the contact is initially moving at speed S2, thus increasing the intensity threshold for generating a click or selection input to intensity threshold 812, and then subsequently slowing to speed S1, thus decreasing the intensity threshold to intensity threshold 808. Specifically, in Figure 8Q , contact A 807 is moving at speed S2, so the intensity required to generate a click or selection input increases to intensity threshold 812, as shown in FIG. Figure 8C Contact A 807 has a strength threshold 808 (corresponding to a strength threshold of contact velocity S1, as shown in FIG. Figure 8B ) but does not meet the intensity threshold 812 (corresponding to the intensity threshold of contact speed S2, as shown in reference Figure 8C As a result, Figure 8Q Contact A 807 in does not generate a click or selection input, and therefore the touch-sensitive surface 805 does not transmit a selection event to the electronic device 500.

[0211] However, if contact A 807 slows without lifting from the touch-sensitive surface 805 (in some embodiments, if it slows for longer than a threshold amount of time), the intensity required to generate a click or selection input is optionally reduced. Figure 8R , contact A 807 slows down to speed S1 while maintaining Figure 8QAs a result, the intensity required to generate a click or selection input has been reduced to an intensity threshold 808 (e.g., an intensity threshold corresponding to contact speed S1, as shown in FIG. Figure 8B Because contact A 807 has an intensity that satisfies intensity threshold 808, the touch-sensitive surface 805 has recognized contact A 807 as a click or selection input and has transmitted a selection event 810 to the electronic device 500 to allow the electronic device to respond accordingly (e.g., as described with reference to FIG. Figures 6A to 6Q described above).

[0212] Figures 9A to 9G is a flow chart illustrating a method 900 for reducing unintentional recognition of click or selection inputs when a user is providing moving touch inputs on a touch-sensitive surface according to some embodiments of the present disclosure. Method 900 is optionally performed on an electronic device (such as device 100, device 300, device 500, or remote control 510), as described above in conjunction with Figure 1A to Figure 1B 、 Figures 2 to 3 and Figures 5A to 5B Some operations in method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0213] As described below, method 900 provides a method for reducing unintentional recognition of click or selection inputs when a user is providing moving touch input on a touch-sensitive surface. The method reduces the cognitive burden on the user when interacting with the device user interface of the present disclosure, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charges.

[0214] In some embodiments, an electronic device (e.g., a mobile phone, a remote control, a media playback device, a set-top box connected to a television, such as device 100, device 300, device 500, or remote control 510) detects (902) a touch input on a touch-sensitive surface (e.g., a touchpad or touchscreen capable of detecting the intensity of one or more contacts on the touchpad or touchscreen) of an input device (e.g., a remote control, a mobile phone, or a media playback device that controls a set-top box) that controls a user interface displayed by a display (e.g., a television connected to a set-top box), such as at Figure 8A wherein detecting touch input includes detecting landing of a contact, movement of a contact, and an increase in a characteristic intensity of the contact (e.g., the force with which the touch-sensitive surface of the touch input device is contacted) to a corresponding intensity, such as in Figures 8A to 8BIn some embodiments, in response to detecting a touch input (904): based on determining that movement of the contact satisfies a first movement criterion when an increase in a characteristic intensity of the contact to a corresponding intensity is detected, wherein the first movement criterion includes a criterion that is satisfied when the contact has a first velocity during the touch input, the device generates (906) a selection input corresponding to an increase in the intensity of the contact to the corresponding intensity, such as in Figure 8B In some embodiments, in response to detecting a touch input (904), the device forgoes generating (908) a selection input corresponding to an increase in the intensity of the contact to the corresponding intensity based on determining that the movement of the contact satisfies a second movement criterion when an increase in the characteristic intensity of the contact to the corresponding intensity is detected, where the second movement criterion includes a criterion that is satisfied when the contact has a second speed greater than the first speed during the touch input, such as in Figure 8C In some embodiments, the force with which a contact must touch the touch-sensitive surface to trigger a "mechanical click" response increases as the contact moves faster across the touch-sensitive surface. In some embodiments, this is to reduce unintentional "mechanical click" responses when the user is providing moving touch input to the touch-sensitive surface, because the user may sometimes inadvertently provide more force on the touch-sensitive surface when providing moving input than when providing stationary input.

[0215] In some embodiments, generating a selection input corresponding to an increase in the intensity of the contact to a corresponding intensity includes initiating an operation to provide tactile feedback at the input device in response to generating the selection input (910), such as in Figure 8B For example, the input device and / or the touch-sensitive surface of the input device is caused to deflect or vibrate to generate tactile output that provides the user with the sensation of "clicking" the touch-sensitive surface.

[0216] In some embodiments, based on determining that the movement of the contact meets a first movement criterion (e.g., the speed of the contact is low enough so that the pressure of the contact is sufficient to trigger a "click" because the pressure required to trigger a "click" is relatively low), and after detecting an increase in the characteristic intensity of the contact to the corresponding intensity, the movement of the contact is less than a movement threshold (e.g., 0.5 mm, 1 mm, 2 mm), the electronic device generates (912) a click and hold input corresponding to the contact, such as in Figures 8D to 8E (e.g., a relatively stationary contact with sufficient pressure to trigger a "click" is optionally recognized as a click and hold input).

[0217] In some embodiments, based on determining that the movement of the contact satisfies a first movement criterion (e.g., the velocity of the contact is low enough so that the pressure of the contact is sufficient to trigger a "click" because the pressure required to trigger a "click" is relatively low), and after detecting an increase in the characteristic intensity of the contact to a corresponding intensity, the movement of the contact is greater than a movement threshold (e.g., 0.5 mm, 1 mm, 2 mm), the electronic device generates (914) a click and drag input corresponding to the movement of the contact, such as in Figures 8F to 8G (e.g., contacts with relative movement with sufficient pressure to trigger a "click" are optionally recognized as click and drag input).

[0218] In some embodiments, based on determining that the movement of the contact meets a second movement criterion (e.g., the speed of the contact is high enough that the pressure of the contact is insufficient to trigger a "click" because the pressure required to trigger a "click" is relatively high), and the movement of the contact is less than a movement threshold (e.g., 0.5 mm, 1 mm, 2 mm), the electronic device generates (916) a tap input corresponding to the contact, such as in Figures 8H to 8I (e.g., a relatively stationary contact without sufficient pressure to trigger a "click" is optionally recognized as a tap input).

[0219] In some embodiments, based on determining that the movement of the contact meets the second movement standard (e.g., the speed of the contact is high enough so that the pressure of the contact is insufficient to trigger a "click" because the pressure required to trigger a "click" is relatively high), and the movement of the contact is greater than a movement threshold (e.g., 0.5 mm, 1 mm, 2 mm), the electronic device generates (918) a swipe input corresponding to the movement of the contact, such as in Figures 8J to 8K (e.g., a contact without relative movement with sufficient pressure to trigger a "click" is optionally recognized as a swipe input).

[0220] In some embodiments, the electronic device includes an input device and a touch-sensitive surface (920) (e.g., the electronic device is a mobile phone with a touch screen that is configured as an input device (e.g., a remote control) for a second electronic device (e.g., a set-top box connected to a television), and generating a selection input (922) includes transmitting, by the electronic device, a corresponding first event (e.g., a remote control command, such as a button press event, a button release event) to a second electronic device different from the electronic device (e.g., a set-top box connected to the television) to select a currently selected user interface element displayed by the second electronic device, such as in Figure 8B In some embodiments, the electronic device processes the touch input and recognizes it as a selection input, and after processing the touch input, transmits a command corresponding to the selection input (e.g., a button press event and a button release event) to a second electronic device. In some embodiments, the electronic device includes a mobile phone (924).

[0221] In some embodiments, in response to detecting the landing of the contact, the electronic device transmits (926) a simulated landing event to the second electronic device, such as in Figure 6A (e.g., in response to detecting contact, the electronic device optionally sends information to a second electronic device indicating that contact has been detected on the touch-sensitive surface).

[0222] In some embodiments, based on determining that the movement of the contact satisfies a first movement criterion (e.g., the velocity of the contact is low enough that the pressure of the contact is sufficient to trigger a "click" because the pressure required to trigger a "click" is relatively low), the electronic device transmits (928) a simulated button press event to the second electronic device, such as in Figure 8B (e.g., a contact with sufficient pressure to trigger a "click" is optionally recognized as an input comprising a "click," and a corresponding simulated button press event for the input is optionally transmitted to the second electronic device.) In some embodiments, the simulated button press event is the same as the button press event sent to the second electronic device when a physical button of the dedicated remote control device is pressed.

[0223] In some embodiments, the electronic device comprises a multifunction device. In some embodiments, the multifunction device is a mobile phone running a remote control application (930) configured to perform multiple functions (such as telephone functions, messaging functions, etc.) independent of controlling content displayed on a display (e.g., the electronic device is configured to run applications unrelated to the control functions of a set-top box), such as in Figures 10A to 10N In one embodiment, the present invention relates to a remote control application (e.g., software on the multifunction device for configuring the multifunction device to operate as a remote control for a second electronic device, such as a set-top box), and the remote control application causes the electronic device to transmit (932) to the second electronic device an event including a corresponding first event, the transmitted event corresponding to an event transmitted to the second electronic device by a dedicated remote control device for the second electronic device, the dedicated remote control device having a touchpad including button click functionality. For example, the application optionally configures the multifunction device to operate in a manner similar to the dedicated remote control device and thereby transmits to the second electronic device a remote control event corresponding to the remote control event that the dedicated remote control device would transmit to the second electronic device. The dedicated remote control device is optionally a remote control device having a physical actuator for allowing physical clicks on a button or surface of the remote control, or a remote control device having a tactile actuator and a pressure detector coupled to a surface of the remote control device (e.g., a touch-sensitive surface, a touch screen, etc.), the pressure detector being used to trigger the tactile actuator when contact is detected at one or more predefined pressures on the surface of the remote control device.

[0224] In some embodiments, the electronic device detects (934) a second touch input on a touch-sensitive surface (e.g., a touchpad or touch screen capable of detecting the intensity of one or more contacts on the touchpad or touch screen) of an input device (e.g., a remote control, a mobile phone, or a media playback device that controls a set-top box), wherein detecting the second touch input includes detecting landing of the second contact, movement of the second contact, and an increase in a characteristic intensity of the second contact (e.g., the force with which the second contact touches the touch-sensitive surface of the input device) to a second corresponding intensity that is greater than the corresponding intensity, such as in Figure 8L In some embodiments, in response to detecting a second touch input (936): based on determining that movement of the second contact satisfies a second movement criterion when an increase in the characteristic intensity of the second contact to a second corresponding intensity is detected, wherein the second movement criterion includes a criterion that is satisfied when the second contact has a second speed greater than the first speed during the touch input, the electronic device generates (938) a selection input corresponding to the increase in the intensity of the second contact to the second corresponding intensity, such as in Figure 8L (e.g., relatively rapid contact movement results in a relatively high intensity threshold for triggering a selection or “click” input. However, the pressure of the second contact is optionally high enough to trigger a “click” on the touch-sensitive surface, although doing so requires a higher pressure than the pressure of the first contact which is optionally insufficient to trigger a “click” when the second movement criterion is met). In some embodiments, in response to detecting a second touch input (936), the electronic device forgoes generating (940) a selection input corresponding to the increase in intensity of the second contact to the second corresponding intensity based on determining that the movement of the second contact satisfies a third movement criterion when an increase in the characteristic intensity of the second contact to a second corresponding intensity is detected, where the third movement criterion includes a criterion that is satisfied when the second contact has a third speed greater than the second speed during the second touch input, such as in Figure 8M (e.g., faster movement of the second contact optionally results in a higher contact intensity threshold, and when the third movement criterion is met, the pressure of the second contact optionally is insufficient to trigger a "click" on the touch-sensitive surface).

[0225] In some embodiments, the movement of the contact satisfies a second movement criterion (942) (e.g., the first contact has a relatively high velocity, thereby increasing the intensity required to trigger a "click" on the touch-sensitive surface, and the first contact does not trigger a "click"), and after detecting liftoff of the contact in the touch input, the electronic device detects (944) a second touch input on a touch-sensitive surface (e.g., a touchpad or touchscreen capable of detecting the intensity of one or more contacts on the touchpad or touchscreen) of an input device (e.g., a remote control, a mobile phone, or a media playback device that controls a set-top box), such as in Figures 8N to 8PIn some embodiments, in response to detecting the second touch input (946), the movement of the second contact satisfies a first movement criterion, wherein the first movement criterion includes a criterion that is satisfied when the second contact has a first speed during the second touch input (e.g., the second contact has a slower speed than the first contact—if the first contact had the first speed rather than the faster second speed, the first contact would have triggered generation of a selection input): based on determining that the landing of the second contact is detected after a time threshold (e.g., 0.2 seconds, 0.5 seconds, 1 second) of the lift-off of the contact, the electronic device generates (948) a second selection input corresponding to an increase in the strength of the second contact to a corresponding strength, such as in Figure 8O In some embodiments, the electronic device forgoes generating (950) a second selection input corresponding to an increase in the intensity of the second contact to a corresponding intensity, such as in a time threshold (e.g., 0.2 seconds, 0.5 seconds, 1 second) of lift-off of the contact, based on determining that the second contact is detected within a time threshold (e.g., 0.2 seconds, 0.5 seconds, 1 second) of lift-off of the contact. Figure 8P (e.g., if a second contact is detected within a relatively short period of time after lift-off of a first contact, the increased intensity caused by the first contact required to trigger a “click” on the touch-sensitive surface is optionally maintained, and the second contact does not trigger a “click”).

[0226] In some embodiments, the movement of the contact satisfies a second movement criterion (952) (e.g., the first contact has a relatively high speed, thereby increasing the strength required to trigger a "click" on the touch-sensitive surface), and before detecting lift-off of the contact, the electronic device detects (954) a slowing of the contact from the second speed, such as at Figures 8Q to 8R In some embodiments, in response to detecting a slowing of the contact from the second speed, based on determining that movement of the contact after detecting the slowing of the contact satisfies a first movement criterion, where the first movement criterion includes a criterion that is satisfied when the contact has the first speed during the touch input, the electronic device generates (956) a selection input corresponding to an increase in the intensity of the contact to a corresponding intensity, such as in Figure 8R For example, initially, the contact optionally has a sufficiently high velocity to increase the intensity required to trigger a "click" on the touch-sensitive surface, and as a result, no "click" will be triggered on the touch-sensitive surface. However, the contact optionally slows down sufficiently to reduce the intensity required to trigger a "click", and thus triggers a "click". In some embodiments, as the required intensity decreases, it decreases all the way back to the base intensity threshold. In some embodiments, as the required intensity decreases, it decreases gradually back to the base intensity threshold (e.g., in steps over time as long as no contact is detected during that time that causes the intensity threshold to increase).

[0227] In some embodiments, the first movement criterion comprises a criterion that is satisfied when the contact has the first speed for longer than a time threshold (e.g., 0.2 seconds, 0.5 seconds, 1 second) after detecting a slowing of the contact from the second speed. In some embodiments, the contact must slow for a sufficiently long period of time before the increasing intensity threshold is reduced.

[0228] It should be understood that Figures 9A to 9G The particular order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will appreciate a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 700, 1100, 1300, 1500, 1700, and 1900) also apply in a similar manner to the details of the processes described above with respect to Figures 9A to 9G Method 900 described herein. For example, the touch-sensitive surfaces, user interface objects, tactile outputs, software remote control applications, simulated buttons, simulated remote control trackpads, and / or touch inputs described above with reference to method 900 optionally have one or more of the features of the touch-sensitive surfaces, user interface objects, tactile outputs, software remote control applications, simulated buttons, simulated remote control trackpads, and / or touch inputs described herein with reference to other methods described herein (e.g., methods 700, 1100, 1300, 1500, 1700, and 1900). For the sake of brevity, these details are not repeated here.

[0229] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A 、 Figure 3 、 Figure 5A and Figure 21 In addition, the above reference Figures 9A to 9G The described operations are optionally performed by Figure 1A to Figure 1B805. For example, the detection operation 902 and the generation operation 906 are optionally implemented by the event classifier 170, the event identifier 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 805, and the event distributor module 174 delivers the event information to the application 136-1. The corresponding event identifier 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know how to Figure 1A to Figure 1B The components depicted in the figure implement other processes.

[0230] Remote Control App User Interface

[0231] Users interact with electronic devices in a variety of different ways, including interacting with content (e.g., music, movies, etc.) available (e.g., stored or otherwise accessible) on the electronic device. In some cases, the user desires to navigate the content and / or user interfaces available on the electronic device. The embodiments described below provide for a multifunction device (such as a mobile device) in which a user can use a plurality of user interfaces that display various functions for controlling and interacting with the electronic device. Figure 5A The invention provides a method for users to interact with electronic devices by using a device 511 in the figure, thereby enhancing the interaction between the user and the electronic device. The enhanced interaction with the device reduces the amount of time required for the user to perform an operation, thereby reducing the power consumption of the device and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, this person is optionally referred to as a user of the device.

[0232] Figures 10A to 10N FIGURES illustrate exemplary ways in which a user can interact with an electronic device using a multifunction device that displays various user interfaces for controlling and interacting with the electronic device according to some embodiments of the present disclosure. The embodiments in these figures are used to illustrate the processes described below, including reference to Figures 11A to 11J Describe the process.

[0233] Figure 10AAn exemplary display 514 is shown. Display 514 optionally displays one or more user interfaces including various content. Figure 10A In the example shown, the display 514 displays a message on the electronic device (e.g., Figure 5A In some embodiments, the content application is used to display or play content (e.g., a movie, a song, a television program, a game, a menu for an application or a menu for navigating to media content, etc.), as shown in FIG. Figures 6A to 6Q and Figures 8A to 8R The content application displays a user interface 1002. The user interface 1002 includes a current focus indicator 1036 for indicating the object with current focus in the user interface 1002 (e.g., as shown in FIG. Figures 6A to 6Q The position of the current focus indicator 1036 is optionally controlled by movement input detected on a touch-sensitive surface of a remote control or other device, as will be described in more detail below. Figure 10A , the content application is playing the song "Thriller" by Michael Jackson on device 500. Providing input to the application (e.g., to control the application, to control content playback on electronic device 500, to control the position of current focus indicator 1036, etc.) is optionally accomplished by detecting a dedicated remote control (e.g., Figure 5B The touch screen 112 may be implemented by various control inputs (e.g., selection inputs, movement inputs, dedicated button inputs, etc.) on the remote control 510 in the embodiment of the present invention, such as clicks on buttons on the remote control, touch inputs on the touch-sensitive surface of the remote control (e.g., as described above with reference to method 600), or clicks on the touch-sensitive surface of the remote control (e.g., as described above with reference to method 800). However, in some embodiments, the user may desire to use a device other than a dedicated remote control to provide input to the electronic device 500; for example, a multi-function device (e.g., a mobile phone, a media playback device, or a wearable device) that is configured to operate in a manner similar to a dedicated remote control. The touch screen 112 optionally corresponds to such a device (e.g., the touch screen 112 is optionally included in a multi-function device that is configured to simulate the functions of a dedicated remote control in the electronic device 500). The device in which the touch screen 112 is included optionally corresponds to Figure 1A Device 100, Figure 2 Device 100, Figure 3 Device 300 and Figure 5AFor ease of description, actions optionally taken by a device including touch screen 112 therein (e.g., transmitting commands to electronic device 500, processing touch input, recognizing contact as specific input, tracking various characteristics of contact, etc.) will be described as being taken by touch screen 112, although it will be understood that in some embodiments, a device other than touch screen 112 takes these actions.

[0234] The touch screen 112 optionally communicates with the electronic device 500 and displays various user interfaces for controlling and interacting with the electronic device 500. Figure 10A , the touch screen 112 is displaying a remote control application user interface, which includes a remote control user interface element 1029 and a content user interface element 1028. The remote control user interface element 1029 includes a simulated dedicated remote control (e.g., Figure 5B For example, the remote control user interface element 1029 includes a corresponding Figure 5B 1024, and 1026. Selection of buttons 1016, 1018, 1020, 1022, 1024, and 1026 (e.g., via one or more taps detected on the buttons) optionally causes touch screen 112 to transmit corresponding commands to electronic device 500 to allow the electronic device to respond accordingly (e.g., as described with reference to FIG. Figures 6A to 6Q and Figures 8A to 8R described above).

[0235] The remote control user interface element 1029 also includes a touch pad area 1051. The touch pad area 1051 optionally corresponds to Figure 5B The touch-sensitive surface 451 on the remote control 510 in FIG. 4 is used to provide a tap, click, select, and / or move input to the electronic device 500 to allow the electronic device to respond accordingly (e.g., as shown in FIG. 4 ). Figures 6A to 6Q and Figures 8A to 8R For example, touch input (eg, a swipe) detected in trackpad area 1051 optionally controls the position of current focus indicator 1036 in user interface 1002 .

[0236] exist Figure 10A In addition to displaying the remote control user interface element 1029, the touch screen 112 also displays a content user interface element 1028. The content user interface element 1028 includes one or more graphical indications of content that is being played on the electronic device 500 and / or being displayed on the display 514. For example, in Figure 10A, content user interface element 1028 includes information 1034 that indicates the artist (Michael Jackson) and the song (Thriller) currently playing on electronic device 500. Content user interface element 1028 also includes a progress bar 1030 (which indicates the current playback position in Thriller) and a play / pause control 1032, both of which allow the user to control the play / pause state of Thriller (e.g., via a tap detected on play / pause control 1032) and give the user an indication of the play / pause state of Thriller (e.g., play / pause control 1032 is displayed as a pause symbol when Thriller is playing on electronic device 500 and is displayed as a play symbol when Thriller is paused on the electronic device to give the user an indication of the result of selecting play / pause control 1032 at that time). In some embodiments, the content user interface element 1028 includes different controls in addition to or instead of the play / pause control 1032 (e.g., fast forward or rewind controls for navigating content playing on the electronic device 500 are included in the content user interface element 1028 because the remote control user interface element 1029 already includes the play / pause button 1020). In some embodiments, the content user interface element 1028 is only displayed on the touch screen 112 if content is currently being played or controlled by the electronic device 500—otherwise, the content user interface element 1028 is optionally not displayed on the touch screen 112.

[0237] In some embodiments, one or more of buttons 1016, 1018, 1020, 1022, 1024, and 1026 and touchpad area 1051 in remote control user interface element 1029 are displayed only when electronic device 500 can be controlled by buttons or touchpad area. Figure 10A In the embodiment of the present invention, the electronic device 500 is optionally able to control the volume of the content being played on the electronic device (for example, the electronic device 500 is connected to one or more speakers so as to allow the electronic device to control the volume level of those speakers that are playing audio from the content being played by the electronic device). Figure 10A The remote control user interface elements 1029 in FIG include volume buttons 1022 and volume buttons 1024. In contrast, in Figure 10B In the embodiment of the present invention, the electronic device 500 is optionally unable to control the volume of the content being played on the electronic device. Figure 10B1024. The remote control user interface element 1029 in FIG. 1025 does not include volume button 1022 and volume button 1024. Conditional display of other controls in remote control user interface element 1029 is similarly contemplated. In some embodiments, certain controls in remote control user interface element 1029 are displayed regardless of the type of content being played on electronic device 500 and / or the configuration of the electronic device. For example, remote control user interface element 1029 optionally always includes menu button 1016 or trackpad area 1051, regardless of any configuration of electronic device 500.

[0238] Figure 10C 1051. In particular, contact 1007 and movement of contact 1007 have been detected in touchpad area 1051. In response, current focus indicator 1036 is moved in user interface 1002 in accordance with the detected movement of contact 1007 in touchpad area 1051 (e.g., similar to movement detected on touch-sensitive surface 451 of remote control 510, as described with reference to FIG. Figure 5B In addition, as Figure 10C As shown, in some embodiments, input provided to remote control user interface element 1029 (e.g., contact 1007 detected in touchpad area 1051) is detected while the display of remote control user interface element 1029 and content user interface element 1028 is maintained on touch screen 112 (e.g., if the input selects a control in remote control user interface element 1029, selection of the control causes a corresponding operation to occur without changing the placement and / or size of remote control user interface element 1029 and content user interface element 1028 on touch screen 112).

[0239] Figure 10D 1020 , the play state of the content being played on the electronic device 500 is shown being controlled in response to a touch input detected on the play / pause button 1020. Specifically, contact 1007 (e.g., a tap) has been detected on the play / pause button 1020. In response, "Thriller" has been paused on the electronic device 500 (indicated by the pause symbol in the user interface 1002 on the display 514). Additionally, the content user interface element 1028 is updated to reflect the changed state of the content being played on the electronic device 500. Specifically, the play / pause control 1032 in the content user interface element 1028 is updated from the pause symbol (e.g., as indicated in the Figure 10C ) changes to a play symbol (e.g., as in Figure 10D), indicating that selection of the play / pause control 1032 will cause "Thriller" to begin playing again on the electronic device 500. Figure 10C , input detected at the play / pause button 1020 is detected while display of the remote control user interface element 1029 and the content user interface element 1028 is maintained on the touch screen 112.

[0240] Figure 10E A change in the content being played on the electronic device 500 is shown, along with a corresponding update to the content user interface element 1028. Specifically, the electronic device 500 has changed from playing "Thriller" by Michael Jackson to playing "Longview" by Green Day (e.g., via one or more appropriate inputs detected in the remote control user interface element 1029), as shown in the user interface 1002 on the display 514. As a result, the information 1034 in the content user interface element 1028 has been updated to indicate that the currently playing content on the electronic device is "Longview" by Green Day, and the progress bar 1030 has been updated to indicate the current playback position in "Longview". Furthermore, in some embodiments, the configuration of the remote control user interface element 1029 is independent of the content being played on the electronic device. Thus, although the electronic device 500 has changed from playing "Thriller" to playing "Longview", Figure 10E The configuration of the remote control user interface element 1029 in (corresponding to the playback of "Longview") is the same as Figure 10D The configuration of remote control user interface element 1029 in (corresponding to the playback of "Thriller") is the same.

[0241] In some embodiments, a touch input detected in the content user interface element 1028 either keeps the content user interface element displayed or expands the content user interface element depending on where the touch input is detected. FIG. 10F to FIG. 10I Specifically, Figure 10F , contact 1007 (e.g., a tap) has been detected on the play / pause control 1032 in the content user interface element 1028. As a result, Figure 10G , “Longview” has been paused on the electronic device 500 (as indicated in the user interface 1002 on the display 514), while the placement and / or size of the remote control user interface element 1029 and the content user interface element 1028 on the touch screen 112 are maintained.

[0242] On the contrary, Figure 10H, contact 1007 (e.g., a tap) has been detected on an area of ​​content user interface element 1028 outside of play / pause control 1032. Figure 10I , the expanded content user interface element 1038 is displayed on the touch screen 112. In some embodiments, the expanded content user interface element 1038 replaces the remote control user interface element 1029 and the content user interface element 1028 on the touch screen 112, such as Figure 10I As shown. Figure 10H The expanded content user interface element 1038 optionally includes additional controls and / or information compared to the expanded content user interface element 1028. For example, Figure 10I , the expanded content user interface element 1038 includes album artwork 1044 associated with the content playing on the electronic device 500 (e.g., GreenDay's "Longview"), a scrubber bar 1046 that both displays an indication of the current playback position in the content playing on the electronic device 500 and allows the user to scrub through the content (e.g., via left / right swipes detected on the scrubber bar 1046), and information 1034 about the artist associated with the title of the content playing on the electronic device 500. The expanded content user interface element 1038 also includes a play / pause control 1032, skip forward and skip back controls 1042 for skipping forward and backward through the content playing on the electronic device 500, and a favorites button 1048 for adding the content playing on the electronic device 500 to the user's favorites list. Additionally, the expanded content user interface element 1038 includes a volume control 1040 for controlling the volume of the content playing on the electronic device 500 (e.g., via left / right swipes detected on the volume control 1040). Finally, in Figure 10I In the embodiment of the present invention, the expanded content user interface element includes a return element 1042 for closing the expanded content user interface element 1038 and returning to the Figure 10H Display of content user interface element 1028 and remote control user interface element 1029.

[0243] In some embodiments, the expanded content user interface element 1038 is customized relative to the content being played by the electronic device 500. For example, the expanded content user interface element 1038 optionally includes customized information such as album art corresponding to the content being played on the electronic device 500, and / or customized controls specific to the content currently being played on the display 514 by the electronic device 500 (e.g., a skip forward button for skipping to the next track if the content being played is a song in a playlist, or a fast forward button for fast-forwarding through the content if the content being played is a movie). Figure 10NA device 500 is shown playing a movie (e.g., Braveheart) instead of music (e.g., Figure 10I implementation plan of the . Figure 10N The expanded content user interface element 1038 in the optionally includes a previous chapter / next chapter control 1043 for jumping to the previous or next chapter in the movie, as in the Figure 10I The skip forward and skip back controls 1042 for skipping forward and backward through songs are completely different.

[0244] In some embodiments, the electronic device 500 is capable of running one or more games. In such cases, the touch screen 112 optionally displays various user interfaces for interacting with the games, such as Figures 10J to 10N Specifically, Figure 10J , the touch screen 112 is displaying the content user interface element 1028 and the remote control user interface element 1029, and the electronic device 500 is optionally playing "Thriller" by Michael Jackson, as shown in FIG. Figure 10A In addition, the electronic device 500 is optionally running Game A, as indicated by the user interface 1002. As a result, the touch screen 112 displays a game controller launch user interface element 1050 for displaying a game controller user interface element on the touch screen 112, as will be described in more detail below. In some embodiments, the game controller launch user interface element 1050 is only displayed on the touch screen 112 if a game is running on the electronic device 500 and / or the game running on the electronic device 500 supports game controller input.

[0245] exist Figure 10K , contact 1007 (e.g., a tap) has been detected on game controller launch user interface element 1050. In response, touch screen 112 stops displaying remote control user interface element 1029 and content user interface element 1028 (e.g., touch screen 112 is in portrait orientation mode) and displays game controller user interface element 1051 (e.g., touch screen 112 is in landscape orientation mode), as shown in FIG. Figure 10L As shown. The game controller user interface element 1051 optionally includes controls and / or information related to playing a game on the electronic device 500. For example, Figure 10L , game controller user interface element 1051 includes a trackpad area 1052 for providing directional input to game A (e.g., with the user's left thumb), and buttons 1054-1, 1054-2, 1054-3, and 1054-4 for providing button input to game A (e.g., with the user's right thumb).

[0246] The touch screen 112 also displays a remote control user interface element 1064 that includes a simulated dedicated remote control (e.g., Figure 5B Various controls on the remote control 510) in the example, similar to Figure 10A However, in some embodiments, the remote control user interface element 1064 includes the remote control user interface element 1029. Figure 10A In the remote control user interface element 1029, different controls and / or controls with different appearances are displayed. Figure 10L , the remote control user interface elements 1064 include a voice assistant button 1058, a menu button 1060, and a play / pause button 1062 (currently showing "pause" because the content on the electronic device 500 is currently playing). The remote control user interface elements 1064 do not include, for example, Figure 10A In addition, the voice assistant button 1058, the menu button 1060 and the play / pause button 1062 in the remote control user interface element 1051 have the same Figure 10A The corresponding buttons in the remote control user interface element 1029 have a different appearance and are displayed in a different arrangement.

[0247] In some embodiments, the game controls included in the game controller user interface element 1051 and / or the configuration of the game controller user interface element 1051 (e.g., the placement of the controls) are game dependent. For example, the game controls in the game controller user interface element 1051 are optionally customized based on the game running on the electronic device 500. As previously described, in Figure 10L In FIG. 5 , the electronic device 500 is running game A, as indicated by the user interface 1002, and the game controller user interface element 1051 has the above-mentioned Figure 10L The configuration shown in . Figure 10M In the example, the electronic device is running game B, as indicated by user interface 1002. As a result, Figure 10M The game controller user interface element 1051 in Figure 10L The game controller user interface elements 1051 are configured differently. Specifically, Figure 10M The game controller user interface element 1051 in FIG. 1 (corresponding to game B) has buttons 1054-5 and 1054-6, while Figure 10LThe game controller user interface element 1051 in FIG. 5 (corresponding to game A) has buttons 1054-1, 1054-2, 1054-3, and 1054-4 arranged differently from buttons 1054-5 and 1054-6. The configuration of the game controller user interface element 1051 can similarly vary in other ways based on the game currently running on the electronic device 500, depending on the characteristics or requirements of the game.

[0248] Figures 11A to 11J 1 is a flow chart illustrating a method 1100 for interacting with an electronic device using a multifunction device that displays various user interfaces for controlling and interacting with the electronic device according to some embodiments of the present disclosure. The method 1100 is optionally performed on an electronic device such as device 100, device 300, or device 500 (described above with reference to FIG. Figure 1A to Figure 1B 、 Figures 2 to 3 and Figures 5A to 5B Some operations in method 1100 may be optionally combined, and / or the order of some operations may be optionally changed.

[0249] As described below, method 1100 provides a method for interacting with an electronic device using a multifunction device that displays various user interfaces for controlling and interacting with the electronic device. This method reduces the cognitive burden on a user when interacting with the device user interface of the present disclosure, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charges.

[0250] In some embodiments, a first electronic device (e.g., a remote control, a mobile phone, a media playback device, or a watch that controls a set-top box, such as device 100, device 300, or device 500) having a display and one or more input devices (e.g., a touch-sensitive surface or touch screen) simultaneously displays (1102) on the display: a remote control user interface element (1104) that includes a first set of controls for navigating a user interface displayed on a remote display (e.g., a television) that is controlled by a second electronic device (e.g., a set-top box connected to the television) that is different from the first electronic device (e.g., displaying virtual input elements such as virtual buttons or motion tracking areas that correspond to physical controls such as buttons on a physical remote control or a touch-sensitive surface that is dedicated to controlling the second electronic device; and a content user interface element (1106) that includes a graphical representation of content (e.g., a movie, a television show, a song, etc.) being played by the second electronic device on the remote display, such as a video clip on the screen. Figure 10AIn some embodiments, when the remote control user interface element and the content user interface element are displayed simultaneously on the display, the electronic device receives (1108) input (e.g., touch input, such as a tap or swipe input) at the first electronic device via one or more input devices, and in response to receiving the input, based on determining that the input is received at a corresponding control in the first group of controls (e.g., a play / pause button, a menu button, a back button, etc.), the electronic device initiates (1110) an operation to navigate the user interface displayed by the second electronic device on the remote display based on the input received at the corresponding control (e.g., by transmitting a corresponding command from the first electronic device to the second electronic device), such as in Figure 10C For example, in response to receiving the input, navigating a menu displayed by the second electronic device, changing a user interface object having a current focus among a stack of user interface objects displayed by the second electronic device, etc.

[0251] In some embodiments, in response to receiving the input (1112), based on determining that the input corresponds to a request to change the state of the content being played by the second electronic device (e.g., skipping to the next song, playing or pausing the currently playing content, skipping to the next episode in a TV series, etc.), the electronic device initiates (1114) an operation to change the state of the content being played by the second electronic device based on the input (e.g., transmitting a command from the first electronic device to the second electronic device to implement the state change requested by the input), and the electronic device updates (1116) a content user interface element to reflect the change in the state of the content being played by the second electronic device, such as in Figure 10D For example, if the input causes a new song to play on the second electronic device, the content user interface element is updated to include the title of the most recently played song, such as in Figure 10E If the input pauses the currently playing content on the second electronic device, the content user interface element is updated to indicate that the content is currently paused, rather than currently playing, such as in Figure 10D medium.

[0252] In some embodiments, the configuration of the remote control user interface element (e.g., the appearance of the remote control user interface element, the controls included in the remote control user interface element, the size of the controls included in the remote control user interface element, etc.) is independent of the content being played by the second electronic device (1118) on the remote display (e.g., the same set of controls is displayed in the remote control user interface element regardless of what content is currently being played by the second electronic device on the remote display device). In some embodiments, if the content being played by the second electronic device changes, the set of controls in the remote control user interface will remain unchanged, such as in Figures 10D to 10E middle.

[0253] In some embodiments, the content user interface elements include (1120) a second set of one or more controls for navigating the content being played by the second electronic device on the remote display (e.g., a play / pause button, a skip forward button, a skip backward button, a scrubber bar that can be scrubbed back and forth to control the current playback position in the content, etc.), such as in Figure 10A In some embodiments, in response to receiving the input (1122), based on determining that the input corresponds to a selection of a corresponding control in the second set of controls in the content user interface element (e.g., a tap on one of the controls in the content user interface element, such as a play / pause button), the electronic device initiates (1124) an operation to control playback of the content being played by the second electronic device on the remote display while maintaining simultaneous display of the remote control user interface element and the content user interface element, such as in Figures 10F to 10G In some embodiments, in response to receiving the input (1122), based on determining that the input corresponds to a selection of a content user interface element other than the one or more controls in the second group of controls (e.g., a tap or swipe in a content user interface element that does not correspond to one of the controls in the content user interface element), the electronic device displays (1126) an expanded content user interface element including the second group of controls and a third group of controls for navigating the content being played by the second electronic device, such as in Figures 10H to 10IFor example, if the input corresponds to an area of ​​the content user interface element that does not include controls, the input causes display of an expanded content user interface element on the display, the expanded content user interface element including additional controls and / or information for navigating the content being played by the second electronic device. In some embodiments, display of the expanded content user interface element is optionally triggered by swiping from the content user interface element away from an edge of the touch-sensitive display (e.g., toward a center area of ​​the touch-sensitive display).

[0254] In some embodiments, the expanded content user interface element is customized relative to the content being played by the second electronic device (1128) (e.g., includes information such as album art corresponding to the content being played on the second electronic device, and / or controls specific to the content currently being played by the second electronic device on the remote display), such as in Figure 10I For example, if the content being played is a song in a playlist, the expanded content user interface element optionally includes a skip forward button for skipping to the next track, and if the content being played is a movie, the expanded content user interface element optionally includes a fast forward button for fast forwarding through the content. In some embodiments, the expanded content user interface element includes (1130) information about the content being played by the second electronic device that was not displayed on the display before receiving the input, such as Figure 10I (e.g., the expanded content user interface element includes album art, content duration, content name, or other content metadata that was not included in the content user interface element or anywhere else on the display prior to receiving the input).

[0255] In some embodiments, the content user interface element includes (1132) a first set of information about the content being played by the second electronic device (e.g., the title of the content and an artist associated with the content), and the expanded content user interface element includes the first set of information and a second set of information about the content being played by the second electronic device, such as in Figure 10I In some embodiments, the first set of information and the second set of information include (1134) one or more of a category of content being played by the second electronic device, a title of the content being played by the second electronic device, an image of the content being played by the second electronic device, and an artist associated with the content being played by the second electronic device.

[0256] In some embodiments, displaying the expanded content user interface element includes ceasing (1136) display of the remote control user interface element on the display, such as in Figure 10I In some embodiments, the second set of controls and the third set of controls (e.g., the content user interface element and the content navigation controls in the expanded content user interface element) include (1138) one or more of a play / pause button, a back skip button, a forward skip button, a scrub bar, a progress bar, a volume control for controlling the volume of the second electronic device, and a favorite button for designating the content being played by the second electronic device as favorite content, such as in Figure 10I middle.

[0257] In some embodiments, initiating an operation to navigate a user interface displayed by the second electronic device based on input received at a corresponding control (e.g., selection of a control in a remote control user interface element) includes maintaining (1140) display of the remote control user interface element and the content user interface element on the display, such as in FIG. 10C to FIG. 10D For example, if the input selects a control in a remote control user interface element, selection of the control causes a corresponding operation to occur without changing the placement and / or size of the remote control user interface element and the content user interface element on the display.

[0258] In some embodiments, in response to receiving the input, based on determining that the input is received at the content user interface element and corresponds to a request for controlling the playback state of the content being played by the second electronic device (e.g., selection of a control in the content user interface element such as a play / pause button), the electronic device initiates (1142) an operation to control the playback state of the content being played by the second electronic device based on the received input, while maintaining display of the remote control user interface element and the content user interface element on the display, such as in Figure 10F For example, if the input selects a control in the content user interface element, selection of the control causes the corresponding operation to occur without changing the placement and / or size of the remote control user interface element and the content user interface element on the display.

[0259] In some embodiments, the first set of controls (e.g., controls in a remote control user interface element) includes (1144) a touchpad area (e.g., for detecting touch input, such as a tap, swipe, click, etc., corresponding to the reference Figure 5B ), one or more of a menu button, a home button, a virtual assistant button, a play / pause button, and a volume control (e.g., corresponding to the reference Figure 5B dedicated remote control buttons as described in Figure 10A middle.

[0260] In some embodiments, based on determining that the second electronic device is configured to adjust the volume level of content being played by the second electronic device (e.g., the second electronic device is connected to one or more speakers so as to allow the second electronic device to control the volume level of those speakers that are playing audio from the content being played by the second electronic device), the first set of controls includes (1146) a volume control, such as Figure 10A In, and based on determining that the second electronic device is not configured to adjust the volume level of the content being played by the second electronic device, the first set of controls does not include (1148) a volume control, such as in Figure 10B For example, if the first electronic device is capable of controlling the volume level of content being played by the second electronic device via the second electronic device, the remote control user interface element only includes a volume control.

[0261] In some embodiments, at least one control from the first set of controls (e.g., controls in a remote control user interface element) is included (1150) in the remote control user interface independent of the context of the second electronic device (e.g., independent of the type of content being played on the second electronic device, independent of the configuration of the second electronic device, etc.) For example, the remote control user interface element optionally always includes a menu button regardless of any configuration of the second electronic device.

[0262] In some embodiments, displaying a content user interface element includes (1152): based on determining that the content is being played by the second electronic device, displaying (1154) a content user interface element on the display, the content user interface element including a graphical representation of the content being played by the second electronic device, such as Figure 10A and, based on determining that the content is not being played by the second electronic device, foregoing displaying (1156) the content user interface element on the display (e.g., the content user interface element is only displayed on the display if the content, such as a song or movie, is being played on the second electronic device).

[0263] In some embodiments, the first electronic device is a portable electronic device and the second electronic device is a set-top box (1158) connected to a remote display. In some embodiments, the first electronic device includes a mobile phone, a media player, or a wearable device (1160) (e.g., a smart watch).

[0264] In some embodiments, when the remote control user interface element and the content user interface element are displayed on the display simultaneously, the electronic device displays (1162) a game controller launch user interface element on the display (e.g., a user interface element for displaying a game controller user interface element on the display), such as in Figure 10JIn some embodiments, the game controller launch user interface element is displayed when the game application is available for play using the remote display (e.g., when a user interface for the game application is displayed on the remote display), and the game controller launch user interface element is not displayed when the game application is not available for play using the remote display. The electronic device optionally receives (1164) a second input corresponding to a selection of the game controller launch user interface element (e.g., a tap on the game controller launch user interface element) via the one or more input devices, and in response to receiving the second input, displays (1166) a game controller user interface element on the display (e.g., a user interface element that includes controls and / or information related to playing the game on the second electronic device), such as on Figures 10K to 10M For example, the game controller user interface element optionally includes a directional input control (such as a directional pad or a trackpad) and / or one or more buttons for providing input to a game running on the second electronic device, such as in Figure 10L middle.

[0265] In some embodiments, based on determining that the game is running on the second electronic device, the electronic device displays (1168) a game controller launch user interface element on the remote display, and based on determining that the game is not running on the second electronic device, the electronic device forgoes displaying (1170) the game controller launch user interface element on the remote display (e.g., the game controller launch user interface element is optionally displayed only when the game is running on the second electronic device and / or when a game that supports a game controller is running on the second electronic device).

[0266] In some embodiments, displaying the game controller user interface element includes stopping (1172) display of the remote control user interface element and / or the content user interface element on the display, such as in Figure 10L For example, when the game controller user interface element is displayed via selection of the game controller launch user interface element, the remote control user interface element and / or the content user interface element are optionally no longer displayed on the display. In some embodiments, the game controller user interface element includes (1174) a corresponding set of one or more controls for controlling a corresponding game running on the second electronic device, such as Figure 10L For example, the game controller user interface element optionally includes a directional input control (such as a directional pad or a trackpad) and / or one or more buttons for providing input to a game running on the second electronic device. In some embodiments, the corresponding set of controls includes (1180) one or more of a directional control and a button input.

[0267] In some embodiments, based on determining that the corresponding game running on the second electronic device is the first game, the corresponding set of controls (1176) is the first set of game controls (e.g., a touchpad and two input buttons), such as in Figure 10L and based on determining that the corresponding game running on the second electronic device is a second game different from the first game, the corresponding set of controls (1178) is a second set of game controls (e.g., a touchpad and three input buttons) different from the first set of game controls, such as in Figure 10M Thus, in some embodiments, the controls in the game controller user interface element are customized based on the game running on the second electronic device.

[0268] In some embodiments, in response to receiving a second input corresponding to a selection of a game controller launch user interface element (e.g., a user interface element for displaying a game controller user interface element on a display), the electronic device simultaneously displays (1182) on the display a game controller user interface element (1184) (e.g., a user interface element including controls and / or information related to playing a game on a second electronic device) and a second remote control user interface element (1186) that is different from the remote control user interface element, the second remote control user interface element including a second set of controls for simulating a remote control for navigating a user interface displayed on a remote display controlled by the second electronic device, such as on a display. Figure 10L For example, when a game controller user interface element is displayed on the display, a second remote control user interface element is displayed on the display that is different from the remote control user interface element displayed with the content user interface element. In some embodiments, the second remote control user interface element includes different controls and / or controls with a different appearance than the remote control user interface element, such as in Figure 10L middle.

[0269] In some embodiments, the second set of controls (1188) of the simulated remote control in the second remote control user interface element is a subset of the first set of controls of the simulated remote control in the remote control user interface element, such as in Figure 10L In some embodiments, a first set of controls in the remote control user interface element is displayed on the display in a first configuration, and a second set of controls in the second remote control user interface element is displayed on the display in a second configuration (e.g., a different spatial arrangement, size, appearance (e.g., specified by the currently playing application)) that is different from the first configuration (1190), such as in Figure 10L middle.

[0270] In some embodiments, the remote control user interface element and the content user interface element are displayed (1192) on the display in a first orientation mode, such as in Figure 10K The remote control user interface elements and the content user interface are displayed with the display in portrait mode, and the game controller user interface elements are displayed (1194) on the display in a second orientation mode that is different from the first orientation mode, such as in Figures 10L to 10M (e.g., when displaying game controller user interface elements, the display switches to landscape mode).

[0271] It should be understood that Figures 11A to 11J The particular order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will appreciate a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 700, 900, 1300, 1500, 1700, and 1900) also apply in a similar manner to the details of the processes described above with respect to Figures 11A to 11J For example, the touch input, software remote control application, simulated button, and / or simulated remote control trackpad described above with reference to method 1100 optionally have one or more of the features of the touch input, software remote control application, simulated button, and / or simulated remote control trackpad described herein with reference to other methods described herein (e.g., methods 700, 900, 1300, 1500, 1700, and 1900). For the sake of brevity, these details are not repeated here.

[0272] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A 、 Figure 3 、 Figure 5A and Figure 22 In addition, the above reference Figures 11A to 11J The described operations are optionally performed by Figure 1A to Figure 1B110 is implemented by the components depicted in . For example, the display operation 1102, the receiving operation 1108, and the initiating operation 1110 are optionally implemented by the event classifier 170, the event identifier 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects contact on the touch screen 112, and the event distributor module 174 delivers the event information to the application 136-1. The corresponding event identifier 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch screen corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know how to update the content displayed by the application based on the GUI updater 178. Figure 1A to Figure 1B The components depicted in the figure implement other processes.

[0273] Text input alert

[0274] Users interact with electronic devices in a variety of different ways, including interacting with content (e.g., music, movies, etc.) available on the electronic device (e.g., stored or otherwise accessible). In some cases, a user can interact with an electronic device by using a multifunction device to provide text input to the electronic device. The embodiments described below provide a way in which the need for text input to the electronic device is indicated on the multifunction device, thereby enhancing the interaction between the user and the electronic device. Enhancing interaction with the device reduces the amount of time required for the user to perform an operation, thereby reducing the power consumption of the device and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, this person is optionally referred to as a user of the device.

[0275] Figures 12A to 12RR FIGURES illustrate exemplary ways in which a need for text input to an electronic device is indicated on a multifunction device according to some embodiments of the present disclosure. The embodiments in these figures are used to illustrate the processes described below, including reference to Figures 13A to 13K Describe the process.

[0276] Figure 12A An exemplary display 514 is shown. Display 514 optionally displays one or more user interfaces including various content. Figure 12A In the example shown, the display 514 displays a message on the electronic device (e.g., Figure 5A 514 ), to which the display 514 is a part or to which the display 514 is connected. The text input user interface 1202 is optionally a user interface for searching for content available for viewing on the electronic device 500, although the text input user interface 1202 is optionally any user interface in which text can be entered. The text input user interface 1202 optionally includes a text input field 1228 and user interface objects 1230, 1232, 1234, and 1236 that can be selected to display respective corresponding content on the display 514. The text input user interface 1202 also has a current focus—indicates which object in the text input user interface 1202 is currently selected— Figure 12A , user interface object 1230 has the current focus, as indicated by the dashed box within user interface object 1230.

[0277] As reference Figures 5A to 5B As described above, a remote control 510 and / or device 511 are optionally used to control the electronic device 500. Specifically, the remote control 510 and device 511 optionally communicate with the electronic device 500 and provide input to the electronic device 500. The remote control 510 optionally has a reference Figure 5B Features described for providing input to electronic device 500. For example, selection of one or more of buttons 516, 518, 520, 522, 524, and 526 optionally causes remote control 510 to transmit corresponding commands to electronic device 500, to which electronic device 500 responds accordingly. Touch-sensitive surface 451 is optionally used to provide tap, click, select, and / or move input to electronic device 500, to which electronic device 500 responds accordingly. For example, touch input (e.g., a swipe) detected on touch-sensitive surface 451 optionally controls the location of the current focus in user interface 1202.

[0278] Device 511 is optionally a multi-function device. In some embodiments, device 511 is a mobile phone configured to run applications and perform multiple functions independent of controlling electronic device 500 (such as phone functions, messaging functions, etc.). In some embodiments, device 511 runs a remote control application that configures device 511 to operate as a remote control for electronic device 500. Figure 12A In FIG, device 511 is running such a remote control application, which causes device 511 to display a remote control user interface including various controls for controlling controls on a simulated dedicated remote control (e.g., remote control 510) of electronic device 500. For example, the remote control user interface includes controls corresponding to reference Figure 5B12. The remote control user interface also includes buttons 1216, 1218, 1220, 1222, 1224, and 1226 of the buttons depicted in FIG. Selection of one or more of buttons 1216, 1218, 1220, 1222, 1224, and 1226 (e.g., via one or more taps detected on the buttons) optionally causes device 511 to transmit corresponding commands to electronic device 500, to which electronic device 500 responds accordingly. The remote control user interface also includes a touchpad area 1251. Touchpad area 1251 optionally corresponds to Figure 5B 12 and 13. The touch-sensitive surface 451 on the remote control 510 in FIG. 12 is used to provide tap, click, selection, and / or movement input to the electronic device 500, to which the electronic device 500 responds accordingly. For example, touch input (e.g., a swipe) detected in the trackpad area 1251 optionally controls the location of the current focus in the user interface 1202.

[0279] As described above, in addition to running the remote control application, device 511 is configured to run other applications and perform a number of other functions independent of controlling electronic device 500 (such as phone functions, messaging functions, etc.). In such cases, device 511 optionally displays a user interface that is not the user interface of the remote control application. For example, in Figure 12B , device 511 is in a locked state and, therefore, is displaying lock screen 1240. In other words, lock screen 1240 is, optionally, the user interface of the operating system of device 511 (not the user interface of the remote control application), and is, optionally, displayed by device 511 when device 511 is in a locked state. In some embodiments, user input on lock screen 1240 is limited to selection of an alert displayed on lock screen 1240 (e.g., a text input alert, an incoming email alert, an incoming call alert, an incoming text message alert, etc.) or entry of authentication information for unlocking device 511. In some embodiments, the text input alert of the present disclosure is displayed on device 511 even when the device does not have the remote control application installed on the device.

[0280] The text input user interface 1202 is optionally a user interface into which text can be input, as previously described. In some embodiments, when the electronic device 500 determines that text input is required for the text input user interface 1202, the electronic device 500 transmits an indication of such a need to the device 511, which receives the indication, making the device 511 aware of the need for text input for the text input user interface 1202. The device 511 in turn responds accordingly, as will be described below.

[0281] Figure 12C1240 is displayed. In response to the swipe of contact 1203, the current focus in text input user interface 1202 moves from user interface element 1230 to text input field 1228 according to the swipe. Figure 12D , a selection input (indicated by contact 1203) is detected on touch-sensitive surface 451 of remote control 510 while text input field 1228 has current focus. In response to the selection input, as shown Figure 12E As shown, the electronic device 500 optionally enters a text input mode, a soft keyboard 1238 is displayed in the text input user interface 1202, and the current focus moves to one of the keys in the soft keyboard 1238 (eg, Figure 12E ). Soft keyboard 1238 optionally includes one or more keys that correspond to text, and selection of a key using remote control 510 and / or device 511 causes the corresponding text to be entered into text entry field 1228. For example, a swipe input detected on touch-sensitive surface 451 optionally causes the current focus in text entry user interface 1202 to move from key to key in soft keyboard 1238, and a selection input detected on touch-sensitive surface 451 optionally causes text corresponding to the key with the current focus to be entered into text entry field 1228.

[0282] Also in response to electronic device 500 entering text input mode and displaying soft keyboard 1238, electronic device 500 optionally transmits an indication to device 511 when device 511 is displaying a user interface that is not the user interface of the remote control application (e.g., lock screen 1240) that text input is required for user interface 1202. In response to receiving the indication, device 511 displays text input alert 1242 on lock screen 1240, as shown in FIG. Figure 12E As shown. Text input alert 1242 optionally covers / replaces a portion of lock screen 1240 and indicates to the user of device 511 that text input can be entered from device 511 into user interface 1202, as will be described in more detail below. Finally, electronic device 500 also optionally displays a visual indication 1250 in text input user interface 1202 that text can be entered into text input user interface 1202 using device 511, so that a user looking at display 514 knows that such text input methods are available to him.

[0283] exist 12D to 12E, a selection input detected on touch-sensitive surface 451 when text input field 1228 has current focus causes electronic device 500 to transmit an indication of a need for text input for text input user interface 1202 to device 511. In some embodiments, electronic device 500 does not transmit the indication until the user moves current focus to soft keyboard 1238. For example, in Figure 12F , soft keyboard 1238 is displayed in text input user interface 1202, and text input field 1228 has current focus (e.g., Figure 12F Optionally by Figure 12D The electronic device 500 has not yet transmitted an indication of the need for text input to the device 511, and therefore, the device 511 does not display a text input alert on the lock screen 1240. Figure 12G In FIG, a swipe of contact 1203 to the lower left is detected on touch-sensitive surface 451. In response to the swipe, the current focus moves from text input field 1228 to the "A" key in soft keyboard 1238 according to the swipe. As a result, electronic device 500 displays indication 1250 in text input user interface 1202 and transmits an indication of the need for text input to device 511, and device 511 displays text input alert 1242 on the lock screen in response to receiving the indication, as shown in FIG. Figure 12G shown.

[0284] In some embodiments, when prompting for text input on device 511, a soft keyboard is not displayed in text input user interface 1202. For example, Figure 12H , text input user interface 1202 does not include a soft keyboard. A selection input (indicated by contact 1203) is detected on touch-sensitive surface 451 of remote control 510 while text input field 1228 has current focus. In response, electronic device 500 transmits an indication of the need for text input to device 511, and device 511 displays text input alert 1242 on the lock screen in response to receiving the indication. Even after a selection input is detected on touch-sensitive surface 451, electronic device 500 optionally does not display a soft keyboard in text input user interface 1202, and uses device 511 to enter text into text input field 1228, as will be described below.

[0285] Now refer to Figures 12I to 12M 1242 and provide text input to the text input user interface 1202. Figure 12I , a text input alert 1242 is displayed on the lock screen 1240, as shown in FIG. Figure 12EIn some embodiments, the text input alert 1242 can be selected from the lock screen 1240 by swiping right on the text input alert 1242. For example, Figure 12J , contact 1203 on text input alert 1242 is swiping text input alert 1242 to the right on lock screen 1240. In response to swiping right on text input alert 1242, device 511 displays the following Figure 12K User interface 1244 is shown, which optionally includes a soft keyboard 1246 and a text field 1248. Text field 1248 optionally mirrors the contents of text input field 1228 in text input user interface 1202. User interface 1244 is optionally the user interface of the operating system of device 511, rather than a reference to a user interface. Figure 12A Input detected on user interface 1244 optionally causes device 511 to provide text input to electronic device 500 for entry into text input user interface 1202. For example, Figure 12L , contact 1203 has been detected on the “M” key in soft keyboard 1246. In response to detecting contact 1203 on the “M” key, device 511 transmits information corresponding to the “M” key to electronic device 500, which in response updates text input field 1228 to include “M”. Device 511 optionally updates text field 1248 to reflect that text input field 1228 includes “M”. Figure 12M , additional text input has been detected on soft keyboard 1246. Specifically, contact 1203 has been detected on the "U" key. In response, device 511 transmits information corresponding to the "U" key to electronic device 500, which in response updates text input field 1228 to include "Mu." Device 511 optionally updates text field 1248 to reflect that text input field 1228 includes "Mu." Additional text input is optionally entered into text input user interface 1202 using device 511 in a similar manner.

[0286] In some embodiments, while the text input alert 1242 is being displayed on the device 511, the remote control 510 may be used to provide text input to the text input user interface 1202, as will be described with reference to FIG. Figures 12N to 12Q Specifically, Figure 12N , a text input alert 1242 is displayed on the lock screen 1240, as shown in FIG. Figure 12E As described above, the "A" key in the soft keyboard 1238 has the current focus. Figure 12O, while device 511 is displaying text entry alert 1242, and while the "A" key in soft keyboard 1238 has current focus, a selection input is detected on touch-sensitive surface 451, as indicated by contact 1203. In response, electronic device 500 enters "A" into text entry field 1228. Figure 12P In FIG, a swipe of contact 1203 to the lower right is detected on touch-sensitive surface 451. In response to the swipe, the current focus moves from the "A" key to the "J" key in soft keyboard 1238 according to the swipe. Figure 12Q , a selection input is detected on touch-sensitive surface 451 while the "J" key in soft keyboard 1238 has current focus, as indicated by contact 1203. In response, electronic device 500 enters "j" into text input field 1228. Thus, as shown above, text can be entered into text input user interface 1202 using remote control 510 even after text input alert 1242 is displayed on device 511.

[0287] In some embodiments, in response to displaying and / or receiving indications corresponding to various alerts, device 511 provides some kind of notification (e.g., a vibration notification, an audible notification, a visual notification, etc.). In addition, when it displays a text input alert, device 511 optionally generates a different type of notification than when it displays other types of alerts (e.g., an email alert, a text message alert, a voicemail alert, etc.). For example, in Figure 12R , device 511 has received an indication of a need for text input in text input user interface 1202. In response, device 511 displays text input alert 1242 on lock screen 1240 and also generates a first type of notification (e.g., Notification A) corresponding to text input alert 1242. In other words, device 511 is optionally configured to generate one type of notification (e.g., only a vibration, or only a visual indication) when it displays a text input alert such as text input alert 1242. Figure 12S, while displaying text input alert 1242, device 511 has determined that John Smith has sent a new email message to device 511 (or a user associated with device 511). In response, in addition to displaying text input alert 1242 on lock screen 1240, device 511 also displays email alert 1252. When device 511 displays email alert 1252, device 511 generates a second type of notification (e.g., Notification B) corresponding to email alert 1252. In other words, device 511 is optionally configured to generate a different type of notification (e.g., vibrate and be visible, or vibrate and ring) when it displays an alert other than a text input alert (e.g., an email alert, a text message alert, a voicemail alert, etc.), such as email alert 1252. This allows the user of device 511 to discern whether a given alert is a text input alert or a different type of alert without looking at device 511.

[0288] In some embodiments, in addition to generating different notifications for text input alerts and other alerts, device 511 also treats text input alerts differently from other alerts in other ways. Specifically, text input alerts are optionally more "persistent" than other types of alerts, as described in detail in the following sections. Figures 12S to 12V As described above, Figure 12S , device 511 is displaying text input alert 1242 and email alert 1252 simultaneously on lock screen 1240. When lock screen 1240 is dismissed and redisplayed, device 511 optionally no longer displays email alert 1252 and the other alerts besides the text input alert. However, text input alert 1242 optionally remains displayed by device 511 even when lock screen 1240 is dismissed and redisplayed as long as text input to text input user interface 1202 is required. For example, Figure 12T , lock screen 1240 has been dismissed and home screen 1254 is being displayed on device 511. Home screen 1254 is, optionally, a user interface of an operating system of device 511 that displays a plurality of selectable icons for running various applications or accessing various functions on device 511. In some embodiments, when the user unlocks device 511 from lock screen 1240 (e.g., by entering authentication information into device 511), lock screen 1240 is dismissed and home screen 1254 is displayed. Figure 12U, lock screen 1240 has re-displayed on device 511 (e.g., because the user locked device 511). Email alert 1252 is no longer displayed on lock screen 1240 (e.g., despite the fact that the new email message corresponding to email alert 1252 has not yet been read). However, text input alert 1242 is optionally still displayed on lock screen 1240 because text input to text input user interface 1202 is optionally still required. Thus, text input alert 1242 is optionally more "persistent" than other types of alerts on lock screen 1240.

[0289] When text input is no longer required for the text input user interface 1202, the text input alert 1242 is optionally dismissed from the lock screen 1240. Figure 12V 12. In the embodiment of the present invention, selection of “Home” button 518 on remote control 510 has been detected, as indicated by contact 1203. In response, electronic device 500 has ceased displaying text input user interface 1202 and has begun displaying home screen 1255 on display 514. Home screen 1255 is, optionally, a user interface of device 500 that displays a plurality of selectable icons for running various applications or accessing various functions on device 500. Because text input user interface 1202 has been dismissed, text input is optionally no longer required for text input user interface 1202, and as a result, device 511 ceases displaying text input alert 1242 on lock screen 1240.

[0290] Will refer to Figures 12W to 12GG Describes the behavior of text input alerts on user interfaces other than lock screen 1240. The behavior of text input alerts on user interfaces other than lock screen 1240 is optionally the same as the behavior of text input alerts on lock screen 1240, except as otherwise described below. For example, Figure 12W , device 511 is displaying home screen 1254. Figures 12W to 12GG The examples of optionally apply to user interfaces other than the home screen 1254 (e.g., user interfaces of applications running on the device 511), other than the lock screen 1240. Figure 12W When the home screen 1254 is displayed, a selection input (indicated by contact 1203) is detected on the touch-sensitive surface 451 of the remote control 510 while the text input field 1228 has the current focus. In response to the selection input, as shown in FIG. Figure 12X As shown, the electronic device 500 optionally enters a text input mode, a soft keyboard 1238 is displayed in the text input user interface 1202, and the current focus moves to one of the keys in the soft keyboard 1238 (eg, Figure 12XAlso in response to electronic device 500 entering text input mode and displaying soft keyboard 1238, electronic device 500 optionally transmits an indication to device 511 while device 511 is displaying home screen 1254 that text input is required for user interface 1202. In response to receiving the indication, device 511 displays text input alert 1242 on home screen 1254.

[0291] Will refer to Figures 12Y to 12BB Depicting selection of text input alert 1242 from home screen 1254 to enable text entry from device 511 into text input user interface 1202. In contrast to text input alert 1242 on lock screen 1240, selection of text input alert 1242 on home screen 1254 is optionally accomplished in response to a downward swipe of text input alert 1242. For example, in Figure 12Y , contact 1203 has been detected on text input alert 1242. Figures 12Z to 12AA , contact 1203 is swiping down on text input alert 1242 and thus pulling text input alert 1242 down onto device 511. Due to the downward swipe on text input alert 1242, device 511 displays Figure 12BB The user interface 1244 shown optionally includes a soft keyboard 1246 and a text field 1248, as previously described with reference to FIG. Figure 12K Text input can be provided to text input user interface 1202 from user interface 1244 .

[0292] Similar to what is described with reference to the lock screen 1240, text entry alerts on the home screen 1254 (or other user interface on the device 511, outside of the lock screen 1240) are optionally more "persistent" than other types of alerts, as will be described with reference to the lock screen 1240. Figures 12CC to 12EE Specifically, Figure 12CC , device 511 is displaying a text input alert 1242 on home screen 1254 (e.g., as shown in FIG. Figure 12X 12.) Text input alerts displayed on home screen 1242, such as text input alert 1242, are optionally dismissed in response to the presence of different conditions, the different conditions being alerts other than text input alerts (e.g., email alerts, text message alerts, voicemail alerts, etc.). For example, alerts other than text input alerts are optionally automatically dismissed once they have been displayed for a predetermined amount of time (e.g., 2 seconds, 3 seconds, or 5 seconds), whereas text input alerts are optionally not automatically dismissed once they have been displayed for a predetermined amount of time (e.g., 2 seconds, 3 seconds, or 5 seconds) as long as text input to text input user interface 1202 is required.

[0293] For example, when Figure 12CC While device 511 is displaying text input alert 1242 on home screen 1254, device 511 optionally determines that John Smith has sent a new email message to device 511 (or a user associated with device 511). In response, device 511 displays email alert 1252 on home screen 1254, as shown in FIG. Figure 12DD In some embodiments, the email alert 1252 is displayed simultaneously with the text input alert 1242, although Figure 12DD In an embodiment, the email alert 1252 replaces the display of the text input alert 1242. After a predetermined amount of time (e.g., 2 seconds, 3 seconds, or 5 seconds) has passed since the email alert 1252 was initially displayed, the device 511 optionally dismisses the email alert 1252. However, because text input to the text input user interface 1202 is still required when the email alert 1252 is dismissed, the text input alert 1242 optionally remains displayed on the home screen 1254, as shown in FIG. Figure 12EE Thus, text entry alert 1242 is optionally more “persistent” than other types of alerts on home screen 1254 .

[0294] The text input alert 1242 is optionally dismissed from the home screen 1254 when the user explicitly dismisses it from the home screen 1254 (in addition to being dismissed when text input is no longer required for the text input user interface 1202). Figure 12FF , an upward swipe of the text input alert 1242 is being detected by the device 511. In response to the swipe, the text input alert 1242 is optionally dismissed and no longer displayed on the home screen 1254, as shown. Figure 12GG shown.

[0295] In some embodiments, multiple multifunction devices can communicate with the electronic device 500. Figures 12HH to 12MM Describes the behavior of text input alerts on these multiple multifunction devices. Figure 12HH , the electronic device 500 is optionally in text input mode and is displaying a text input user interface 1202 (e.g., as shown in FIG. Figure 12E As described above). In addition, electronic device 500 is optionally in communication with device 511A and device 511B. Device 511A and device 511B are optionally multi-function devices, such as device 511 described previously. Device 511A is displaying home screen 1254A, and device 511B is displaying home screen 1254B. Although the description will be made with the case where device 511A and device 511B display home screen 1254A and home screen 1254B, respectively, Figures 12HH to 12MM However, it should be understood that according to the above disclosure, Figures 12HH to 12MM The examples are optionally implemented in a case where device 511A and device 511B are displaying a lock screen or where one of device 511A and device 511B is displaying a lock screen and the other of device 511B is displaying a home screen (or any combination of user interfaces on device 511A and device 511B).

[0296] In some embodiments, in response to determining that text input is required for the text input user interface 1202, the electronic device 500 transmits an indication of the need for text input only to a subset of the devices with which the electronic device 500 is communicating. In some embodiments, the electronic device 500 transmits the indication to different devices based on different criteria met. For example, 1) one or more devices closest to the electronic device 500 are optionally the devices that receive the indication; 2) one or more devices associated with a user account authorized on the electronic device 500 (e.g., logged into the user account) are optionally the devices that receive the indication; 3) one or more devices that have previously been paired with the electronic device 500 are optionally the devices that receive the indication; 4) one or more devices on the same Wi-Fi network as the electronic device 500 are optionally the devices that receive the indication; 5) one or more devices that are currently providing other input to the electronic device 500 (e.g., currently controlling the electronic device 500) are optionally the devices that receive the indication; and / or 6) one or more devices within a threshold distance of the electronic device 500 are optionally the devices that receive the indication.

[0297] exist Figure 12HH , device 511B is optionally closer to electronic device 500 than device 511A. Figure 12II As shown, electronic device 500 optionally transmits an indication of the need for text input for text input user interface 1202 to device 511B rather than to device 511A. As a result, device 511B optionally displays text input alert 1242, while device 511A does not display a text input alert.

[0298] In some embodiments, the electronic device 500 transmits an indication of a need for text input for the text input user interface 1202 to multiple devices. Figure 12JJ, both device 511A and device 511B have received an indication of the need for text input. As a result, device 511A is displaying text input alert 1242A, and device 511B is displaying text input alert 1242B, both indicating that text input is needed for text input user interface 1202. In some embodiments, to limit the number of devices that are simultaneously providing text input to text input user interface 1202, if a user of one of device 511A and device 511B selects its respective text input alert, then the display of the text input alert on the other of device 511A and device 511B is optionally stopped. For example, in Figures 12KK to 12LL , the user of device 511B has swiped down text input alert 1242B to select it. As a result, device 511B displays user interface 1244 for entering text into text input user interface 1202, as shown in FIG. Figure 12MM Because the text input alert 1242B on device 511B is selected, device 511A stops displaying the text input alert 1242A, as shown. Figure 12MM shown.

[0299] In some embodiments, authentication is required on device 511 before soft keyboard 1246 is displayed on device 511 (e.g., if text input alert 1242 is displayed on lock screen 1240 of device 511). Whether authentication is required optionally depends on whether device 511 is a trusted device of electronic device 500 (e.g., device 511 and electronic device 500 are on the same secure Wi-Fi network, or logged into the same user account such as iCloud account). For example, Figure 12NN , the device 511 is displaying a text input alert 1242, as shown in FIG. Figure 12E As described. In addition, device 511 is a trusted device of electronic device 500 (indicated by "Trusted" above the connection between device 511 and electronic device 500). In addition, device 511 has detected the selection of text input alert 1242, as indicated by contact 1203. In response to the selection, because device 511 is a trusted device of electronic device 500, device 511 displays user interface 1244 including soft keyboard 1246 for providing text input to text input user interface 1202 without requiring device 511 to be authenticated, as shown. Figure 12OO Reference Figures 12K to 12M Exemplary details of user interface 1244 are described.

[0300] exist Figure 12PP, device 511 is not a trusted device of electronic device 500 (indicated by “Untrusted” above the connection between device 511 and electronic device 500). Device 511 has detected selection of text input alert 1242, as indicated by contact 1203. In response to the selection, because device 511 is not a trusted device of electronic device 500, device 511 requests user authorization (e.g., a passcode) on lock screen 1240, as shown in FIG. Figure 12QQ If user authorization is not provided, the device 511 optionally does not display the soft keyboard 1246. On the other hand, if Figure 12QQ If user authorization is provided in , the device 511 displays a user interface 1244 including a soft keyboard 1246 for providing text input to the text input user interface 1202, such as Figure 12RR shown.

[0301] Figures 13A to 13K 1 is a flow chart illustrating a method of indicating a need for text input to an electronic device on a multifunction device according to some embodiments of the present disclosure. Method 1300 is optionally performed on an electronic device such as device 100, device 300, device 500, or device 511 (as described above with reference to FIG. Figure 1A to Figure 1B 、 Figures 2 to 3 and Figures 5A to 5B Some operations in method 1300 may be optionally combined, and / or the order of some operations may be optionally changed.

[0302] As described below, method 1300 provides a way to indicate the need for text input to an electronic device on a multifunction device. This method reduces the cognitive burden on a user when interacting with the device user interface of the present disclosure, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charges.

[0303] In some embodiments, a first electronic device (e.g., a smartphone) (such as a Figure 1A Device 100, Figure 3 300 of them, Figure 5A 500 and / or 511) displaying (1302) a first user interface on a display of a first electronic device, wherein the first user interface is not a user interface for controlling an application of a second electronic device, such as in Figure 12B (e.g., the first electronic device is optionally capable of running a remote control application for controlling the second electronic device from the first device, but the first user interface is not a user interface of the remote control application). For example, the first user interface is optionally the home screen of the first electronic device (such as in Figure 12W), the lock screen of the first electronic device (such as Figure 12B In some embodiments, the first electronic device is configured to communicate with a second electronic device (e.g., a set-top box), and the second electronic device is controlling the display of a text input user interface (e.g., a text input user interface, such as a search user interface) on an independent display device (e.g., a television) separate from the first electronic device, such as in Figure 5A middle.

[0304] In some embodiments, while the first user interface is displayed on the display of the first electronic device, the first electronic device receives (1304) an indication from the second electronic device that text input is required for a text input user interface displayed on a separate display device, such as in Figure 12E In some embodiments, in response to receiving an indication from the second electronic device that text input is required for a text input user interface displayed on an independent display device, the first electronic device displays (1306) a text input alert on the display of the first electronic device (e.g., replacing the display of at least a portion of the first user interface with the text input alert), such as in Figure 12E . Thus, the user of the first electronic device is notified of the need for text input into the text input user interface and the ability to provide such text input from the first electronic device. This increases the efficiency of the interactions between the user and the second electronic device, thereby reducing the power consumption associated with these interactions. The first electronic device optionally receives (1308) a sequence of inputs via the one or more input devices of the first electronic device, including input to interact with the text input alert and input of one or more text characters (e.g., selection of the text input alert followed by input of one or more characters on a soft keyboard displayed on a touch-sensitive display of the first electronic device), such as in Figures 12J to 12MIn some embodiments, in response to receiving a sequence of one or more inputs, the first electronic device transmits (1310) information from the first electronic device to the second electronic device that enables the one or more text characters to be provided as text input for a text input user interface displayed on the independent display device, wherein providing the one or more text characters as text input for a text input user interface displayed on the independent display device enables updating the text input user interface on the independent display device based on the one or more text characters (e.g., updating a user name input field to show a user name, performing a search query based on the one or more text characters, etc.), such as in Figures 12J to 12M middle.

[0305] In some embodiments, based on the one or more text characters being the first text characters, updating (1312) the text input user interface with a first update, such as in Figure 12L In accordance with the one or more text characters being a second text character different from the first text character, optionally updating (1314) the text input user interface with a second update different from the first update (e.g., updating the text input user interface differently based on the text characters provided thereto), such as in Figure 12M For example, if "A" is provided as input, the text input user interface is updated based on the "A" input (e.g., updated to display "A" in the text input field), and if "B" is provided as input, the text input user interface is updated based on the "B" input (e.g., updated to display "B" in the text input field).

[0306] In some embodiments, a text input user interface displayed on a separate display device includes a soft keyboard (1316) (e.g., a soft keyboard having keys that are selectable to enter text corresponding to the selected keys into the text input user interface), such as in Figure 12E The soft keyboard is optionally used to provide text input to the text input user interface using a remote control or a multifunction device configured to operate as a remote control, because the second electronic device optionally does not include a hardware keyboard. In some embodiments, an indication that text input is required for the text input user interface is received (1318) in response to the soft keyboard gaining current focus in the text input user interface (e.g., focus in the text input user interface is moved to the soft keyboard based on input from the remote control, the first electronic device, or another electronic device controlling the second electronic device), such as in Figure 12GIn some embodiments, an indication that text input is required for a text input user interface displayed on a separate display device is received (1320) in response to a request received by the second electronic device to enter text into a text input user interface without a soft keyboard displayed in the text input user interface (e.g., selection of a text field in the text input user interface causes the second electronic device to send an indication to the first electronic device that text input is required in the text input user interface without the second electronic device displaying a soft keyboard in the text input user interface), such as in Figure 12H Instead, a soft keyboard is optionally displayed on the display of the first electronic device for entering text input.

[0307] In some embodiments, input for interacting with the text input alert includes input that selects the text input alert (e.g., tapping the text input alert, swiping right on the text input alert, swiping down on the text input alert, touching the text input alert with a force above a force threshold, above a tap force threshold), such as in Figure 12J In response to receiving input selecting a text input alert, the first electronic device optionally displays (1322) a soft keyboard on the display of the first electronic device, wherein inputting the one or more text characters includes inputting the one or more text characters at the soft keyboard on the display of the first electronic device (e.g., providing text input to the second electronic device via the soft keyboard displayed on the first electronic device), such as in Figures 12K to 12M middle.

[0308] In some embodiments, based on determining that the text input alert is displayed on a first corresponding user interface of the first electronic device (e.g., a lock screen of the first electronic device), the input for selecting the text input alert is a first input (1322) (e.g., a right swipe on the text input alert, or a touch on the text input alert with a force greater than a force threshold, greater than a tap force threshold), such as in Figure 12J Based on determining that the text input alert is displayed on a second corresponding user interface of the first electronic device (e.g., a home screen or other user interface of an application running on the first electronic device) that is different from the first corresponding user interface, the input for selecting the text input alert is optionally a second input (1326) that is different from the first input (e.g., a downward swipe on the text input alert), such as Figures 12Y to 12AA middle.

[0309] In some embodiments, an indication that text input is required for a text input user interface displayed on a separate display device is received (1328) in response to a request received by a second electronic device to enter text into a text input user interface (e.g., selection of a text field in the text input user interface, display of a soft keyboard in the text input user interface, changing the current focus in the text input user interface to a soft keyboard displayed in the text input user interface), the request being received by the second electronic device from a remote control device that is different from the first electronic device and the second electronic device, such as at 12C to 12H In some embodiments, after the text input alert is displayed on the display of the first electronic device, the second electronic device receives (1330) input from the remote control device for entering a second one or more text characters into the text input user interface (e.g., input selecting one or more keys of a soft keyboard displayed in the text input user interface), such as in Figures 12O to 12Q Input from the remote control device optionally causes (1332) updating of a text input user interface according to a second one or more text characters, such as in Figures 12O to 12Q In some embodiments, the remote control device is a dedicated remote control device that enters characters into the text input user interface via directional input that moves the focus in the text input user interface between keys in a virtual keyboard displayed in the text input user interface, such as in Figures 12O to 12Q middle.

[0310] In some embodiments, after transmitting information from the first electronic device to the second electronic device that enables the one or more text characters to be provided as text input for a text input user interface, the first electronic device receives (1334) input for running a remote control application on the first electronic device via the one or more input devices of the first electronic device (for example, after providing text input to the second electronic device via a soft keyboard that is part of the operating system of the first electronic device, launching a remote control application on the first electronic device for controlling the second electronic device), such as in Figure 12A In some embodiments, in response to receiving (1336) input for running a remote control application on the first electronic device, the first electronic device runs (1338) the remote control application on the first electronic device, such as in Figure 12AThe first electronic device optionally controls (1340) the second electronic device via one or more inputs received at the remote control application (e.g., receiving directional input or other input in the remote control application and controlling the second electronic device based on those inputs), such as in Figure 12A middle.

[0311] In some embodiments, the first electronic device displays (1342) multiple categories of alerts (e.g., alerts for incoming text messages, alerts for incoming calls, alerts for incoming emails, etc.) on a display of the first electronic device, including alerts of a first category (e.g., text input alerts) and alerts of a second category (e.g., alerts for incoming text messages, etc.), wherein the text input alerts are included in the alerts of the first category, such as in Figures 12R to 12S In response to displaying an alert in the first category of alerts (including a text input alert), the first electronic device optionally generates (1344) a first notification type at the first electronic device (e.g., a visual notification with vibration of the first electronic device but no sound, or a visual notification without sound or vibration at the first electronic device), such as at Figure 12R In some embodiments, in response to displaying an alert in the second category of alerts, the first electronic device generates (1346) a second notification type that is different from the first notification type (e.g., vibration and sound of the first electronic device), such as in Figure 12S (e.g., the first electronic device optionally treats text input alerts and other types of alerts differently). In this way, the user of the first electronic device can easily distinguish whether the first electronic device is displaying a text input alert or a different type of alert without looking at the first electronic device. This saves power on the first electronic device because the display of the electronic device can remain off. For example, other types of alerts optionally cause the first electronic device to generate a sound and / or vibration, while the text input alert optionally causes the first electronic device to generate only a vibration of the first electronic device, or causes the first electronic device to not generate a vibration or sound at all.

[0312] In some embodiments, the text input alert display (1348) is on the lock screen of the first electronic device (e.g., the user interface of the first electronic device displayed when the first electronic device is in a locked state), such as on Figure 12RIn some embodiments, user input on the lock screen is limited to selection of an alert displayed on the lock screen (e.g., a text input alert, an incoming email alert, an incoming call alert, an incoming text message alert, etc.) or entry of authentication information for unlocking the first electronic device. In some embodiments, the first electronic device simultaneously displays (1350) a text input alert and a second alert on the lock screen of the first electronic device (e.g., multiple types of alerts are simultaneously displayed on the lock screen of the first electronic device, such as a text input alert and an incoming email alert), such as in Figure 12S In some embodiments, when text input is required (1352) for a text input user interface displayed on a separate display device (e.g., when the second electronic device indicates to the first electronic device that text input is required for the text input user interface): when a text input alert and a second alert (e.g., an incoming email alert) are simultaneously displayed on the lock screen of the first electronic device, the first electronic device receives (1354) input for releasing the lock screen of the first electronic device (e.g., input for unlocking the first electronic device) via the one or more input devices of the first electronic device, such as in Figure 12T In response to receiving input for releasing the lock screen, the first electronic device optionally stops (1356) displaying the lock screen on the display of the first electronic device (e.g., displaying the home screen of the first electronic device after the first electronic device is unlocked), such as in Figure 12T In some embodiments, after stopping display of the lock screen of the first electronic device, the first electronic device receives (1358) input for displaying the lock screen on the display of the first electronic device (e.g., receiving input to lock the first electronic device) via the one or more input devices of the first electronic device, such as in Figure 12U In response to receiving input for displaying a lock screen of the first electronic device, the first electronic device optionally displays (1360) a lock screen on the display of the first electronic device, wherein the lock screen includes a text input alert but does not include a second alert, such as in Figure 12U (e.g., dismissing the lock screen of the first electronic device optionally causes alerts other than text input alerts to be dismissed and no longer displayed on the lock screen. In contrast, text input alerts are optionally "persistent" in that they always appear on the lock screen of the first electronic device as long as text input is required in the text input user interface of the second electronic device). In this way, the user of the first electronic device remains aware of the need for text input in the text input user interface, which increases the efficiency of interactions between the user and the second electronic device and reduces power consumption associated with these interactions.

[0313] In some embodiments, the text input alert display (1362) is on a corresponding user interface other than the lock screen of the first electronic device (e.g., the home screen and the user interface of the application running on the first electronic device), such as on Figure 12CC In some embodiments, when text input is required (1364) for a text input user interface displayed on a separate display device (e.g., when a second electronic device indicates to a first electronic device that text input is required for a text input user interface): the first electronic device simultaneously displays (1366) a text input alert and a second alert (e.g., an incoming email alert) on a corresponding user interface of the first electronic device, such as a reference to Figure 12DD Based on determining that one or more first dismissal criteria are met (e.g., the user dismisses the text input alert, etc.), the first electronic device optionally stops (1368) displaying the text input alert on a corresponding user interface of the first electronic device, such as Figure 12EE In some embodiments, based on determining that one or more second dismissal criteria different from the one or more first dismissal criteria are satisfied (e.g., a time threshold has been reached, the user dismisses the second alert, etc.), the first electronic device stops (1370) displaying the second alert on the corresponding user interface of the first electronic device, such as referring to Figure 12EE (e.g., the criteria for dismissing a text input alert are optionally different from the criteria for dismissing other alert types because text input alerts are optionally more "persistent" than other alert types as long as text input is required in the text input user interface of the second electronic device. For example, other alert types are optionally dismissed in response to user input dismissing them or a time threshold being reached since the alert was displayed. In contrast, text input alerts are optionally displayed until the user dismisses them—text input alerts are optionally not dismissed in response to reaching a time threshold.)

[0314] In some embodiments, while the text input alert is displayed on the display of the first electronic device, the second electronic device displays (1372) a visual indication on a separate display device indicating that the first electronic device can be used to provide text input to the text input user interface of the second electronic device (e.g., a visual indication is displayed in the text input user interface indicating to the user that the first electronic device can be used to provide text input), such as in Figure 12EThis visual indication on the separate display device notifies users who can see the separate display of the ability to provide text input to the text input user interface using the first electronic device—something these users may not have known was possible. This increases the efficiency of the interactions between the user and the second electronic device, thereby reducing power consumption associated with these interactions. In some embodiments, when the text input alert is displayed on the display of the first electronic device, the first electronic device determines (1374) that text input is no longer required for the text input user interface displayed on the separate display device, such as in Figure 12V In response to determining that text input is no longer required for the text input user interface displayed on the independent display device, the first electronic device optionally stops (1376) displaying the text input alert on the display of the first electronic device (e.g., when text input is no longer required, the text input alert is optionally no longer displayed), such as in Figure 12V middle.

[0315] In some embodiments, the first electronic device is one of a plurality of electronic devices from which text input can be provided to a text input user interface and on which a text input alert can be displayed (1378), such as in Figure 12HH In some embodiments, the second electronic device is configured to transmit (1380) to the first electronic device an indication that text input is required for the text input user interface, such as in the case where the second electronic device is in proximity to the second electronic device (e.g., multiple smartphones in proximity to the second electronic device have the ability to provide text input to the second electronic device via soft keyboards displayed on their respective touch screens). For example, multiple users with independent smartphones may be interacting with the second electronic device / text input user interface simultaneously in a group setting, providing the multiple users with the ability to interact with the second electronic device in parallel, thereby increasing the efficiency of those interactions with the second electronic device. In some embodiments, the second electronic device is configured to: based on determining that the first set of criteria are met, transmit (1380) to the first electronic device an indication that text input is required for the text input user interface, such as in the case where the second electronic device is in proximity to the second electronic device (e.g., multiple smartphones in proximity to the second electronic device have the ability to provide text input to the second electronic device via soft keyboards displayed on their respective touch screens). Figure 12II In some embodiments, the second electronic device is configured to: transmit (1382) an indication that text input is required for the text input user interface to a corresponding electronic device of the plurality of electronic devices that is different from the first electronic device based on determining that a second set of criteria that is different from the first set of criteria is satisfied, such as in Figure 12II(e.g., not each of the plurality of electronic devices receives an indication of required text input from the second electronic device, and therefore not each of the plurality of electronic devices displays a text input alert corresponding to the need for text input at the second electronic device). Different electronic devices, optionally, receive indications from the second electronic device based on different criteria that are met. For example, 1) one or more electronic devices that are closest to the second electronic device are optionally the electronic devices that receive the indication; 2) one or more electronic devices that are associated with a user account authorized on the second electronic device (e.g., logged into the user account) are optionally the electronic devices that receive the indication; 3) one or more electronic devices that have been previously paired with the second electronic device are optionally the electronic devices that receive the indication; 4) one or more electronic devices on the same Wi-Fi network as the second electronic device are optionally the electronic devices that receive the indication; 5) one or more electronic devices that are currently providing other input to the second electronic device (e.g., currently controlling the second electronic device) are optionally the electronic devices that receive the indication; and / or 6) one or more electronic devices within a threshold distance of the second electronic device are optionally the electronic devices that receive the indication.

[0316] In some embodiments, the second electronic device transmits (1384) to the first electronic device (e.g., a first smart phone in the vicinity of the second electronic device) and the third electronic device (e.g., a second smart phone in the vicinity of the second electronic device) an indication that text input is required for the text input user interface, such as in Figure 12JJ In response to receiving the indication, the third electronic device optionally displays (1386) a second text input alert on a display of the third electronic device, such as Figure 12JJ In some embodiments, when a sequence of inputs is received at the first electronic device, the third electronic device stops displaying (1388) the second text input alert on the display of the third electronic device, such as in Figures 12KK to 12MM (e.g., once one of the electronic devices on which the text input alert is displayed receives input for selecting its text input alert, the text input alert displayed on the other devices is dismissed, so that only one electronic device provides text input to the second electronic device at any time).

[0317] In some embodiments, in response to receiving a sequence of inputs at a first electronic device, the first electronic device displays (1390) on a display of the first electronic device a text input user interface (e.g., a soft keyboard) for inputting the one or more text characters, wherein the text input alert and the text input user interface are user interfaces of the operating system of the first electronic device (e.g., the text input alert and the text input user interface are built into the first electronic device and / or its operating system software and are not part of a separate remote control application on the first electronic device for controlling the second electronic device), such as in Figures 12J to 12K In some embodiments, the input for interacting with the text input alert includes an input that selects the text input alert (1390) (e.g., a tap on the text input alert, a swipe to the right on the text input alert, a swipe down on the text input alert, a touch on the text input alert with a force above a force threshold, a touch above a tap force threshold), such as in Figure 12J In some embodiments, in response to receiving (1394) input selecting a text input alert: based on determining that the first electronic device is a trusted device of the second electronic device (e.g., the first electronic device and the second electronic device are on the same secure Wi-Fi network, or logged into the same user account such as an iCloud account), the first electronic device displays (1396) a soft keyboard on the display of the first electronic device without requiring user authentication on the first electronic device, such as in Figures 12NN to 12OO In some embodiments, based on determining that the first electronic device is not a trusted device of the second electronic device, the first electronic device requires (1398) user authentication on the first electronic device, and in response to receiving the user authentication, displays a soft keyboard on the display of the first electronic device, such as in Figure 12PP to Figure 12RR In one embodiment, the user may authenticate the user before allowing text input from an untrusted device to the second electronic device (e.g., if the first electronic device is not a trusted device of the second electronic device, the user must unlock the first electronic device or otherwise enter authentication credentials for the first electronic device before allowing text input to be provided to the second electronic device via the first electronic device), wherein entering the one or more text characters includes entering the one or more text characters at a soft keyboard on a display of the first electronic device (e.g., providing text input to the second electronic device via a soft keyboard displayed on the first electronic device). Requiring user authentication before allowing text input from an untrusted device helps ensure that unexpected and / or unauthorized input to the text input user interface is avoided.

[0318] It should be understood that Figures 13A to 13KThe particular order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 700, 900, 1100, 1500, 1700, and 1900) also apply in a similar manner to the details of the processes described above with respect to Figures 13A to 13K For example, the touch input, software remote control application, simulated button, and / or simulated remote control trackpad described above with reference to method 1300 may optionally have one or more of the features of the touch input, software remote control application, simulated button, and / or simulated remote control trackpad described herein with reference to other methods described herein (e.g., methods 700, 900, 1100, 1500, 1700, and 1900). For the sake of brevity, these details are not repeated here.

[0319] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A 、 Figure 3 、 Figure 5A and Figure 23 In addition, the above reference Figures 13A to 13K The described operations are optionally performed by Figure 1A to Figure 1B 1302 and 1306, receiving operations 1304 and 1308, and transmitting operation 1310 are optionally implemented by event classifier 170, event identifier 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact on the touch screen of device 511, and event dispatcher module 174 delivers the event information to application 136-1. The corresponding event identifier 180 of application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at a first location on the touch screen corresponds to a predefined event or sub-event, such as the selection of an object on a user interface. When a corresponding predefined event or sub-event is detected, event identifier 180 activates event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally utilizes or calls data updater 176 or object updater 177 to update application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know how to update the content displayed by the application based on the GUI updater 178. Figure 1A to Figure 1BThe components depicted in the figure implement other processes.

[0320] Primary touch navigation area selection

[0321] Users interact with electronic devices in a variety of different ways, including interacting with content (e.g., music, movies, etc.) available (e.g., stored or otherwise accessible) on the electronic device. In some cases, a user can interact with an electronic device by alternating between using a dedicated remote control and a multifunction device to provide navigation input to the electronic device (e.g., a swipe to scroll through content). However, in some cases, the sizes of the touch-sensitive surfaces used to provide such navigation input on the dedicated remote control and the multifunction device are different. The embodiments described below provide a method in which the multifunction device selects a primary touch navigation area on its touch-sensitive surface that behaves similarly to the touch-sensitive surface of a dedicated remote control to provide the user with a consistent input experience across the remote control and the multifunction device, thereby enhancing the user's interaction with the electronic device. Enhancing interaction with the device reduces the amount of time required for the user to perform an operation, thereby reducing the power consumption of the device and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, this person is optionally referred to as a user of the device.

[0322] Figures 14A to 14GG An exemplary manner in which a multifunction device selects a primary touch navigation area on its touch-sensitive surface that behaves like a dedicated remote control according to some embodiments of the present disclosure is shown. The embodiments in these figures are used to illustrate the processes described below, including reference to Figures 15A to 15H Describe the process.

[0323] Figure 14A An exemplary display 514 is shown. Display 514 optionally displays one or more user interfaces including various content. Figure 14A In the example shown, the display 514 displays a user interface 1402 including a cursor 1404 that corresponds to a currently selected position of the user interface 1402 (e.g., receiving a selection input from an input device such as a dedicated remote control optionally selects the item in the user interface 1402 over which the cursor 1404 is located). The user interface 1402 is optionally displayed on an electronic device (e.g., Figure 5A1402 is displayed as an application running on an electronic device 500 of which the display 514 is part or connected. Although the user interface 1402 is shown as including a cursor 1404, it should be understood that the cursor 1404 optionally corresponds to and / or represents any object or action that can be controlled via directional input or navigation input received from an input device. For example, the cursor 1404 moving left in the user interface 1402 in response to a left directional input received from the input device optionally additionally or alternatively represents scrolling a list in the user interface 1402 to the left, moving a character in a game to the left, scrubbing backward (e.g., "left") through content playing on the electronic device, etc.

[0324] As reference Figures 5A to 5B As described above, a remote control 510 and / or device 511 are optionally used to control the electronic device 500. Specifically, the remote control 510 and device 511 optionally communicate with the electronic device 500 and provide input to the electronic device 500. The remote control 510 optionally has a reference Figure 5B Features described for providing input to electronic device 500. For example, selection of one or more of buttons 516, 518, 520, 522, 524, and 526 optionally causes remote control 510 to transmit corresponding commands to electronic device 500, to which electronic device 500 responds accordingly. Touch-sensitive surface 451 is optionally used to provide tap, click, select, navigate, and / or move input to electronic device 500, to which electronic device 500 responds accordingly. For example, touch input (e.g., a swipe) detected on touch-sensitive surface 451 optionally controls the position of cursor 1404 in user interface 1402.

[0325] Device 511 is optionally a multi-function device. In some embodiments, device 511 is a mobile phone configured to run applications and perform multiple functions independent of controlling electronic device 500 (such as phone functions, messaging functions, etc.). In some embodiments, device 511 runs a remote control application that configures device 511 to operate as a remote control for electronic device 500, or device 511 is configured as part of its operating system to operate as a remote control for electronic device 500. Figure 14A, device 511 includes a touch screen 1451 that includes a touch navigation area 1452. Touch navigation area 1452 is optionally visible on touch screen 1451 (e.g., visually distinguished from other UI elements on the display, such as by being displayed with a visible border or in a different color from surrounding UI elements) or invisible. Touch navigation area 1452 is optionally an area of ​​touch screen 1451 that provides tap, click, selection, navigation, and / or movement input to electronic device 500, and electronic device 500 responds accordingly to these inputs. For example, touch input (e.g., a swipe) detected in touch navigation area 1452 optionally controls the position of cursor 1404 in user interface 1402. In some embodiments, device 511 ignores touch input detected outside of touch navigation area 1452 and / or does not transmit it to electronic device 500. In some embodiments, the touch navigation area 1452 is a touch input area in which the device accepts free-form touch inputs such as swipes, flicks, ...

Claims

1. A method comprising: At a first electronic device having a display and one or more input devices: Displaying a user interface on the display, the user interface comprising: a first selectable control associated with a simulated remote control for controlling a second electronic device, the second electronic device being different from the first electronic device; and According to determining that the second electronic device is playing content: a content user interface element including a second set of controls for controlling playback of the content being played by the second electronic device; receiving input at the first electronic device via the one or more input devices while displaying the first selectable control and the content user interface element including the second set of controls on the display; and In response to receiving the input: Based on determining that the input is received at the first selectable control, initiating an operation associated with controlling the second electronic device based on the input received at the first selectable control; and Based on determining that the input corresponds to a selection of the content user interface element, an expanded content user interface element is displayed, wherein the expanded content user interface element includes the second group of controls and a third group of controls, and the third group of controls is used to control the content being played by the second electronic device.

2. The method of claim 1, wherein the second set of controls for controlling the playback of the content being played comprises one or more of: a rewind control for rewinding the playback of the content; and a fast-forward control for fast-forwarding the playback of the content.

3. The method according to claim 1, further comprising: In response to receiving the input, based on determining that the input has been received at a given control in the second set of controls, initiating operations to control playback of the content being played by the second electronic device based on the input received at the given control.

4. The method according to claim 1, further comprising: In response to receiving the input, based on determining that the input corresponds to a request to change a state of the content being played by the second electronic device: Initiating an operation to change the state of the content being played by the second electronic device according to the input; as well as The content user interface element is updated to reflect the change in the state of the content being played by the second electronic device.

5. The method according to any one of claims 1 to 4, further comprising: In response to receiving the input: Initiating an operation to control playback of the content being played by the second electronic device while maintaining display of the user interface and the content user interface element based on determining that the input corresponds to a selection of a corresponding control in the second set of controls in the content user interface element, the operation corresponding to the selected corresponding control in the second set of controls; and Based on determining that the input corresponds to a selection of the content user interface element other than the one or more controls in the second group of controls, the expanded content user interface element including the second group of controls and a third group of controls is displayed, and the third group of controls is used to control the content being played by the second electronic device. The method of claim 5 , wherein the expanded content user interface element is customized relative to the content being played by the second electronic device.

7. The method according to any one of claims 5 to 6, wherein the expanded content user interface element comprises: Information about the content being played by the second electronic device that was not displayed on the display before receiving the input.

8. The method according to any one of claims 5 to 7, wherein the second group of controls and the third group of controls include one or more of the following: a display / pause button, a reverse skip button, a forward skip button, a scrub bar, a progress bar, a volume control for controlling the volume of the second electronic device, and a favorite button for designating the content being played by the second electronic device as favorite content.

9. The method according to any one of claims 1 to 8, further comprising: In response to receiving the input, based on determining that the input is received at the content user interface element and corresponds to a request for controlling the playback state of the content being played by the second electronic device, an operation is initiated to control the playback state of the content being played by the second electronic device based on the received input, while maintaining the display of the first optional control and the second group of controls.

10. The method according to any one of claims 1 to 9, further comprising: A first set of controls is displayed, the first set of controls comprising one or more of: a trackpad area, a menu button, a home button, a virtual assistant button, a display / pause button, and a volume control.

11. The method according to claim 10, wherein: Based on determining that the second electronic device is configured to adjust a volume level of the content being played by the second electronic device, the first set of controls includes the volume control; and Based on determining that the second electronic device is not configured to adjust the volume level of the content being played by the second electronic device, the first set of controls does not include the volume control.

12. The method according to any one of claims 1 to 11, further comprising: Based on determining that the second electronic device is compatible with the corresponding remote control function, displaying a corresponding control in a corresponding group of controls corresponding to the corresponding remote control function; as well as According to determining that the second electronic device is not compatible with the corresponding remote control function, the display of the corresponding control in the corresponding group of controls corresponding to the corresponding remote control function is abandoned.

13. The method of any one of claims 1 to 12, wherein the first electronic device is a portable electronic device and the second electronic device is a set-top box connected to the remote display.

14. The method according to any one of claims 1 to 13, further comprising: displaying a game controller activation user interface element on the display while the first selectable control and the second set of controls are displayed on the display; receiving, via the one or more input devices, a second input corresponding to a selection of the game controller initiated user interface element; as well as In response to receiving the second input, a game controller user interface element is displayed on the display.

15. The method of claim 14, wherein the game controller user interface element comprises: A corresponding set of one or more controls for controlling a corresponding game running on the second electronic device.

16. A method according to any one of claims 14 to 15, wherein: The user interface including the first selectable control and the content user interface element including the second set of controls are displayed on the display in a first orientation mode, and The game controller user interface element is displayed on the display in a second orientation mode different from the first orientation mode.

17. The method according to any one of claims 1 to 16, further comprising: In response to receiving the input: Based on determining that the input has been received at a second corresponding control in the second set of controls, an operation is initiated to control playback of the content being played by the second electronic device based on the input received at the second corresponding control.

18. The method according to any one of claims 1 to 17, further comprising: displaying a game controller activation user interface element on the display while the first selectable control and the second set of controls are displayed on the display; receiving, via the one or more input devices, a second input corresponding to a selection of the game controller initiated user interface element; as well as In response to receiving the second input, a game controller user interface element is displayed on the display.

19. The method of any one of claims 1-18, wherein the user interface includes a first set of controls including respective controls associated with one or more commands simulating the remote control.

20. The method of any of claims 1-19, wherein the selection of the content user interface element is in an area outside of the second set of controls.

21. The method according to any one of claims 1 to 20, wherein: According to determining that the corresponding function of the second electronic device can be controlled from the first electronic device, the user interface includes a control corresponding to the corresponding function; as well as Based on determining that the corresponding function of the second electronic device associated with the playback of the content cannot be controlled from the first electronic device, the user interface does not include the control corresponding to the corresponding function.

22. The method according to any one of claims 1 to 21, further comprising: Information identifying the content being played by the second electronic device and the user interface and the content user interface element are displayed, the user interface including the first optional control, and the content user interface element including information identifying the content being played by the second electronic device and the second group of controls.

23. A first electronic device, comprising: one or more processors; Memory; Display devices; one or more input devices; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: A user interface is displayed on the display device, wherein the user interface includes: a first selectable control associated with a simulated remote control for controlling a second electronic device, the second electronic device being different from the first electronic device; and According to determining that the second electronic device is playing content: a content user interface element including a second set of controls for controlling playback of the content being played by the second electronic device; receiving input at the first electronic device via the one or more input devices while displaying the first selectable control and the content user interface element including the second set of controls on the display device; and In response to receiving the input: Based on determining that the input is received at the first selectable control, initiating an operation associated with controlling the second electronic device based on the input received at the first selectable control; and Based on determining that the input corresponds to a selection of the content user interface element, an expanded content user interface element is displayed, wherein the expanded content user interface element includes the second group of controls and a third group of controls, and the third group of controls is used to control the content being played by the second electronic device.

24. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of a first electronic device having a display device and one or more input devices, cause the first electronic device to: A user interface is displayed on the display device, wherein the user interface includes: a first selectable control associated with a simulated remote control for controlling a second electronic device, the second electronic device being different from the first electronic device; as well as According to determining that the second electronic device is playing content: a content user interface element including a second set of controls for controlling playback of the content being played by the second electronic device; receiving input at the first electronic device via the one or more input devices while displaying the first selectable control and the content user interface element including the second set of controls on the display device; as well as In response to receiving the input: Based on determining that the input is received at the first selectable control, initiating an operation associated with controlling the second electronic device based on the input received at the first selectable control; as well as Based on determining that the input corresponds to a selection of the content user interface element, an expanded content user interface element is displayed, wherein the expanded content user interface element includes the second group of controls and a third group of controls, and the third group of controls is used to control the content being played by the second electronic device.

Images