User interface for device control

Abstract

The present disclosure generally relates to displaying a user interface with device control.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 343,076, filed on May 17, 2022, entitled “USER INTERFACES FOR DEVICE CONTROLS” and U.S. Patent Application No. 18 / 197,681, filed on May 15, 2023, entitled “USER INTERFACES FOR DEVICE CONTROLS”. The contents of each of these patent applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to computer user interfaces and, more particularly, to techniques for controlling external devices. Background Art

[0004] The computer system may include hardware and / or software for providing improved techniques for interacting with a computer for users who prefer to use different input mechanisms and / or users who have one or more impairments (e.g., motor impairments and / or visual impairments). Summary of the invention

[0005] A user may prefer to use alternative input techniques and / or may have limited ability to provide certain inputs to control a computer (e.g., using a mouse and / or touch screen). A user may configure certain accessibility settings of a computer system to allow the user to more easily control the device. A user may request to use a different computer system (e.g., an external device) to control a computer system.

[0006] However, some techniques for controlling external devices using electronic devices are often cumbersome and inefficient. For example, some prior art techniques use complex and time-consuming user interfaces that may include multiple keystrokes or keystrokes. Prior art techniques require more time than necessary, which results in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, the present technology provides a faster and more efficient method and interface for electronic devices to control external devices. Such methods and interfaces optionally supplement or replace other methods for controlling external devices. Such methods and interfaces reduce the cognitive burden on users and produce a more efficient man-machine interface. For battery-driven computing devices, such methods and interfaces save power and increase the time interval between battery charges. When a user desires to control an external (e.g., remote) device, automatically displaying a specific user interface based on auxiliary settings of one or more devices (e.g., local devices and / or external devices) reduces the number of inputs required to control one or more devices. Additionally or alternatively, automatically displaying a specific user interface based on auxiliary settings to control external devices improves the user's man-machine interface.

[0008] According to some embodiments, a method is described that is performed at a first computer system that communicates with a display generation component and one or more input devices. The method includes: detecting an input corresponding to a request to control a second computer system via the one or more input devices; and in response to detecting the input corresponding to the request to control the second computer system, and simultaneously communicating with the second computer system: displaying one or more first user interface objects that control graphical elements displayed by the second computer system to navigate one or more user interfaces of the second computer system based on a determination that a first set of control criteria are met, wherein a first control criterion in the first set of control criteria is met when a first mode of controlling the computer system is enabled on the second computer system; and abandoning displaying one or more first user interface objects that control graphical elements to navigate one or more user interfaces of the second computer system based on a determination that the first set of control criteria are not met.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a first computer system that communicates with a display generation component and one or more input devices, the one or more programs including instructions for the following operations: detecting an input corresponding to a request to control a second computer system via the one or more input devices; and in response to detecting the input corresponding to the request to control the second computer system, and simultaneously communicating with the second computer system: displaying one or more first user interface objects based on a determination that a first set of control criteria is met, the first user interface objects controlling a graphical element displayed by the second computer system to navigate one or more user interfaces of the second computer system, wherein when a first mode of controlling the computer system is enabled on the second computer system, a first control criterion in the first set of control criteria is met; and based on a determination that the first set of control criteria is not met, abandoning displaying one or more first user interface objects that control a graphical element to navigate one or more user interfaces of the second computer system.

[0010] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a first computer system, the first computer system communicating with a display generation component and one or more input devices, the one or more programs including instructions for the following operations: detecting an input corresponding to a request to control a second computer system via the one or more input devices; and in response to detecting the input corresponding to the request to control the second computer system, and simultaneously communicating with the second computer system: displaying one or more first user interface objects based on a determination that a first set of control criteria is met, the first user interface objects controlling a graphical element displayed by the second computer system to navigate one or more user interfaces of the second computer system, wherein a first control criterion in the first set of control criteria is met when a first mode of controlling the computer system is enabled on the second computer system; and abandoning displaying one or more first user interface objects that control a graphical element to navigate one or more user interfaces of the second computer system based on a determination that the first set of control criteria is not met.

[0011] According to some embodiments, a first computer system configured to communicate with a display generation component and one or more input devices is described. The computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: detecting an input corresponding to a request to control a second computer system via the one or more input devices; and in response to detecting the input corresponding to the request to control the second computer system, and simultaneously communicating with the second computer system: displaying one or more first user interface objects based on a determination that a first set of control criteria is met, the first user interface objects controlling a graphical element displayed by the second computer system to navigate one or more user interfaces of the second computer system, wherein when a first mode of controlling the computer system is enabled on the second computer system, a first control criterion in the first set of control criteria is met; and based on a determination that the first set of control criteria is not met, abandoning displaying one or more first user interface objects that control a graphical element to navigate one or more user interfaces of the second computer system.

[0012] According to some embodiments, a first computer system configured to communicate with a display generation component and one or more input devices is described. The computer system includes: a component for detecting an input corresponding to a request to control a second computer system via the one or more input devices; and a component for the following operations in response to detecting the input corresponding to the request to control the second computer system and communicating with the second computer system at the same time: displaying one or more first user interface objects based on a determination that a first set of control criteria are met, the first user interface objects controlling a graphical element displayed by the second computer system to navigate one or more user interfaces of the second computer system, wherein when a first mode of controlling the computer system is enabled on the second computer system, a first control criterion in the first set of control criteria is met; and abandoning displaying one or more first user interface objects that control a graphical element to navigate one or more user interfaces of the second computer system based on a determination that the first set of control criteria are not met.

[0013] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component and one or more input devices. The one or more programs include instructions for the following operations: detecting an input corresponding to a request to control a second computer system via the one or more input devices; and in response to detecting an input corresponding to a request to control the second computer system, and simultaneously communicating with the second computer system: displaying one or more first user interface objects based on a determination that a first set of control criteria are met, the first user interface object controls a graphical element displayed by the second computer system to navigate one or more user interfaces of the second computer system, wherein when a first mode of controlling the computer system is enabled on the second computer system, a first control criterion in the first set of control criteria is met; and based on a determination that the first set of control criteria are not met, abandoning displaying one or more first user interface objects that control graphical elements to navigate one or more user interfaces of the second computer system.

[0014] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0015] Thus, a faster, more efficient method and interface is provided to the device for controlling external devices, thereby improving the effectiveness, efficiency and user satisfaction of such devices.Such methods and interfaces can supplement or replace other methods for controlling external devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the drawings.

[0017] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0018] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments.

[0019] Figure 2 A portable multifunction device with a touch screen according to some embodiments is illustrated.

[0020] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0021] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device according to some embodiments is illustrated.

[0022] Figure 4B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is illustrated according to some embodiments.

[0023] Figure 5A Personal electronic devices according to some embodiments are illustrated.

[0024] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0025] FIG. 6A to FIG. 6R An exemplary user interface for device control according to some embodiments is illustrated.

[0026] Figure 7 A flowchart illustrating a method for displaying a user interface to control a device according to some embodiments is depicted. DETAILED DESCRIPTION

[0027] The following description sets forth exemplary methods, parameters, etc. However, it should be appreciated that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0028] There is a need for electronic devices that provide efficient methods and interfaces for controlling external devices. For example, a technique enables display of a particular user interface based on what auxiliary settings are enabled at one or more devices. Such techniques can reduce the cognitive burden on users attempting to control external devices, thereby increasing productivity. In addition, such techniques can reduce processor power and battery power that would otherwise be wasted on redundant user input.

[0029] under Figure 1A to Figure 1B , Figure 2 , Figure 3 ,as well as FIG. 4A to FIG. 4B and FIG. 5A to FIG. 5B A description of an exemplary device for performing techniques for management event notifications is provided. FIG. 6A to FIG. 6R An exemplary user interface for controlling an external device is illustrated. Figure 7 is a flowchart illustrating a method of displaying a user interface to control a device according to some embodiments. FIG. 6A to FIG. 6R The user interface in is used to illustrate the processes described below, which include Figure 7 process.

[0030] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, controlling a device with one or more auxiliary settings, improving the human-machine interface for the user, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0031] In addition, in the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0032] Although the following description uses the terms "first", "second", etc. to describe various elements, these elements should not be limited by the terms. In some embodiments, these terms are used to distinguish one element from another element. For example, a first touch can be named a second touch and similarly a second touch can be named a first touch without departing from the scope of the various described embodiments. In some embodiments, the first touch and the second touch are two separate references to the same touch. In some embodiments, both the first touch and the second touch are touches, but they are not the same touch.

[0033] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limiting. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0034] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]," or "in response to detecting [a stated condition or event]," depending on the context.

[0035] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Equipment and Device. Other portable electronic devices are optionally used, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads). In some embodiments, the electronic device is a computer system that communicates with the display generation component (e.g., via wireless communication, via wired communication). The display generation component is configured to provide visual output, such as display via a CRT display, display via an LED display, or display via image projection. In some embodiments, the display generation component is integrated with the computer system. In some embodiments, the display generation component is separated from the computer system. As used herein, "display" content includes sending data (e.g., image data or video data) to an integrated or external display generation component via a wired or wireless connection to visually generate content to display content (e.g., video data rendered or decoded by display controller 156).

[0036] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0037] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0038] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports various applications with a user interface that is intuitive and clear to the user.

[0039] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0040] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited real estate, which is used to display an enable indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control, such as a knob or button).

[0041] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical property of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, movement of a touch-sensitive surface may optionally be interpreted or sensed by a user as "roughness" of the touch-sensitive surface even when there is no change in the smoothness of the touch-sensitive surface. While such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0042] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have a different configuration or arrangement of the components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0043] Memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0044] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs (such as computer programs (e.g., including instructions)) and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0045] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memories, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuits for detecting near field communications (NFC) fields, such as through short-range communications radio components. Wireless communication may optionally use any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11d), IEEE 802.11e, IEEE 802.11f, IEEE 802.11g, IEEE 802.11g). 11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)) and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols that have not been developed as of the filing date of this document.

[0046] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and sends the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and sends the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or sent to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0047] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, a depth camera controller 169, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some embodiments, the input controller 160 is optionally coupled to any of the following (or none of the following): a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an increase / decrease button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2206 in). In some embodiments, the electronic device is a computer system that communicates with one or more input devices (e.g., via wireless communication, via wired communication). In some embodiments, the one or more input devices include a touch-sensitive surface (e.g., a touchpad as part of a touch-sensitive display). In some embodiments, the one or more input devices include one or more camera sensors (e.g., one or more optical sensors 164 and / or one or more depth camera sensors 175), such as for tracking a user's gestures (e.g., hand gestures and / or air gestures) as input. In some embodiments, the one or more input devices are integrated with the computer system. In some embodiments, the one or more input devices are separate from the computer system. In some embodiments, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and is based on detected movement of a part of the user's body through the air, including movement of the user's body relative to an absolute reference (for example, the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (for example, movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (for example, a tap gesture involving a hand moving a predetermined amount and / or speed in a predetermined posture, or a shake gesture involving a predetermined speed or amount of rotation of a part of the user's body).

[0048] A quick press of the push button optionally disengages the lock of the touch screen 112 or optionally begins the process of unlocking the device using gestures on the touch screen, as described in U.S. Patent Application 11 / 322,549, filed on December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image" (i.e., U.S. Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0049] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0050] The touch screen 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0051] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® from Apple Inc. (Cupertino, California). and iPod The technology used in.

[0052] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive touch pad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, while the touch-sensitive touch pad does not provide visual output.

[0053] The touch-sensitive display in some embodiments of the touch screen 112 is described in the following applications: (1) U.S. patent application No. 11 / 381,313, filed on May 2, 2006, "Multipoint Touch Surface Controller"; (2) U.S. patent application No. 10 / 840,862, filed on May 6, 2004, "Multipoint Touchscreen"; (3) U.S. patent application No. 10 / 903,964, filed on July 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) U.S. patent application No. 11 / 048,264, filed on January 31, 2005, "Gestures For Touch Sensitive Input Devices"; (5) U.S. patent application No. 11 / 038,590, filed on January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices”; (6) U.S. patent application Ser. No. 11 / 228,758, filed on September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. patent application Ser. No. 11 / 228,700, filed on September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. patent application Ser. No. 11 / 228,737, filed on September 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. patent application Ser. No. 11 / 367,749, filed on March 3, 2006, “Multi-Functional Hand-Held Device”. All of these applications are incorporated herein by reference in their entirety.

[0054] The touch screen 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positioning or commands for performing the actions desired by the user.

[0055] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touch screen 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0056] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0057] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures a static image or a video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to the touch screen display 112 on the front of the device, so that the touch screen display can be used as a viewfinder for static images and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device so that the image of the user can be optionally obtained for video conferencing when the user views other video conference participants on the touch screen display. In some embodiments, the positioning of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that a single optical sensor 164 is used with the touch screen display for both video conferencing and static image and / or video image acquisition.

[0058] Device 100 optionally also includes one or more depth camera sensors 175 . Figure 1A A depth camera sensor coupled to a depth camera controller 169 in the I / O subsystem 106 is shown. The depth camera sensor 175 receives data from the environment to create a three-dimensional model of an object (e.g., a face) within the scene from a viewpoint (e.g., a depth camera sensor). In some embodiments, in conjunction with the imaging module 143 (also referred to as a camera module), the depth camera sensor 175 is optionally used to determine a depth map of different portions of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is located at the front of the device 100, so that an image of the user with depth information is optionally obtained for video conferencing while the user is viewing other video conference participants on the touch screen display, and a selfie with depth map data is captured. In some embodiments, the depth camera sensor 175 is located at the rear of the device, or at both the rear and front of the device 100. In some embodiments, the positioning of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that the depth camera sensor 175 is used with the touch screen display for both video conferencing and still image and / or video image acquisition.

[0059] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1AA contact force sensor is shown coupled to a force sensor controller 159 in the I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite to touch screen display 112 located on the front of device 100.

[0060] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent application numbers: 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0061] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. The tactile output generator 167 optionally includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112) and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0062] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 is optionally implemented as described in the following U.S. Patent Publication Nos.: 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0063] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: active application state, which indicates which application (if any) is currently active; display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and position information related to the device's position and / or posture.

[0064] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitating communication between various hardware components and software components.

[0065] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to (trademark of Apple Inc.) devices.

[0066] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction) and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touch pad.

[0067] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen can be set to any one of a large range of predefined thresholds. Additionally, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0068] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0069] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0070] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application or the like, along with coordinate data and other graphic attribute data if necessary, and then generates screen image data to be output to the display controller 156.

[0071] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0072] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email client module 140, IM module 141, browser module 147, and any other application requiring text input).

[0073] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone module 138 for use in location-based dialing; to the camera module 143 as picture / video metadata; and to applications that provide location-based services, such as a weather widget, a local yellow pages widget, and a map / navigation widget).

[0074] The application 136 may optionally include the following modules (or instruction sets) or a subset or superset thereof:

[0075] ● Contacts module 137 (sometimes called address book or contact list);

[0076] ● Telephone module 138;

[0077] ● Video conferencing module 139;

[0078] ● Email client module 140;

[0079] ●Instant messaging (IM) module 141;

[0080] ●Fitness support module 142;

[0081] ● Camera module 143 for still images and / or video images;

[0082] ●Image management module 144;

[0083] ●Video player module;

[0084] ●Music player module;

[0085] ●Browser module 147;

[0086] ●Calendar module 148;

[0087] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock market widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0088] A widget creator module 150 for forming a user-created widget 149-6;

[0089] ●Search module 151;

[0090] ● Video and music player module 152, which merges the video player module and the music player module;

[0091] ●Note module 153;

[0092] ● Map module 154; and / or

[0093] ●Online video module 155.

[0094] Examples of other applications 136 that may optionally be stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0095] In combination with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132 and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (for example, stored in the application internal state 192 of the contact module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating telephone numbers, email addresses, physical addresses or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers or email addresses to initiate and / or facilitate communications via the telephone module 138, video conferencing module 139, email client module 140 or IM module 141; and the like.

[0096] In conjunction with RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contact module 137, modify an entered phone number, dial a corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0097] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, the text input module 134, the contact module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0098] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0099] In conjunction with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the instant messages sent and / or received may optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0100] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154 and the music player module, the fitness support module 142 includes executable instructions for creating fitness (e.g., with time, distance and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing and sending fitness data.

[0101] In conjunction with the touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132 and image management module 144, the camera module 143 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0102] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0103] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with user instructions, including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0104] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0105] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is a mini-application that is optionally downloaded and used by a user (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm widget 149-4, and dictionary widget 149-5) or a mini-application created by a user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! widget).

[0106] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the widget creator module 150 is optionally used by a user to create widgets (e.g., converting a user-specified portion of a web page into a widget).

[0107] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0108] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0109] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the note module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0110] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the GPS module 135 and the browser module 147, the map module 154 is optionally used to receive, display, modify and store maps and data associated with the maps (e.g., driving directions, data related to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0111] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for the following operations: allowing a user to access, browse, receive (e.g., by streaming and / or downloading), playback (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a specific online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a specific online video. Additional descriptions of online video applications may be found in U.S. Provisional Patent Application No. 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0112] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs (such as computer programs (e.g., including instructions)), processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0113] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0114] A predefined set of functions that are uniquely performed through a touch screen and / or a touch pad optionally includes navigation between user interfaces. In some embodiments, the touch pad, when touched by a user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using a touch pad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touch pad.

[0115] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0116] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on the touch-sensitive display 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0117] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by application 136-1, a state queue for enabling a user to return to a previous state or view of application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0118] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 sends information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0119] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 sends event information. In other embodiments, peripheral device interface 118 sends event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0120] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0121] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0122] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events that are recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0123] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should handle the sub-events. In most cases, the hit view is the lowest level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0124] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0125] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receivers 182.

[0126] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0127] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178 and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0128] The corresponding event identifier 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event based on the event information. The event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0129] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (for example, from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0130] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sub-event sequence), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in events (e.g., 187-1 and / or 187-2) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0131] In some embodiments, event definition 186 includes a definition of an event for a corresponding user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and the object that triggered the hit test.

[0132] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0133] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0134] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0135] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and executes a predefined process.

[0136] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and executes a predetermined process.

[0137] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positioning of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0138] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0139] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, not all of which are initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be distinguished.

[0140] Figure 2A portable multifunction device 100 with a touch screen 112 according to some embodiments is illustrated. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more of these graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics will occur when the user interrupts contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic will not select the graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0141] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0142] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a depressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the strength of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0143] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference Figure 1A The tactile output generator 167 described above), the sensor 359 (e.g., an optical sensor, an acceleration sensor, a proximity sensor, a touch sensor, and / or a contact intensity sensor (similar to the above reference Figure 1A Memory 370 may include high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and may optionally include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 may optionally include one or more storage devices located away from CPU 310. In some embodiments, memory 370 stores data related to portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0144] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the previously mentioned memory device. Each module in the above-mentioned modules corresponds to the instruction set for performing the function described above. The above-mentioned modules or computer programs (for example, instruction sets or including instructions) do not have to be implemented with separate software programs (such as computer programs (for example, including instructions)), processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subsets of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0145] Attention is now turned to embodiments of a user interface that may optionally be implemented on, for example, portable multifunction device 100 .

[0146] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 according to some embodiments is illustrated. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0147] ● Signal strength indicators 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0148] ●Time 404;

[0149] ●Bluetooth indicator 405;

[0150] ●Battery status indicator 406;

[0151] A tray 408 with icons for commonly used applications, such as:

[0152] o An icon 416 labeled “Phone” of the phone module 138 , which optionally includes an indicator 414 of the number of missed calls or voicemails;

[0153] o an icon 418 of the email client module 140 labeled “Mail”, which optionally includes an indicator 410 of the number of unread emails;

[0154] o an icon 420 labeled "Browser" of the browser module 147; and

[0155] o an icon 422 labeled “iPod” of the video and music player module 152 (also referred to as the iPod (trademark of Apple Inc.) module 152); and

[0156] ● Icons for other apps, such as:

[0157] o Icon 424 labeled “Messages” of IM module 141;

[0158] o Icon 426 labeled “Calendar” of calendar module 148;

[0159] o Icon 428 labeled “Photos” of the image management module 144;

[0160] o Icon 430 labeled “Camera” of camera module 143;

[0161] o Icon 432 labeled “Online Video” of the online video module 155;

[0162] o Icon 434 labeled “Stock Market” of the stock market widget 149 - 2 ;

[0163] o an icon 436 labeled “MAP” of the map module 154;

[0164] o Icon 438 labeled “Weather” of the weather widget 149 - 1 ;

[0165] o Icon 440 labeled “Clock” of the alarm clock widget 149 - 4 ;

[0166] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0167] o an icon 444 labeled "Notes" of the notes module 153; and

[0168] o An icon 446 of a settings application or module labeled “Settings” that provides access to settings for the device 100 and its various applications 136 .

[0169] It should be noted that Figure 4A The illustrated icon labels are exemplary only. For example, the icon 422 of the video and music player module 152 is labeled "Music" or "Music Player". Other labels may be used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0170] Figure 4B The example embodiment has a touch-sensitive surface 451 (eg, touch screen display 112) that is separate from a display 450 (eg, touch screen display 112). Figure 3 tablet computer or touch pad 355) device (e.g., Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0171] Although some of the examples below are given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the principal axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B 460 corresponds to 468 and 462 corresponds to 470) in contact with touch-sensitive surface 451 (e.g., Figure 4B Thus, when the touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B When the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is separated, the device is used to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0172] Additionally, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact) followed by movement of a cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0173] Figure 5AAn exemplary personal electronic device 500 is illustrated. Device 500 includes a body 502. In some embodiments, device 500 may include components related to devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in the foregoing. In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. Alternatively, or in addition to touch screen 504, device 500 also has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of contact (e.g., touch) applied. One or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of touch. The user interface of device 500 can respond to touch based on the intensity of the touch, which means that touches of different intensities can invoke different user interface operations on device 500.

[0174] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International patent application serial number PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO patent publication number WO / 2013 / 169849; and International patent application serial number PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO patent publication number WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0175] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0176] Figure 5B An exemplary personal electronic device 500 is depicted. In some embodiments, the device 500 may include information about Figure 1A , Figure 1B and Figure 3 Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. I / O portion 514 may be connected to a display 504, which may have a touch-sensitive component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). In addition, I / O portion 514 may be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 may be optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, input mechanism 508 may be optionally a button.

[0177] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as GPS sensor 532, accelerometer 534, orientation sensor 540 (e.g., compass), gyroscope 536, motion sensor 538, and / or combinations thereof, all of which are operably connected to I / O portion 514.

[0178] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the techniques described below, including process 700 ( Figure 7). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5B components and configurations, but may include other components or additional components in a variety of configurations.

[0179] As used herein, the term "indicative representation" refers to an optional feature of the device 100, 300, and / or 500 ( Figure 1A , Figure 3 and FIG. 5A to FIG. 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) may each optionally constitute an affordance.

[0180] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface that a user is interacting with. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that a touch-screen display (e.g., Figure 1A A touch-sensitive display system 112 or Figure 4AIn some implementations of the touch screen 112 in FIG. 1 , a contact detected on the touch screen acts as a “focus selector” such that when an input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted in accordance with the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0181] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may optionally include a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0182] As used herein, an "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched (e.g., become open) on the device. In some embodiments, a downloaded application becomes an installed application using an installer that extracts program portions from a downloaded software package and integrates the extracted portions with the operating system of a computer system.

[0183] As used herein, the term "open application" or "executing application" refers to a software application that has retained state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or executing application is optionally any of the following types of applications:

[0184] ● The active app currently displayed on the display of the device on which the app is being used;

[0185] ● Background applications (or background processes) that are not currently displayed but for which one or more processes are being processed by one or more processors; and

[0186] ●Not running but has data stored in memory (volatile and non-volatile respectively)

[0187] A suspended or dormant application that contains status information that can be used to resume execution of the application.

[0188] As used herein, the term "closed application" refers to a software application that does not have retained state information (e.g., the state information of a closed application is not stored in the memory of the device). Therefore, closing an application includes stopping and / or removing the application process of the application and removing the state information of the application from the memory of the device. Generally speaking, when in a first application, opening a second application does not close the first application. When the second application is displayed and the first application stops being displayed, the first application becomes a background application.

[0189] Attention is now turned to embodiments of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0190] FIG. 6A to FIG. 6R Illustrated are exemplary user interfaces for controlling a device according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figure 7 process.

[0191] FIG. 6A to FIG. 6D An exemplary graphical user interface for auxiliary settings is illustrated. The auxiliary settings provide different options that can enable different modes of the device, allowing the user to better interact with and use the devices 600 and 614. These different options in the auxiliary settings allow the user to control the local device (e.g., device 600) and / or the remote device (e.g., device 614) in different ways.

[0192] exist Fig. 6A, the device 600 displays an interface 602a of an auxiliary settings menu. In some embodiments, the device 600 includes one or more features of the device 100, 300 and / or 500. Interface 602a includes a menu option 604 for accessing the settings of the screen reader and a menu option 606 for accessing the settings of the hand gestures. As explained in more detail herein, when the screen reader is enabled, the device 600 will output a simulated voice of the content of the user interface being displayed (e.g., sequentially) so that the user can hear the different options provided by the user interface. When hand gestures are enabled, when the device is worn by the user (e.g., on the user's wrist and / or on the user's head), the device 600 will detect air-based movement or action. Using one or more sensors (e.g., optical sensors (e.g., heart rate sensors), cameras, gyroscopes, accelerometers)), the device 600 can distinguish different hand gestures and execute corresponding commands. In some embodiments, device 600 detects hand gestures by detecting changes in the user's wrist, blood flow, and / or heart rate, rather than using a camera to detect gestures (e.g., using a camera to detect hand movement or finger placement). In some embodiments, device 600 detects hand gestures without using a camera that has the user's hand in the camera's field of view.

[0193] exist Fig. 6A , while displaying interface 602a, device 600 detects input 650a (e.g., a tap gesture and / or a mouse click). Interface 602a is a settings user interface that enables a user to modify how a user of device 600 interacts / interfaces with device 600. In response to detecting input 650a, device 600 displays interface 602b of a settings menu, such as Figure 6B Depicted.

[0194] exist Figure 6B In the example, interface 602b includes menu option 608 to display available hand gestures. While displaying interface 602b, device 600 detects input 650b (e.g., a tap gesture and / or a mouse click) directed to menu option 608. In response to detecting input 650b, device 600 displays interface 602c of a settings menu, such as Figure 6C Depicted.

[0195] exist Figure 6C At 602c, the interface 602c includes a menu option for customizing a specific gesture to perform a specific command (eg, function). The interface 602c also includes a trigger switch 610 for enabling (and / or disabling) the gesture. Figure 6CAs illustrated, the trigger switch 610 is enabled. Therefore, the device 600 will detect hand gestures and take action on them, otherwise if the trigger switch is disabled, the device 600 will not detect the hand gestures and / or take action on them. In some embodiments, the device 600 can use one or more sensors (e.g., accelerometers, gyroscopes and / or optical sensors (e.g., one or more cameras and / or heart rate sensors)) to distinguish between different hand gestures. In some embodiments, the device 600 detects gestures based on how the device 600 is worn. As illustrated, the device 600 is a watch worn on the user's wrist. Therefore, in some embodiments, the device 600 detects different hand gestures, such as described and referred to as hand gestures in U.S. Provisional Application 63 / 221,331, which is hereby incorporated by reference in its entirety. In some implementations, a hand gesture (e.g., a double pinch gesture) can initiate a command, such as answering or ending a phone call, canceling a notification, taking a photo, playing or pausing media, and starting, pausing, or resuming a workout on device 600 (and / or device 614). In some embodiments, device 600 (and / or device 614) displays instructions indicating a hand gesture for executing a particular command using auxiliary touch (e.g., double pinching to play music). For example, in response to device 600 receiving a call, device 600 can display instructions to answer the call using a pinch gesture 609c.

[0196] exist Figure 6C At, the function can be customized for multiple gestures. In some embodiments, in order to customize a specific hand gesture, the device 600 provides an enable representation for a specific hand gesture, including, in some embodiments, a pinch enable representation 608a for a pinch gesture 609a, a double pinch enable representation 608b for a double pinch gesture 609b (e.g., two pinches that occur within a period of time and / or in a continuous order), a clench enable representation 608c for a clench gesture 609c, and / or a double clench enable representation 608d for a double clench gesture 609d (e.g., two clenchings that occur within a period of time and / or in a continuous order). In some embodiments, if the device 600 is a device worn on different parts of the body, the device 600 provides enable representations for different gestures. In an embodiment in which the device 600 is a head-mounted device, the gestures associated with the enable representations 608a-d optionally include gestures performed in a manner that detects gestures in the field of view of one or more cameras of the head-mounted device. In some embodiments, for a head mounted device, a single gesture includes detected movements of both hands of a user.

[0197] exist Figure 6CAt , when displaying interface 602c, device 600 detects input 650c (e.g., a tap gesture and / or a mouse click) on double pinch enable representation 608b. In response to detecting input 650c, device 600 displays interface 602d of a settings menu, such as Fig.6D Depicted.

[0198] exist Fig.6D At , interface 602d includes affordances 612a-c for configuring (e.g., specifying or setting) a double pinch gesture to perform a command (e.g., function and / or operation) of a visual indicator (e.g., visual indicator 622) (e.g., highlight and / or selector). As illustrated, affordances 612a-c include affordances 612a-c that customize a command for advancing a visual indicator forward (e.g., to a subsequent item in a series of items) in response to detecting a corresponding gesture, affordances 612b that customize a command for moving a visual indicator backward (e.g., to a previous item in a series of items) in response to detecting a corresponding gesture, and / or affordances 612c that customize a command for initiating a selection (e.g., of an item identified by a visual indicator and / or a highlighted item) in response to detecting a corresponding gesture.

[0199] exist Fig.6D , option 612 includes an option to perform a system action (e.g., display a user interface for notifications and / or display a control user interface). As illustrated, option 612 provides an option for configuring a gesture to display an action menu 612d, as described in more detail below. Figure 6C , the device 600 is configured so that double clenching executes a command to display an action menu 612d.

[0200] exist Fig.6D At , affordances 612e-612g include affordances for executing input provided by rotatable input device 601 in response to detecting a hand gesture. As illustrated, affordances 612e-612g include affordances 612e configured to simulate a press (e.g., press down) of rotatable input device 601 as a hand gesture, affordances 612f configured to simulate a rotation of rotatable input device 601 in a first direction (e.g., upward and / or clockwise) as a hand gesture, and / or affordances 612g configured to simulate a rotation of rotatable input device 601 in a second direction (e.g., downward and / or counterclockwise) as a hand gesture.

[0201] exist Fig.6D, options 612 include an option to execute a shortcut. A shortcut may include one or more (e.g., two or three consecutive commands) commands configured by a user, such as shortcuts for opening a specific application, automatically taking an action within an application, and / or modifying system settings (e.g., muting notifications and / or turning on airplane mode). As illustrated, interface 602d includes an option to execute a fitness shortcut 612h, such as a shortcut for executing a fitness shortcut 612h. Fig. 6F In some embodiments, the interface 602 includes other shortcuts, such as a commuting shortcut 612i.

[0202] exist Fig.6D In some embodiments, the shortcuts can be configured by the device 600 and / or the device 614. In some embodiments, the device 600 and the device 614 communicate wirelessly, wherein system settings, modes, positions, and / or commands (e.g., inputs and / or system actions) are communicated between the devices. In some embodiments, the device 600 and the device 614 log into the same user account and communicate over a network. In some embodiments, the device 600 and the device 614 communicate directly, for example, using short-range wireless communications.

[0203] exist Fig.6D At , option 612 is displayed for a double pinch gesture. However, option 612 can be configured for other hand gestures, such as for a pinch gesture, a clench gesture, and a double clench gesture.

[0204] exist Fig.6D At , when the option to execute the fitness shortcut 612h is displayed, the device 600 detects an input 650d (e.g., a tap and / or a mouse click) regarding the option to execute the fitness shortcut 612h. In response, the device 600 configures the double pinch gesture to correspond to a command to execute the fitness shortcut 612h. Once the double pinch gesture corresponds to (assigned to) a command to execute the fitness shortcut 612h, the device 600 will execute the fitness shortcut 612h in response to detecting that the user of the device 600 has performed the double pinch gesture.

[0205] exist Fig. 6E , the user has navigated device 600 to user interface 616a. While displaying interface 616a, device 600 detects a double pinch gesture 609b (e.g., in the air, without touching the touch-sensitive display, and / or without touching a button of device 600). In response to detecting double pinch gesture 609b, device 600 executes fitness shortcut 612k, which includes causing device 600 and / or device 614 to display a corresponding fitness interface 616b, such as Fig. 6F Described in.

[0206] exist Fig. 6F, the device 600 has executed a fitness shortcut 612h. The fitness shortcut 612h optionally includes a command to open a media application and a command to play (e.g., automatically play) a song using the media application. Thus, multiple commands can be executed in response to detecting a gesture (e.g., a single hand gesture). As illustrated, "Fitness Song 1" is playing in the user interface 616b of the media application.

[0207] exist Fig. 6F In some embodiments, a hand gesture detected by device 600 causes a command to be executed at device 614 (and / or device 600). In some embodiments, device 600 (and / or device 614) displays instructions indicating a hand gesture for using auxiliary touch to execute a particular command. For example, when displaying interface 618b, device 600 (and / or device 614) may display an indication that a particular gesture causes device 600 (and / or device 614) to play and / or pause music (e.g., double pinch to pause music) (e.g., as opposed to launching a shortcut). In some embodiments, the indication indicates that a double pinch gesture 609b pauses music. In some embodiments, the indication indicates that a triple pinch gesture pauses music. In some embodiments, the indication indicates that a triple pinch gesture plays music. In some embodiments, a gesture detected by device 600 causes device 600 to launch a shortcut. For example, in response to detecting Fig. 6E In response to the double pinch gesture 609b at the device 614, the device 614 executes the fitness shortcut 612h, which causes the media application to be displayed at the device 614, such as Fig. 6F In some embodiments, based on a set of criteria (e.g., proximity of device 600 and device 614, device 600 and device 614 are signed in the same user account, both device 600 and device 614 are unlocked, and / or device 600 and device 614 are paired), when fitness shortcut 612h is executed, device 600 causes device 614 to display a media application. In some embodiments, the shortcut causes the corresponding media application to be displayed at both devices 600 and 614. In some embodiments, the fitness shortcut causes the media application to be displayed at only one of devices 600 and 614 (and not the other device).

[0208] exist Figure 6G , the user has navigated back to interface 616a, and device 614 is displaying interface 618a. In some embodiments, device 614 displays a lock screen instead of interface 618a. In some embodiments, device 614 is in a low-power state and / or has a dimmed display when displaying interface 618a or the lock screen (e.g., in response to a timeout and / or not receiving user input within a threshold time period).

[0209] exist Figure 6G At , while displaying interface 616a, device 600 detects input (e.g., a series of touch inputs to display action menu 612d and / or double pinch gesture 609d). Figure 6G The device 600 displays an action menu 612d, such as Figure 6H Described in.

[0210] exist Figure 6H At , action menu 612d includes affordances for performing specific actions. As illustrated, action menu 612d includes affordance 620, which, when selected, causes display of one or more shortcuts (e.g., commute shortcut 612i and / or fitness shortcut 612h). Action menu 612d also includes affordance 623, which, when selected, causes display of a list of external (e.g., remote) devices that can be controlled by device 600, as described in more detail below.

[0211] exist Figure 6H At , device 600 displays visual indicator 622 (e.g., highlighting, shading, symbol, crosshairs, and / or shape) to distinguish affordance 624 from affordance 623. Visual indicator 622 provides an indication that device 600 will execute a command (e.g., selection and / or touch and drag) about a user interface object (e.g., application icon, menu option, and / or affordance) associated with visual indicator 622. In some embodiments, device 600 moves (e.g., translates and / or shifts) visual indicator 622 from one user interface object to another user interface object in response to detecting user input (e.g., hand gesture). In some embodiments, device 600 displays an animation of visual indicator 622 moving. In some embodiments, device 600 does not display an animation of visual indicator 622 moving.

[0212] exist Figure 6H At , device 600 displays visual indicator 622 on affordance 624 (e.g., upon initial display of action menu 612d) before displaying visual indicator 622 on affordance 623. Device 600 sequentially moves visual indicator 622 through action menu 612d in response to detecting an input corresponding to a request to move visual indicator 622. For example, in response to detecting an input (e.g., pinch gesture 609a and / or touch input 650a), device 600 stops displaying visual indicator 622 at affordance 624 and displaying visual indicator 622 on affordance 620. In response to detecting another input (e.g., pinch gesture 609a and / or touch input), device 600 stops displaying visual indicator 622 on affordance 620 and displays visual indicator 622 on affordance 623.

[0213] exist Figure 6H At 640, when visual indicator 622 is displayed on affordance 623, device 600 detects input (e.g., pinch gesture 609c, mouse click, and / or tap 650h) selecting affordance 623. In response to detecting the input selecting affordance 623, device 600 displays device selector menu 626, such as Fig.6I Described in.

[0214] exist Fig.6I At , device 600 displays device selector menu 626. Device selector menu 626 provides options to select an external device to be controlled, such as device 614 or a different external device. In some embodiments, device selector menu 626 includes options to control a tablet, a phone (e.g., device 614), and / or a computer (e.g., a laptop and / or a desktop computer). In some embodiments, other computer systems (e.g., a smart TV and / or a head mounted device) can be controlled by device 626.

[0215] exist Fig.6I In some embodiments, device 600 and device 614 (and / or other devices in device selector menu 626) are logged into the same user account. In some embodiments, device selector menu 626 includes (e.g., only) options for controlling devices logged into the same account as device 600. In some embodiments, device selector menu 626 includes options for controlling devices within a threshold distance from each other (e.g., both devices are located in the same room, home, and / or city). For example, device 600 and / or device 614 may communicate distance based on GPS data, based on being within a short-range communication range, and / or based on being connected to the same LAN. In some embodiments, device selector menu 626 includes a first group of device types, and excludes a second group of device types that are different from the first group of device types. For example, a device larger than device 600 (e.g., a larger display area and / or physical size) is included, while a device with a smaller display than device 600 is excluded (even if the device with a smaller display is within a threshold distance and / or logged into the same user account as device 600). As another example, a phone, tablet, laptop, and / or desktop computer are included in device selector menu 626 while a smart TV and / or streaming computing device for TV are not included (even if those devices are within a threshold distance and / or logged into the same user account as device 600).

[0216] exist Fig.6IIn some embodiments, before receiving input selecting an option for controlling device 614, device 600 and device 614 are in communication (e.g., wireless communication, such as LAN, Bluetooth, and / or via a network). Fig.6I After the input (for example, in Figure 6J to Figure 6P When device 600 controls device 614), device 600 and device 614 are in / maintain communication. In some embodiments, when device 600 no longer controls device 614, device 600 and device 614 maintain communication.

[0217] exist Fig.6I In some embodiments, device 600 and device 614 are in wireless communication, wherein system settings (e.g., auxiliary settings), modes, and / or commands (e.g., inputs, system actions, and / or shortcuts) are communicated between the devices prior to receiving input selecting an option for controlling device 614. In some embodiments, in response to receiving input selecting an option for controlling device 614 at device 600, system settings (e.g., auxiliary settings), modes, and / or commands (e.g., inputs and / or system actions) are communicated between devices 600 and 614.

[0218] exist Fig.6I At , when device selector menu 626 is displayed, device 600 detects input (e.g., pinch gesture 609c, mouse click and / or tap 650i) that selects an option for controlling device 614. In response to detecting the input that selects an option for controlling device 614, device 600 displays a particular user interface depending on how the auxiliary settings are configured at device 614 and / or device 600. As described herein, at Figures 6J to 6L At , in response to input selecting an option for controlling device 614, and when a screen reader is disabled on device 600 and adaptive navigation is enabled on device 614, device 600 displays user interface 630. Figure 6M to Figure 6P , in response to detecting an input selecting an option for controlling device 614, and when a screen reader is enabled on device 600 and adaptive navigation is disabled on device 614, device 600 displays user interface 632. Automatically displaying user interface 630 or user interface 632 based on how device 600 and device 614 are configured causes device 600 to display appropriate controls based on how device 600 and device 614 are configured, which reduces the number of inputs at device 600 (and / or device 614) and provides visual feedback about how each device is configured.

[0219] exist Figures 6J to 6L, a screen reader is disabled on device 600 and adaptive navigation is enabled on device 614. Adaptive navigation is an additional (or, alternative) input setting that allows a user (e.g., a user with a motor impairment) to navigate a user interface using user-configurable controls and / or devices. Figure 6R , enabling adaptive navigation (e.g., using a trigger switch and / or button) under system settings (e.g., accessibility settings) of device 614. Fig. 6A As described in , a screen reader is enabled at device 600 by navigating to menu option 605 for assistive settings.

[0220] exist Figure 6J At the same time, device 614 displays visual indicator 622 based on adaptive navigation enabled at device 614. In some embodiments, adaptive navigation is enabled at device 614 (e.g., by user input detected at device 614) before device 600 controls device 614. In some embodiments, if adaptive navigation is enabled before device 600 controls device 614, device 614 displays visual indicator 622 before device 600 controls device 614. In some embodiments, before (or after) using device 600 to control device 614, the user controls device 614 using visual indicator 622 and input at device 614. In some embodiments, visual indicator 622 displayed on device 614 has an appearance similar to or different from visual indicator 622 displayed on device 600. In some embodiments, in response to detecting that device 600 is controlling device 614, visual indicator 622 displayed on device 600 stops being displayed. In some embodiments, visual indicator 622 continues to be displayed on device 600 while device 600 controls device 614. In such embodiments, hand gestures are used to control visual indicator 622 at device 600 to navigate and select affordances (e.g., affordance 638 and / or system affordance 640), which in turn initiates commands of visual indicator 622 at mobile device 614.

[0221] exist Figure 6JIn some embodiments, the device displays a notification 636 (e.g., a notification overlaid on a user interface) that the device is being controlled by another device (or is controlling another device). As illustrated, device 600 displays a notification 636 indicating that device 600 is ready to control device 614. Device 614 displays a notification 636 indicating that device 614 is controlled by device 600. In some embodiments, notification 636 is persistent (e.g., always displayed when device 600 controls device 614). In some embodiments, notification 636 is temporary (e.g., device 614 displays notification 636 for a threshold amount of time, and then stops displaying notification 636 once the threshold amount of time is reached).

[0222] exist Figure 6J In response to Fig.6I , and based on disabling the screen reader at device 600 and enabling adaptive navigation at device 614, device 600 displays interface 630. Interface 630 is an interface that includes one or more user interface objects for controlling device 614. As illustrated, interface 630 includes one or more affordances 638 (e.g., a next affordance 638a for advancing forward, a back affordance 638b for advancing backward, and a selection affordance 638c) for controlling visual indicators 622 displayed on device 614 (e.g., similar to Fig.6D The interface 630 includes system actions (similar to the input controls 612a-612c) for controlling the device 614. Fig.6D In some embodiments, the one or more system enable representations 640 include an enable representation of open applications (e.g., an application switcher) for displaying device 614 and / or an enable representation of a home screen for displaying device 614.

[0223] exist Figure 6J In some embodiments, device 614 responds to input received at device 614 (e.g., although device 600 is controlling device 614 at the same time, the user can still control device 614 as the user normally would). For example, controlling device 614 through input at 600 does not prohibit device 614 from detecting input received at device 614. In some embodiments, device 614 does not respond to input at device 614 when being controlled by device 600 (e.g., the user cannot control device 614 using input at device 614 based on being controlled by device 600). In some embodiments, when device 600 displays interface 630, device 614 does not display interface 630 (e.g., and / or one or more of its user interface objects).

[0224] exist Figure 6JAt , while displaying interface 630, device 600 detects input to move visual indicator 622 (e.g., pinch gesture 609a and / or tap 650j on next affordance 638). In response to detecting the input to move visual indicator 622, device 600 initiates a command for device 614 to move (e.g., advance) visual indicator 622 in user interface 618a (e.g., from mail application 642a to calendar application 642b), as shown. Figure 6K Described in.

[0225] exist Figure 6K At , while displaying interface 630, device 600 detects an input corresponding to a selection (e.g., a pinching gesture 609c and / or a tap 650k). In response to device 600 detecting the input corresponding to the selection, device 600 initiates a command for device 614 to execute the selection. Because visual indicator 622 was on calendar application 642b when the input (e.g., pinching gesture 609c and / or tap 650k) was received, device 614 selects calendar application 642b, which results in display of user interface 618a, as shown. Figure 6L Described in.

[0226] exist Figure 6L At , device 614 displays user interface 618c of calendar application 642b. In some embodiments, in response to detecting further input at device 600 (e.g., hand gestures and / or touch input), device 600 optionally moves visual indicator 622 displayed on device 614. Figure 6N As further described in detail, in some embodiments, in response to the device 600 detecting a rotational input of the rotatable input device 601 and / or a hand gesture corresponding to (e.g., simulating) a crown rotation (e.g., a hand gesture corresponding to (e.g., simulating) a rotation of the rotatable input device 601 in a first direction and / or a rotation of the rotatable input device 601 in a second direction), the visual indicator 622 moves more quickly through the user interface (e.g., interface 618a and / or user interface 618c).

[0227] In some embodiments, interface 630 includes completion affordance 644 that terminates a session of device 600 controlling device 614 using interface 630. In some embodiments, after receiving input to affordance 644 and / or after visual indicator 622 that device 600 no longer controls device 614, device 600 communicates with device 614. In some embodiments, device 600 is paired with device 614.

[0228] exist Figure 6M to Figure 6P, in response to input selecting an option for controlling device 614, device 600 displays user interface 632. As illustrated, a screen reader is enabled on device 600, and adaptive navigation is disabled on device 614. A screen reader is an additional (or, alternative) input setting that allows a user (e.g., a user with a visual impairment) to navigate a user interface on a device. In some embodiments, the screen reader includes a display of a visual indicator 622. In some embodiments, the visual indicator 622 for the screen reader has an appearance similar to or different from the visual indicator 622 for adaptive navigation. When operating with a screen reader enabled, information about a graphical user interface object that visual indicator 622 turns on (e.g., the name of the object and / or the function that the object will perform) will be output by a device (e.g., device 600, device 614, a wirelessly connected headset, and / or another device that will provide audio output) via simulated speech.

[0229] Temporary turn Figure 6R In some embodiments, the screen reader is enabled or disabled via user input at device 614 (e.g., using a trigger switch and / or button) under system settings of device 614 (e.g., accessibility settings).

[0230] exist Figure 6M In some embodiments, when the screen reader at device 600 is enabled and device 600 begins to control device 614, the screen reader is enabled (e.g., automatically) on device 614. For example, in some embodiments, when the screen reader is enabled on device 600, and in response to device 600 detecting that a selection is made Fig.6I In response to input of affordance 623, device 600 initiates a command to enable a screen reader on device 614. In response to receiving the command to enable a screen reader on device 614, device 614 enables the screen reader.

[0231] exist Figure 6M At , device 614 provides an audio output 646a of a mail application 642a ("email application"). In some embodiments, the audio output includes an audio output corresponding to the function being performed (e.g., "open email application"). In some embodiments, device 600 provides an audio output 646a of a mail application 642a ("email application") (e.g., in addition to and / or instead of device 614 providing an audio output).

[0232] exist Figure 6MAt, in some embodiments, device 600 and device 614 display notification 636. As described above, in some embodiments, device 614 responds to input at device 614 (e.g., a user can still control device 614 despite being controlled by device 600). For example, controlling device 600 by input detected at device 614 does not prohibit device 614 from detecting input received at device 614. In some embodiments, device 614 does not respond to input at device 614 when being controlled by device 600 (e.g., a user cannot control device 614 using input at device 614 based on being controlled by device 600). In some embodiments, when device 600 displays interface 632, device 614 does not display interface 632 (e.g., and / or one or more of its user interface objects).

[0233] exist Figure 6M At, in response to enabling the screen reader at device 600 and disabling adaptive navigation at device 614, device 600 displays interface 632. As illustrated, interface 632 includes a touchpad area 633 for controlling device 614. In some embodiments, touchpad area 633 occupies an area of ​​the user interface and corresponds to an area where device 600 detects one or more touch inputs (e.g., taps, swipes, two-finger swipes, and / or two-finger rotations). The one or more touch inputs allow a user of device 600 to navigate the user interface of device 614. In some embodiments, one or more touch inputs control visual indicators 622 (e.g., move forward, move backward, and / or select) (e.g., similar to Fig.6D 612a-c associated with the affordances 612a-c). In some embodiments, one or more touch inputs (e.g., a set of inputs and / or all inputs) detectable at device 614 when a screen reader is enabled at device 614 can be detected via touchpad area 633 (and the same operations are performed if the inputs are detected by device 614). In some embodiments, touchpad area 633 controls device 614 beyond (e.g., additionally and / or alternatively) controlling visual indicators 622 (e.g., using pinch to perform a zoom command and / or performing a scroll based on a two-finger swipe). In some embodiments, interface 632 includes one or more user interface objects for controlling device 614. As illustrated, interface 632 includes system actions (similar to Fig.6D One or more system enable indications 640 of system action 611).

[0234] exist Figure 6MAt , while displaying interface 632, device 600 detects input (e.g., pinch 609a and / or swipe 650m) to move visual indicator 622. In response to detecting the input to move visual indicator 622, device 600 instructs device 614 to move visual indicator 622 from mail application 642a to calendar application 642b, as shown in FIG. Figure 6N Described in.

[0235] exist Figure 6N At , device 614 displays visual indicator 622 on calendar application 642b. In some embodiments, device 614 (and / or device 600) provides an audio output 646b corresponding to calendar application 642b (e.g., a simulated voice for outputting audio corresponding to "calendar application" and / or "open calendar application").

[0236] exist Figure 6N In some embodiments, in response to a rotational input of the rotatable input device 601 and / or a gesture corresponding to (e.g., simulating) a rotational input control (e.g., a hand gesture corresponding to (simulating) a rotation of the rotatable input device 601 in a first direction and / or a rotation of the rotatable input device 601 in a second direction), the visual indicator 622 moves more quickly through the user interface. In some embodiments, when the interface 630 is displayed, the device 600 detects an input (e.g., configured to perform a rotational input of the rotatable input device 601 and / or a hand gesture of a rotational input 650n). In response to detecting the input, the device 600 instructs the device 614 to move the visual indicator 622 from the mail application 642a to the movie application 642c, as shown in FIG. Fig.6O Described in.

[0237] exist Fig.6O At 614, visual indicator 622 is displayed on movie application 642c. As illustrated, device 614 provides audio output 646 corresponding to movie application 642c (e.g., "movie application" and / or "open movie application"). While displaying interface 632, device 600 detects input corresponding to a selection (e.g., clenching 609c and / or tapping 650o). In response to device 600 detecting the input corresponding to the selection, device 600 instructs device 614 to perform the selection. Because visual indicator 622 is on movie application 642c, device 614 displays user interface 618d of movie application 642c, as shown in FIG. Figure 6P Described in.

[0238] exist Figure 6PAt , device 614 displays user interface 618d of movie application 642c. Device 614 provides audio output 646d corresponding to movie category 648 (e.g., "movies" and / or "open movie category"). In some embodiments, in response to detecting further input at device 600 (e.g., hand gestures and / or touch input), device 600 may control visual indicator 622 on device 614. In some embodiments, interface 630 includes completion affordance 644, which terminates the session of device 600 controlling device 614 using interface 632. In some embodiments, after receiving input for completion affordance 644 and / or after device 600 no longer controls device 614, device 600 communicates with device 614.

[0239] exist Figure 6Q , the embodiments described with respect to device 600 and device 614 are applicable to device 652 (e.g., a smart phone) and device 654 (e.g., a tablet computer). In some embodiments, device 652 can be used to control device 654 using similar techniques described with respect to device 600 and device 614 (e.g., user interfaces, inputs, and / or gestures (e.g., detected using one or more cameras 656, such as a camera with a user in the field of view). In some embodiments, device 652 displays interface 630 or interface 632 based on assistive settings enabled at one or more devices. As illustrated, when adaptive navigation is disabled at device 654, a screen reader is enabled at device 652. Thus, device 652 displays interface 632 to control device 654 and provides audio output 646a (e.g., "email application"). Additionally or alternatively, if the screen reader is disabled at device 652 and adaptive navigation is enabled at device 654, device 652 displays interface 630 to control device 654.

[0240] exist Figure 6QIn some embodiments, if adaptive navigation is enabled at device 652 (and / or device 600) and a screen reader is enabled at device 654 (and / or device 614), device 652 (and / or device 600) displays interface 630 (and abandons displaying interface 632) (e.g., interface 630 will take precedence over interface 632). In some embodiments, if adaptive navigation is enabled at device 652 (and / or device 600) and a screen reader is enabled at device 654 (and / or device 614), device 652 (and / or device 600) displays interface 632 (and abandons displaying interface 630) (e.g., interface 632 will take precedence over interface 630). In some embodiments, if adaptive navigation is enabled at device 652 and a screen reader is enabled at device 652, device 652 (and / or device 600) displays interface 630 (and abandons displaying interface 632) (e.g., interface 630 will take precedence over interface 632).

[0241] exist Figure 6R At , device 614 displays the screen reader and adaptive navigation as disabled. The screen reader or adaptive navigation may be enabled by user input at device 614 (e.g., using a trigger switch and / or button) under system settings (e.g., assistive settings). Figure 6R In some embodiments, when the screen reader is disabled at device 600 and adaptive navigation is disabled at device 614, device 600 can still control device 614. In such an embodiment, device 600 displays a user interface including one or more affordances (one or more system affordances 640) for controlling system actions. Additionally or alternatively, a hand gesture detected by the device can initiate a command to launch a shortcut at device 614, such as shortcut 612i and / or shortcut 614h. In some specific implementations, a hand gesture (e.g., a double pinch gesture) can initiate a command such as: answering or ending a phone call, canceling a notification, taking a photo, playing or pausing media, and starting, pausing, or resuming a workout.

[0242] Figure 7It is a flowchart illustrating a method for controlling a second computer using a first computer system according to some embodiments. Method 700 is performed at a first computer system (e.g., 100, 300, 500, 600, or 652) (e.g., a smart watch, a smart phone, a tablet, a laptop computer, and / or a head-mounted device (e.g., a head-mounted augmented reality and / or extended reality device)), which communicates with a display generation component (e.g., a display of device 600) (e.g., a display controller, a touch-sensitive display system, a monitor, and / or a head-mounted display system) and one or more input devices (e.g., a touch-sensitive surface, a keyboard, a controller, a rotatable input device, a microphone, and / or a mouse). In some embodiments, the computer system communicates with one or more sensors (e.g., one or more cameras, one or more biometric sensors (e.g., a heart rate sensor), a gyroscope, an accelerometer). Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0243] As described below, method 700 provides an intuitive way to display a user interface with device control. The method reduces the cognitive burden on the user to control the device, thereby creating a more efficient human-computer interface. For battery-powered computing devices, enabling users to control the device more quickly and more efficiently saves power and increases the time interval between battery charges.

[0244] A first computer system (e.g., 600) detects (702) input (e.g., 650i and / or 609c) corresponding to a request to control a second computer system (e.g., 614, 654) (e.g., a smart watch, a smart phone, a tablet, a laptop computer, and / or a head-mounted device (e.g., a head-mounted augmented reality and / or extended reality device)) via one or more input devices. In response to (704) detecting an input corresponding to a request to control the second computer system (e.g., to execute a command at a user interface of the second computer system and / or to navigate a user interface of the second computer system), while in communication with the second computer system (e.g., using a wired or wireless connection to the second computer system), and based on a determination that a first set of control criteria is satisfied, the first computer system displays (706) (e.g., via a display generation component) one or more first user interface objects (e.g., one or more of the user interface objects of interface 630, including 638a-c and / or 640) (e.g., user interface objects that activate a first type of input and / or a user interface object having a first size) that control a graphical element (e.g., 622) displayed by the second computer system (e.g., , cursor, pointer, and / or highlighting) (e.g., indicating that a corresponding function (e.g., opening an application, executing a function within an application, and / or closing an application) is to be performed on the second computer system, such as when a confirmation input is received) to navigate one or more user interfaces of the second computer system (e.g., 618a and / or 618b), wherein when a first mode of controlling the computer system is enabled on the second computer system (e.g., by navigating a graphical user interface and / or executing a function on the computer system), a first control criterion in a first set of control criteria is satisfied (e.g., an input setting is active, a setting that provides one or more additional (and / or alternative) options to control how input is detected, and / or a mode in which a particular input will not cause a function to be performed unless the particular input setting is active) (e.g., such as Figures 6J to 6L Enabling adaptive navigation on device 614 as described in ).

[0245] In response to (704) detecting an input corresponding to a request to control a second computer system (e.g., executing a command at a user interface of the second computer system and / or navigating a user interface of the second computer system) and while communicating with the second computer system (e.g., using a wired or wireless connection to the second computer system), and based on a determination that the first set of control criteria is not met, the first computer system abandons (708) displaying one or more user interface objects that control graphical elements to navigate one or more user interfaces (e.g., 638a-c and / or 640) of the second computer system. In some embodiments, in response to the input corresponding to the request to control the second computer system, the first computer system is wirelessly connected to the second computer system. In some embodiments, the first mode of controlling the computer system is enabled via a system setting (e.g., enabling or disabling the first mode through a system settings menu). In some embodiments, the first computer system uses one or more sensors (e.g., a biometric sensor (e.g., a heart rate sensor and / or an optical heart rate sensor), an accelerometer, and / or a gyroscope) to detect and distinguish various gestures performed by a user's hand (and / or other body part) when the first computer system is worn by the user (e.g., on the user's wrist and / or other body part). In some embodiments, various gestures may be used to initiate (e.g., send and / or transmit) one or more commands to be executed by the second computer system. In some embodiments, the various gestures do not include input at a button or touch-sensitive surface of the computer system. In some embodiments, one or more user interface objects that control a graphical element displayed at the second computer system are not displayed at the second computer system (and / or are displayed only at the first computer system). In some embodiments, the first computer system (and / or the second computer system) displays an indication (e.g., a notification and / or a banner) (e.g., a non-persistent indication and / or a persistent indication) that the first computer system (and / or the second computer system) is controlling the second computer system (and / or the first computer system). In some embodiments, the first mode of the second computer system is enabled independently of communication with the first computer system. In some embodiments, the first mode of the second computer system is not enabled in response to the second computer system communicating with the first computer system. In some embodiments, the first mode of the second computer system is enabled before the second computer system communicates with the first computer system. In some embodiments, the first mode of the second computer system is enabled after the second computer system communicates with the first computer system. In some embodiments, the first mode of the second computer system is manually enabled (e.g., by a user). In some embodiments, the first mode of the second computer system is enabled locally on the second computer system (and / or cannot be enabled via the first computer system).Automatically displaying specific controls when assistive settings are enabled on the second computer system allows the user to better control the second device when the user has a physical impairment, and provides the user with visual feedback about the status of the second computer system, which enhances the user-machine interface for users with physical impairments and provides improved visual feedback about the status of the second computer system.

[0246] In some embodiments, in response to detecting an input corresponding to a request to control a second computer system and while communicating with the second computer system: based on a determination that a first set of control criteria is not met, the first computer system displays, via a display generation component, a control user interface object (e.g., 640) (e.g., a selectable user interface object and / or a trigger switch) (e.g., one or more) that controls a function of the second computer system (e.g., navigating to a home screen, viewing one or more open applications (e.g., an application switcher and / or an application navigation view), and / or opening a notification center) (e.g., as Fig.6D Depicted in and as referenced Figure 6J In some embodiments, when the first set of control criteria is met, a user interface object that controls a function of the second computer system is displayed. Automatically displaying controls when the auxiliary setting is not enabled on the second computer system allows the user to better control the second computer system to effectively perform different operations at the second computer system, which enhances the user-machine interface and provides improved visual feedback about the status of the second computer system.

[0247] In some embodiments, in response to detecting an input corresponding to a request to control a second computer system, and while communicating with the second computer system: based on a determination that a second set of control criteria are met, the first computer system displays, via a display generation component, one or more second user interface objects (e.g., one or more of the interface objects of interface 632, including 633 and / or 640) (e.g., user interface objects activated using a second type of input and / or user interface objects having a second size), which control a graphical element (e.g., 622) displayed by the second computer system, wherein the one or more second user interface objects are different from the one or more first user interface objects (e.g., the second user interface objects are activated using at least one type of input that does not activate the first user interface object (or vice versa), and / or the second user interface objects have a size that is different from (larger than or smaller than) the first user interface objects), and wherein the second set of control criteria includes the first control criteria that, when the first computer system (e.g., such as Figure 6M to Figure 6QThe first control criteria are met when a second mode of controlling the computer system that is different from the first mode (e.g., by navigating a graphical user interface and / or performing functions on the computer system) is enabled on the device 600 or device 652 (e.g., an input setting is active, a setting that provides one or more additional (and / or alternative) options to control how input is detected, and / or a mode in which a particular input does not cause a function to be performed unless a particular input setting is active). In some embodiments, the screen reader is enabled when the computer system is in the second mode. In some embodiments, the screen reader is not enabled when the computer system is in the first mode. In some embodiments, the computer system displays a second option (e.g., an enable representation and / or a trigger switch) (e.g., the same as the first option and / or different from the first option) that controls a second function of the computer system (e.g., navigating to a home screen, viewing one or more open applications (e.g., an "app switcher" and / or an application navigation view), and / or opening a notification center). In some embodiments, the second option is displayed when the second set of control criteria is met. Automatically displaying specific controls when specific assistive settings (e.g., a screen reader or text reader) are enabled on the first computer system allows the user to better control the second computer system when the user has a physical impairment, and provides the user with visual feedback about the status of the second computer system, which enhances the user-machine interface for users with physical impairments and provides improved visual feedback about the status of the first computer system.

[0248] In some embodiments, the first set of control criteria includes a second control criterion, and the second control criterion is met when the second set of control criteria is not met. In other words, the first set of control criteria is not met, and therefore, when the second set of control criteria is met, one or more first user interface objects are not displayed. Not displaying specific controls (e.g., non-screen reader and / or non-text reader controls) at the first computer system when specific settings (e.g., screen reader or text reader) are enabled on the first computer system reduces the number of controls displayed at the first computer system, which enhances the user-machine interface for users with physical disabilities and tidies up the user interface at the first computer system.

[0249] In some embodiments, the second set of control criteria does not include control criteria based on whether the second mode of controlling the computer system is enabled (or optionally disabled) on the second computer system. In some embodiments, the second set of control criteria is met or not met, regardless of whether the second mode of controlling the computer system is enabled on the second computer system. Displaying specific controls when an assistive setting (e.g., a screen reader or text reader) is enabled on the first computer system allows the user to better control the second computer system because it prioritizes local assistive settings and how the user is using the first computer system based on his or her physical impairment, which enhances the user-machine interface for users with physical impairments and provides improved visual feedback about the status of the first computer system.

[0250] In some embodiments, in response to detecting an input corresponding to a request to control a second computer system, and while communicating with the second computer system: based on a determination that the second set of control criteria is satisfied (and / or optionally, based on a determination that the second mode is disabled on the second computer system), the first computer system initiates a command (e.g., initiates sending and / or instructing) to enable the second mode on the second computer system (e.g., device 600 initiates a command to enable the second mode on the second computer system). Figure 6M When a particular assistive setting is enabled on the first computer system, enabling the particular assistive setting (e.g., a screen reader or text reader) at the remote device allows the user to control the second computer system based on how the user is controlling the first computer system, which enhances the user-machine interface for users with physical impairments.

[0251] In some embodiments, when the second mode is enabled on the second computer system, one or more second user interface objects are not displayed at the second computer system (e.g., user interface objects of interface 632 are not displayed on device 614 and / or device 654). Not displaying control options displayed at the first computer system at the remote device allows the user to control the second computer system and limits the amount of control options displayed at the second computer system, which provides additional control options without cluttering the user interface.

[0252] In some embodiments, the second set of control criteria does not include criteria based on whether the first mode is enabled on the second computer system (e.g., interface 632 is displayed at device 600 and / or device 652 regardless of whether adaptive navigation is enabled at device 614 and / or device 654) (e.g., one or more second user interface objects are displayed independent of satisfying the first set of criteria). Displaying controls independent of specific accessibility settings enabled on the second computer system allows the user to better control the second computer system because the first computer system displays one or more second user interface objects regardless of whether the first mode is enabled on the second computer system, which enhances the user-machine interface for users with physical impairments and provides improved visual feedback about the state of the first computer system.

[0253] In some embodiments, where the first set of control criteria includes a third criterion for enabling the second mode on the first computer system and enabling the first mode on the second computer system (e.g., when a screen reader is enabled on device 600 and adaptive navigation is enabled on device 614, display 638a-c and / or 640). In some embodiments, the second set of control criteria includes criteria for enabling the second mode at the first computer system and enabling the first mode on the second computer system. Displaying controls for specific accessibility settings when two different modes are enabled on the respective devices allows the user to better control the second computer system because it prioritizes display of specific interface objects at the first computer system based on how the user is using the second computer system, which enhances the user-machine interface for users with physical impairments and provides improved visual feedback about the status of the second computer system.

[0254] In some embodiments, displaying one or more second user interface objects includes displaying a touch input area (e.g., 633) (e.g., a touchpad area and / or a trackpad area). In some embodiments, when the touch input area is displayed, the first computer system detects a gesture at the touch input area via one or more input devices (e.g., 650m). In response to detecting the gesture at the touch input area, the first computer system sends an indication of the gesture to the second computer system (e.g., swiping 650 causes device 614 to move visual indicator 622, such as Figures 6M to 6NIn some embodiments, the touch input area is used to control the second computer system using motion-based gestures (e.g., swipes and / or rotations). In some embodiments, the touch input area (e.g., for detecting motion-based gestures) allows detection of multi-finger inputs (e.g., two-finger taps and / or three-finger swipes). Providing a touch input area at a first computer system that receives inputs typically received at a remote device allows the user to better control the second computer system through motion-based gestures received at the first computer system, which enhances the user-machine interface for users with physical impairments.

[0255] In some embodiments, in response to detecting an input corresponding to a request to control a second computer system, and while communicating with the second computer system, the first computer system abandons displaying a touch input area (e.g., interface 630 does not include touchpad area 633) based on a determination that a first set of control criteria is satisfied. In some embodiments, one or more first user interface objects include an affordance for controlling graphical elements displayed by the second computer system that are not included in the one or more second user interface objects. Not providing a touchpad area at the first computer system when a particular accessibility setting (e.g., an accessibility setting other than a screen reader setting) is enabled on the second computer system limits the number of unnecessary graphical objects based on the user's physical impairment, which enhances the user-machine interface for users with physical impairments by including relevant controls.

[0256] In some embodiments, the first computer system includes a rotatable input device (e.g., 601) (e.g., a rotatable input mechanism and / or a crown). In some embodiments, the first computer system detects a rotational input (e.g., 650n) via a rotatable input device. In response to detecting the rotational input, the first computer system sends a scroll instruction to the second computer system to scroll the content displayed at the second computer system (e.g., scrolling the content displayed on the device 614 and / or when the indicator 622 moves through the content of the user interface displayed on the device 614, the device 614 scrolls the displayed content). In some embodiments, the scroll instruction includes a scroll direction (first direction to second direction), a scroll rate (first rate to second rate) and / or a scroll duration (first duration to second duration). In some embodiments, in response to detecting the rotational input, the computer system sends an instruction to the second computer system to move a graphical element displayed by the second computer system. Scrolling items on the second computer system in response to rotation of a crown or dial at the first computer system allows the user to better control the second computer system because it provides an efficient way to quickly move graphical indicators through a remote user interface, which enhances the user-machine interface for users with physical impairments and provides additional control options without cluttering the user interface.

[0257] In some embodiments, when displaying one or more first user interface objects, the first computer system detects non-touch hand gestures (e.g., such as a touch gesture) via one or more sensors (e.g., a biometric sensor (e.g., a heart rate sensor and / or an optical heart rate sensor), an accelerometer, and / or a gyroscope). Figures 6J to 6L In response to detecting the non-touch hand gesture, the first computer system initiates a command (e.g., initiates sending and / or instructs the second computer system) to perform an operation at the second computer system (e.g., such as performing an air gesture such as an air pinch or clenching of the hand). Figures 6J to 6K The moving visual indicator 622 described in and / or as Figures 6K to 6L ) (moving a graphical indicator, making a selection, and / or opening a notification center). In some embodiments, the hand gesture does not include an input at a button or touch-sensitive surface of a computer system. Controlling a second computer system in response to detecting a gesture made by a user at a first computer system (e.g., an air pinch or clench) provides the user with additional control options without requiring the user to press buttons and / or touch-sensitive surfaces, which enhances the user-machine interface for users with physical disabilities and provides additional control options without cluttering the user interface.

[0258] In some embodiments, initiating a command to perform an operation includes: based on a determination that the non-touch hand gesture is a first type of non-touch hand gesture (e.g., 609a), the first computer system sends a first command (e.g., Figures 6J to 6K and based on a determination that the non-touch hand gesture is a second type of non-touch hand gesture different from the first type of non-touch hand gesture (e.g., 609c), the first computer system sends a second command to the second computer system (e.g., Figures 6K to 6L Performing a different operation at a second computer system in response to detecting a gesture (e.g., an air pinch or clench) made by a user at a first computer system provides additional control options to the user without requiring the user to press buttons and / or touch-sensitive surfaces to perform the same operation at the second computer system, which enhances the user-machine interface for users with physical impairments and provides additional control options without cluttering the user interface.

[0259] In some embodiments, the first computer system displays, via a display generation component, a plurality of options for controlling the plurality of computer systems (e.g., 626), including a first option for controlling the second computer system (e.g., Fig.6I a telephone option in the example above) and a second option to control a third computer system different from the first computer system and the second computer system (e.g., Fig.6I In some embodiments, detecting an input corresponding to a request to control a second computer system includes detecting a selection of a first option (e.g., 650i and / or 609c) to control the second computer system via one or more input devices. Displaying multiple computer systems to be controlled provides the user with the ability to select which computer system the user wants to control using corresponding auxiliary settings, which enhances the user-machine interface for users with physical disabilities and provides additional control options.

[0260] In some embodiments, displaying multiple options for controlling multiple computer systems includes: based on the determination that the corresponding computer system meets a set of one or more display conditions, the first computer system displays the option of controlling the corresponding computer system via the display generation component. In some embodiments, the set of one or more display conditions includes a first display condition, which is satisfied when the corresponding computer system is logged into the same user account as the first computer system (e.g., device 600 is logged into the same user account as device 614, device 600 is logged into the same user account as the tablet computer associated with the tablet option displayed in the device selector menu 626, and / or device 600 is logged into the same user account as the computer associated with the computer option displayed in the device selector menu 626). In some embodiments, displaying multiple options for controlling multiple computer systems includes: based on the determination that the corresponding computer system does not meet the set of one or more display conditions, the first computer system abandons the option of displaying the corresponding computer system (e.g., devices not logged into the same user account are not displayed in the device selector menu 626). Displaying multiple computer systems based on whether the devices are logged into the same user account provides the user with the ability to select which computer system the user wants to control, which enhances the user-machine interface for users with physical disabilities and provides additional control options.

[0261] In some embodiments, the set of one or more display conditions includes a second display condition that is satisfied when the corresponding computer system is within a threshold distance of the first computer system (e.g., device 600 is within 200 feet of device 614). In some embodiments, the first computer system receives an indication that the corresponding computer system is within a threshold distance of the first computer system (e.g., from a server, from a global positioning satellite, and / or from a corresponding computer system) (e.g., using a local area network and / or Bluetooth). Displaying multiple devices based on whether the device is within a threshold distance of the computer system provides the user with the ability to select nearby devices that the user wants to control and limits the number of inputs to identify nearby devices, which enhances the user-machine interface for users with physical disabilities and reduces the number of inputs required to perform operations.

[0262] In some embodiments, the first computer system receives user input via one or more input devices that corresponds a user-specified non-touch hand gesture (e.g., 609b, 609d, and / or 609c) (e.g., an air gesture and / or a gesture detected independently of a touch-sensitive surface of the first computer system) to a first function (e.g., rotation) of a rotatable input device of the first computer system (e.g., via rotation of rotatable input device 601 associated with enable representation 612f and / or enable representation 612g, device 600 receives input corresponding to one of the gestures associated with enable representations 608a-d), wherein the user-specified non-touch hand gesture does not use the rotatable input device of the first computer system. In some embodiments, when the user-specified non-touch hand gesture corresponds to the first function of the rotatable input device of the first computer system: the first computer system receives the user-specified non-touch hand gesture via one or more input devices. In response to receiving a user-specified non-touch hand gesture, the first computer performs a first operation (e.g., a scrolling operation) corresponding to a first function of a rotatable input device of the first computer system (e.g., if the computer system detects input using the first function of the rotatable input device (e.g., rotation), the first computer system will perform the same operation). In some embodiments, after performing the first operation corresponding to the first function of the rotatable input device, the first computer system receives user input via one or more input devices that corresponds the user-specified non-touch hand gesture (e.g., an air gesture and / or a gesture detected independently of the touch-sensitive surface of the first computer system) to a second function of the rotatable input device of the first computer system that is different from the first function (e.g., a rotation or press input) (e.g., by pressing (e.g., depressing) the rotatable input device 601 associated with the enable representation 612g, the device 600 receives input corresponding to one of the gestures associated with the enable representation 608a-d), wherein the user-specified non-touch hand gesture does not use the rotatable input device of the first computer system. In some embodiments, when the user-specified non-touch hand gesture corresponds to the second function of the rotatable input device of the first computer system: the first computer system receives the user-specified non-touch hand gesture via one or more input devices. In response to receiving the user-specified non-touch hand gesture, the computer system performs a second operation corresponding to the second function of the rotatable input device of the first computer system (e.g., a press input) (e.g., if the computer system detects input using the second function of the rotatable input device (e.g., a press input), the computer system will perform the same operation). In some embodiments, the computer system receives user input to confirm a non-touch hand gesture that does not use the rotatable input device to mimic (or replace) the function of the rotatable input device.Customizing hand gestures (e.g., air pinches or squeezes) made by the user to perform operations typically detected by a crown or dial provides the user with an alternative way to provide input to control the device (either locally or remotely), which enhances the user-machine interface for users with physical impairments and provides additional control options.

[0263] For the purpose of explanation, the preceding description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to best explain the principles of these technologies and their practical applications. Others skilled in the art can thus best utilize these technologies and various embodiments with various modifications suitable for the intended specific use.

[0264] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the disclosure and examples defined by the claims.

[0265] As described above, one aspect of the present technology is to collect and use data from various sources to improve the delivery of inspirational content or any other content that may be of interest to users. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identification or personal information.

[0266] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data can be used to customize user profiles and / or improve what auxiliary settings are provided to users. Therefore, the use of such personal information data enables users to have planned control over the content delivered. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0267] The disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information data will comply with perfect privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be convenient for users to access and should be updated with changes in the collection and / or use of data. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside these legal purposes. In addition, such collection / sharing should be carried out after receiving the informed consent of the user. Additionally, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data, and ensure that other entities with access to personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0268] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates hardware elements and / or software elements to prevent or block access to such personal information data. For example, with respect to auxiliary settings, the present technology may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for a service. In another example, a user may choose not to provide auxiliary settings data for a target content delivery service. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that the user's personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0269] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting data collection and data deletion. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0270] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, information about auxiliary settings may be selected and delivered to a user by inferring preferences based on non-personal information data or an absolute minimum of personal information (e.g., content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information).

Claims

1. A method, comprising: At a first computer system in communication with a display generating component and one or more input devices: detecting, via the one or more input devices, an input corresponding to a request to control a second computer system; and In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system: displaying one or more first user interface objects that control graphical elements displayed by the second computer system to navigate one or more user interfaces of the second computer system based on determining that a first set of control criteria is satisfied, wherein a first control criterion in the first set of control criteria is satisfied when a first mode of controlling a computer system is enabled on the second computer system; and Based on determining that the first set of control criteria is not satisfied, displaying the one or more first user interface objects that control the graphical elements to navigate the one or more user interfaces of the second computer system is abandoned.

2. The method according to claim 1, further comprising: In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system: Based on determining that the first set of control criteria is not met, displaying, via the display generation component, a control user interface object that controls a function of the second computer system.

3. The method according to any one of claims 1 to 2, further comprising: In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system: Based on determining that a second set of control criteria are satisfied, one or more second user interface objects that control graphical elements displayed by the second computer system are displayed via the display generation component, wherein the one or more second user interface objects are different from the one or more first user interface objects, and wherein the second set of control criteria includes a first control criterion that is satisfied when a second mode of controlling the computer system that is different from the first mode is enabled on the first computer system.

4. The method according to claim 3, wherein: The first set of control criteria includes a second control criterion that is met when the second set of control criteria is not met.

5. The method according to any one of claims 3 to 4, wherein: The second set of control criteria does not include a control criterion based on whether the second mode of controlling the computer system is enabled on the second computer system.

6. The method according to any one of claims 3 to 5, further comprising: In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system: Based on determining that the second set of control criteria is satisfied, a command is initiated to enable the second mode on the second computer system.

7. The method according to claim 6, wherein: When the second mode is enabled on the second computer system, the one or more second user interface objects are not displayed at the second computer system.

8. The method according to any one of claims 3 to 7, wherein: The second set of control criteria does not include criteria based on whether the first mode is enabled on the second computer system.

9. The method according to any one of claims 3 to 8, wherein: The first set of control criteria includes a third criterion for enabling the second mode on the first computer system and enabling the first mode on the second computer system.

10. The method according to any one of claims 3 to 7, wherein: Displaying the one or more second user interface objects includes displaying a touch input area, the method further comprising: detecting, via the one or more input devices, a gesture at the touch input area while the touch input area is displayed; and In response to detecting the gesture at the touch input area, an indication of the gesture is sent to the second computer system.

11. The method according to claim 10, further comprising: In response to detecting the input corresponding to the request to control the second computer system, and in communication with the second computer system, based on determining that the first set of control criteria is satisfied, displaying the touch input area is relinquished.

12. The method according to any one of claims 3 to 11, wherein: The first computer system includes a rotatable input device, the method further comprising: detecting a rotational input via the rotatable input device; and In response to detecting the rotational input, a scroll instruction is sent to the second computer system to scroll content displayed at the second computer system.

13. The method according to any one of claims 1 to 12, further comprising: detecting, via one or more sensors, a touchless hand gesture while displaying the one or more first user interface objects; as well as In response to detecting the touchless hand gesture, a command is initiated to perform an operation at the second computer system.

14. The method according to claim 13, wherein: Initiating the command for performing the operation includes: Based on determining that the non-touch hand gesture is a first type of non-touch hand gesture, sending a first command to the second computer system; and Based on determining that the touchless hand gesture is a second type of touchless hand gesture different from the first type of touchless hand gesture, a second command is sent to the second computer system.

15. The method according to any one of claims 1 to 14, further comprising: displaying, via the display generating component, a plurality of options for controlling a plurality of computer systems, the plurality of options including a first option for controlling the second computer system and a second option for controlling a third computer system different from the first computer system and the second computer system; and Wherein detecting the input corresponding to a request to control the second computer system comprises detecting, via the one or more input devices, a selection of the first option to control the second computer system.

16. The method according to claim 15, wherein: Displaying the plurality of options for controlling the plurality of computer systems comprises: displaying, via the display generating component, options for controlling the corresponding computer system based on determining that the corresponding computer system satisfies a set of one or more display conditions, wherein the set of one or more display conditions includes a first display condition that is satisfied when the corresponding computer system is logged into the same user account as the first computer system; and Based on determining that the corresponding computer system does not satisfy the set of one or more display conditions, the option of displaying and controlling the corresponding computer system is abandoned.

17. The method according to claim 16, wherein: The set of one or more display conditions includes a second display condition that is satisfied when the corresponding computer system is within a threshold distance of the first computer system.

18. The method according to any one of claims 1 to 17, further comprising: receiving user input via the one or more input devices to correspond a user-specified touchless hand gesture to a first function of the rotatable input device of the first computer system, wherein the user-specified touchless hand gesture does not utilize the rotatable input device of the first computer system; When the user-specified non-touch hand gesture corresponds to the first function of the rotatable input device of the first computer system: receiving, via the one or more input devices, a touchless hand gesture specified by the user; and In response to receiving the user-specified touchless hand gesture, performing a first operation corresponding to the first function of the rotatable input device of the first computer system; After performing the first operation corresponding to the first function of the rotatable input device, receiving user input via the one or more input devices to correspond the user-specified touchless hand gesture to a second function of the rotatable input device of the first computer system that is different from the first function, wherein the user-specified touchless hand gesture does not use the rotatable input device of the first computer system; and When the user-specified non-touch hand gesture corresponds to the second function of the rotatable input device of the first computer system: receiving, via the one or more input devices, a touchless hand gesture specified by the user; and In response to receiving the user-specified touchless hand gesture, a second operation corresponding to the second function of the rotatable input device of the first computer system is performed.

19. The method according to any one of claims 1 to 17, further comprising: receiving user input via the one or more input devices to correspond a user-specified non-touch hand gesture to a first set of commands for one or more applications and / or one or more settings, wherein the user-specified non-touch hand gesture does not include a touch input; When the user-specified touchless hand gesture corresponds to the first set of commands for the one or more applications and / or the one or more settings: receiving, via the one or more input devices, a touchless hand gesture specified by the user; and in response to receiving the user-specified touchless hand gesture, performing a first set of operations corresponding to the first set of commands for the one or more applications and / or the one or more settings; After performing the first set of operations corresponding to the first set of commands for the one or more applications and / or the one or more settings, receiving user input via the one or more input devices to correspond the user-specified touchless hand gesture to a second set of commands for the one or more applications and / or one or more settings that is different from the first set of commands, wherein the user-specified touchless hand gesture does not include touch input; and When the user-specified touchless hand gesture corresponds to the second set of commands for the one or more applications and / or the one or more settings: receiving, via the one or more input devices, a touchless hand gesture specified by the user; and In response to receiving the user-specified touchless hand gesture, a second set of operations corresponding to the second set of commands for the one or more applications and / or the one or more settings are performed.

20. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first computer system in communication with a display generating component and one or more input devices, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 19.

21. A first computer system, the first computer system being configured to communicate with a display generation component and one or more input devices, the first computer system comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 19.

22. A first computer system, the first computer system being configured to communicate with a display generation component and one or more input devices, the first computer system comprising: A component for carrying out the method according to any one of claims 1 to 19.

23. A computer program product comprising one or more programs configured to be executed by one or more processors of a first computer system in communication with a display generating component and one or more input devices, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 19.

24. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for: detecting, via the one or more input devices, an input corresponding to a request to control a second computer system; and In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system: Based on determining that the first set of control criteria is satisfied, displaying one or more first user interface objects that control graphical elements displayed by the second computer system to navigate one or more user interfaces of the second computer system, wherein a first control criterion in the first set of control criteria is satisfied when a first mode of controlling a computer system is enabled on the second computer system; as well as Based on determining that the first set of control criteria is not satisfied, displaying the one or more first user interface objects that control the graphical elements to navigate the one or more user interfaces of the second computer system is abandoned.

25. A first computer system, the first computer system being configured to communicate with a display generation component and one or more input devices, the first computer system comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: detecting, via the one or more input devices, an input corresponding to a request to control a second computer system; and In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system: displaying one or more first user interface objects that control graphical elements displayed by the second computer system to navigate one or more user interfaces of the second computer system based on determining that a first set of control criteria is satisfied, wherein a first control criterion in the first set of control criteria is satisfied when a first mode of controlling a computer system is enabled on the second computer system; as well as Based on determining that the first set of control criteria is not satisfied, displaying the one or more first user interface objects that control the graphical elements to navigate the one or more user interfaces of the second computer system is abandoned.

26. A first computer system, the first computer system being configured to communicate with a display generation component and one or more input devices, the first computer system comprising: means for detecting, via the one or more input devices, an input corresponding to a request to control a second computer system; and In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system, means for: displaying one or more first user interface objects that control graphical elements displayed by the second computer system to navigate one or more user interfaces of the second computer system based on determining that a first set of control criteria is satisfied, wherein a first control criterion in the first set of control criteria is satisfied when a first mode of controlling a computer system is enabled on the second computer system; as well as Based on determining that the first set of control criteria is not satisfied, displaying the one or more first user interface objects that control the graphical elements to navigate the one or more user interfaces of the second computer system is abandoned.

27. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for: detecting, via the one or more input devices, an input corresponding to a request to control a second computer system; and In response to detecting the input corresponding to the request to control the second computer system, and while in communication with the second computer system: Based on determining that the first set of control criteria is satisfied, displaying one or more first user interface objects that control graphical elements displayed by the second computer system to navigate one or more user interfaces of the second computer system, wherein a first control criterion in the first set of control criteria is satisfied when a first mode of controlling a computer system is enabled on the second computer system; as well as Based on determining that the first set of control criteria is not satisfied, displaying a first graphical element that controls navigation of the one or more first user interface objects of the one or more user interfaces of the second computer system is abandoned.

Images