Navigating a user interface using hand gestures

Abstract

The present disclosure relates generally to navigating user interfaces using hand gestures.

Description

[0001] This application is a divisional application of the invention patent application with an international filing date of May 19, 2022, national application number 202280005772.7 (international application number PCT / US2022 / 030021) and an invention title of “Navigation User Interface Using Hand Gestures”.

[0002] Cross-reference application

[0003] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 190,783, filed May 19, 2021, entitled “NAVIGATING USER INTERFACES USING HAND GESTURES”; U.S. Provisional Patent Application Serial No. 63 / 221,331, filed July 13, 2021, entitled “NAVIGATING USER INTERFACES USING HAND GESTURES”; and U.S. Patent Application Serial No. 17 / 747,613, filed May 18, 2022, entitled “NAVIGATING USER INTERFACES USING HAND GESTURES”. The contents of these patent applications are incorporated herein by reference in their entirety. Technical Field

[0004] This disclosure relates in general to computer user interfaces, and more specifically to techniques for navigating user interfaces using hand gestures. Background Technology

[0005] Users of smartphones and other personal electronic devices are using their devices more frequently. Some existing technologies allow users to navigate the user interface on their devices. Summary of the Invention

[0006] However, some techniques for using hand gestures to navigate user interfaces on electronic devices are often cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may involve multiple keystrokes or button presses. These existing technologies require a significant amount of time, resulting in wasted user time and device power. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, this technology provides users of electronic devices with a faster and more efficient method and interface for navigating user interfaces using hand gestures. Such methods and interfaces optionally complement or replace other methods for navigating user interfaces using hand gestures. These methods and interfaces reduce the cognitive burden on users and result in a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charging sessions.

[0008] According to some embodiments, a method is described. The method is performed at a computer system communicating with a display generation component and an optical sensor. The method includes: displaying a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected via the display generation component; while displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting a hand gesture at least via the optical sensor; and in response to detecting the hand gesture at least via the optical sensor: based on determining that the hand gesture is a first type of gesture, displaying an indication that a second user interface object is selected via the display generation component; and based on determining that the hand gesture is a second type of gesture different from the first type of gesture, displaying an indication that a third user interface object is selected via the display generation component.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and an optical sensor. The one or more programs include instructions for performing the following operations: displaying a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected via the display generating component; detecting a hand gesture at least via the optical sensor while displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected; and in response to detecting a hand gesture at least via the optical sensor: displaying an indication that the second user interface object is selected via the display generating component based on determining that the hand gesture is a first type of gesture; and displaying an indication that the third user interface object is selected via the display generating component based on determining that the hand gesture is a second type of gesture different from the first type of gesture.

[0010] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and an optical sensor. The one or more programs include instructions for performing the following operations: displaying a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected via the display generating component; detecting a hand gesture at least via the optical sensor while displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected; and in response to detecting a hand gesture at least via the optical sensor: displaying an indication that the second user interface object is selected via the display generating component based on determining that the hand gesture is a first type of gesture; and displaying an indication that the third user interface object is selected via the display generating component based on determining that the hand gesture is a second type of gesture different from the first type of gesture.

[0011] According to some embodiments, a computer system is described. The computer system includes: a display generation component; an optical sensor; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying via the display generation component a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected; detecting a hand gesture at least via the optical sensor while displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected; and in response to detecting the hand gesture at least via the optical sensor: displaying via the display generation component an indication that the second user interface object is selected based on determining that the hand gesture is a first type of gesture; and displaying via the display generation component an indication that the third user interface object is selected based on determining that the hand gesture is a second type of gesture different from the first type of gesture.

[0012] According to some embodiments, a computer system is described. The computer system includes: a display generation component; an optical sensor; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: means for displaying, via the display generation component, a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected; means for detecting a hand gesture, at least via the optical sensor, while displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected; and means for performing the following operations in response to detecting a hand gesture, at least via the optical sensor: displaying, via the display generation component, an indication that the second user interface object is selected, based on determining that the hand gesture is a first type of gesture; and displaying, via the display generation component, an indication that the third user interface object is selected, based on determining that the hand gesture is a second type of gesture different from the first type of gesture.

[0013] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and an optical sensor. The one or more programs include instructions for performing the following operations: displaying a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected via the display generation component; detecting a hand gesture at least via the optical sensor while displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected; and in response to detecting a hand gesture at least via the optical sensor: displaying an indication that the second user interface object is selected via the display generation component based on determining that the hand gesture is a first type of gesture; and displaying an indication that the third user interface object is selected via the display generation component based on determining that the hand gesture is a second type of gesture different from the first type of gesture.

[0014] According to some embodiments, a method is described. The method is performed at a computer system communicating with a display generation component. The method includes: displaying a user interface including an optional user interface object and a cursor displayed at a first position on the user interface via the display generation component; while displaying the optional user interface object and the cursor at the first position on the user interface, detecting a request to move the cursor from the first position on the user interface to a second position; and in response to detecting the request to move the cursor from the first position to the second position: displaying the cursor at the second position; based on determining that the second position corresponds to the position of the optional user interface object, displaying an animation providing a visual indication of how long the cursor needs to be positioned at the second position to perform an operation, wherein the visual indication is updated within a certain time period; and based on determining that the second position does not correspond to the position of the optional user interface object, abandoning the display of the animation.

[0015] According to some embodiments, a non-transitory computer-readable storage medium is described. This non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component. The one or more programs include instructions for performing the following operations: displaying a user interface including an optional user interface object and a cursor displayed at a first position on the user interface via the display generation component; detecting a request to move the cursor from the first position to a second position on the user interface while the optional user interface object and the cursor at the first position are displayed; and in response to detecting the request to move the cursor from the first position to the second position: displaying the cursor at the second position; displaying an animation providing a visual indication of how long the cursor needs to be positioned at the second position to perform an operation, wherein the visual indication is updated within a certain time period, based on the determination that the second position does not correspond to the position of the optional user interface object; and abandoning the display of the animation based on the determination that the second position does not correspond to the position of the optional user interface object.

[0016] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component. The one or more programs include instructions for performing the following operations: displaying a user interface including an optional user interface object and a cursor displayed at a first position on the user interface via the display generation component; detecting a request to move the cursor from the first position to a second position on the user interface while the optional user interface object and the cursor at the first position are displayed; and in response to detecting the request to move the cursor from the first position to the second position: displaying the cursor at the second position; displaying an animation providing a visual indication of how long the cursor needs to be positioned at the second position to perform an operation, wherein the visual indication is updated within a certain time period, based on determining that the second position does not correspond to the position of the optional user interface object; and abandoning the display of the animation based on determining that the second position does not correspond to the position of the optional user interface object.

[0017] According to some embodiments, a computer system is described. The computer system includes: a display generation component; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a user interface including optional user interface objects and a cursor displayed at a first position on the user interface via the display generation component; detecting a request to move the cursor from the first position on the user interface to a second position while the optional user interface objects and the cursor at the first position on the user interface are displayed; and in response to detecting the request to move the cursor from the first position to the second position: displaying the cursor at the second position; displaying an animation providing a visual indication of how long the cursor needs to be positioned at the second position to perform an operation, wherein the visual indication is updated within a certain time period, based on determining that the second position does not correspond to the position of the optional user interface objects; and abandoning the display of the animation based on determining that the second position does not correspond to the position of the optional user interface objects.

[0018] According to some embodiments, a computer system is described. The computer system includes: a display generation component; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: means for displaying via the display generation component a user interface including an optional user interface object and a cursor displayed at a first position on the user interface; means for detecting a request to move the cursor from the first position on the user interface to a second position when the optional user interface object and the cursor at the first position on the user interface are displayed; and means for performing the following operations: in response to detecting the request to move the cursor from the first position to the second position: displaying the cursor at the second position; displaying an animation providing a visual indication of how long the cursor needs to be positioned at the second position to perform an operation, wherein the visual indication is updated within a certain time period, based on determining that the second position does not correspond to the position of the optional user interface object; and abandoning the display of the animation based on determining that the second position does not correspond to the position of the optional user interface object.

[0019] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component. The one or more programs include instructions for performing the following operations: displaying a user interface including an optional user interface object and a cursor displayed at a first position on the user interface via the display generation component; detecting a request to move the cursor from the first position on the user interface to a second position while the optional user interface object and the cursor at the first position on the user interface are displayed; and in response to detecting the request to move the cursor from the first position to the second position: displaying the cursor at the second position; displaying an animation providing a visual indication of how long the cursor needs to be positioned at the second position to perform an operation, wherein the visual indication is updated within a certain time period, based on determining that the second position does not correspond to the position of the optional user interface object; and abandoning the display of the animation based on determining that the second position does not correspond to the position of the optional user interface object.

[0020] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0021] Therefore, faster and more efficient methods and interfaces are provided for devices to navigate user interfaces using hand gestures, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. These methods and interfaces can complement or replace other methods used for navigating user interfaces using hand gestures. Attached Figure Description

[0022] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0023] Figure 1A This is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some embodiments.

[0024] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some implementation schemes.

[0025] Figure 2 A portable multi-functional device with a touchscreen is shown according to some embodiments.

[0026] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some implementation schemes.

[0027] Figure 4A An exemplary user interface for a menu on an application on a portable multifunction device, according to some implementation schemes, is shown.

[0028] Figure 4B An exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0029] Figure 5A A personal electronic device according to some implementation schemes is shown.

[0030] Figure 5B This is a block diagram illustrating a personal electronic device according to some implementation schemes.

[0031] Figure 6 A set of exemplary hand gestures according to some implementation schemes are shown.

[0032] Figures 7A to 7AA An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0033] Figures 8A to 8J An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0034] Figures 9A to 9HAn exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0035] Figures 10A to 10F An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0036] Figures 11A to 11H An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0037] Figures 12A to 12J An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0038] Figures 13A to 13G An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0039] Figure 14 An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown.

[0040] Figure 15 This is a flowchart illustrating a method for navigating a user interface using hand gestures, according to some implementation schemes.

[0041] Figure 16 This is a flowchart illustrating a method for navigating a user interface using hand gestures, according to some implementation schemes. Detailed Implementation

[0042] The following description illustrates exemplary methods, parameters, etc. However, it should be understood that such description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.

[0043] There is a need for electronic devices that provide efficient methods and interfaces for navigating user interfaces using hand gestures. For example, a user may want to navigate the user interface without touching their device's display. Such technologies can reduce the cognitive and / or physical burden on users navigating the user interface, thereby increasing productivity. Furthermore, such technologies can reduce processor and battery power that would otherwise be wasted on redundant user input.

[0044] under Figures 1A to 1B , Figure 2 , Figure 3 , Figures 4A to 4B and Figures 5A to 5B A description is provided of an exemplary device for implementing techniques for navigating a user interface using hand gestures. Figure 6 A set of exemplary hand gestures according to some implementation schemes are shown. Figures 7A to 7AAAn exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figures 8A to 8J An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figures 9A to 9H An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figures 10A to 10F An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figures 11A to 11H An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figures 12A to 12J An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figures 13A to 13G An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figure 14 An exemplary user interface for navigating a user interface using hand gestures, according to some implementation schemes, is shown. Figure 15 This is a flowchart illustrating a method for navigating a user interface using hand gestures, according to some implementation schemes. Figure 16 This is a flowchart illustrating a method for navigating a user interface using hand gestures, according to some implementation schemes. Figure 6 , Figures 7A to 7AA , Figures 8A to 8J , Figures 9A to 9H , Figures 10A to 10F , Figures 11A to 11H , Figures 12A to 12J , Figures 13A to 13G and Figure 14 The user interface in the document is used to illustrate the process described below, including Figure 15 and Figure 16 The process in.

[0045] Furthermore, in methods described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the method may be repeated in multiple repetitions such that, during the repetitions, all conditions determining the steps in the method are satisfied in different repetitions of the method. For example, if a method requires performing a first step (if a condition is satisfied) and a second step (if a condition is not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions determining the steps in the method are satisfied. Those skilled in the art will also understand that, similar to methods having discretionary steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.

[0046] Although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch may be named a second touch and similarly, a second touch may be named a first touch, without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0047] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0048] Depending on the context, the term "if" may optionally be interpreted as meaning "when," "at," or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" may optionally be interpreted as meaning "in response to determination..." or "in response to detection [the stated condition or event]."

[0049] This document describes implementations of electronic devices, user interfaces for such devices, and related processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). In some embodiments, the electronic device is a computer system that communicates with a display generating component (e.g., via wireless or wired communication). The display generating component is configured to provide visual output, such as a display via a CRT monitor, a display via an LED monitor, or a display via image projection. In some embodiments, the display generating component is integrated with the computer system. In some embodiments, the display generating component is separate from the computer system. As used herein, “display” content includes displaying content (e.g., video data rendered or decoded by display controller 156) by transmitting data (e.g., image data or video data) to an integrated or external display generating component via a wired or wireless connection to visually generate content.

[0050] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0051] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0052] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0053] Now let’s turn our attention to implementation schemes for portable devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and is sometimes referred to as or called a “touch-sensitive display system.” Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (e.g., generating haptic output on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0054] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute (alternative) for the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface near the contact, and / or the resistance and / or variation of the touch-sensitive surface near the contact may optionally be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement) has been exceeded. In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure) has been exceeded. Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited physical space, which is used (e.g., on a touch-sensitive display) to display an indication and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0055] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. For example, even when the smoothness of the tactile surface remains unchanged, the movement of the tactile surface can optionally be interpreted or sensed by the user as the "roughness" of the tactile surface. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user.

[0056] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits.

[0057] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0058] Peripheral interface 118 can be used to couple the device's input and output peripherals to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs (such as computer programs (e.g., including instructions)) and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0059] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals and vice versa, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via near-field communication radio components. Wireless communication may optionally employ any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution, Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE...). 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol that has not been developed as of the date of this document submission.

[0060] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a single-ear or dual-ear headphone) and input (e.g., a microphone).

[0061] I / O subsystem 106 couples input / output peripherals on device 100, such as touchscreen 112 and other input control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, depth camera controller 169, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / send electrical signals to the other input control device 116. The other input control device 116 optionally includes physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click dials, etc. In some embodiments, input controller 160 is optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing device such as mouse. One or more buttons (e.g., Figure 2 Optionally, 208) includes an increase / decrease button for volume control of speaker 111 and / or microphone 113. The one or more buttons optionally include a push-button (e.g., Figure 2(Ref. 206 in the original text). In some embodiments, the electronic device is a computer system that communicates with one or more input devices (e.g., via wireless communication or via wired communication). In some embodiments, the one or more input devices include a touch-sensitive surface (e.g., a touchpad, as part of a touch-sensitive display). In some embodiments, the one or more input devices include one or more camera sensors (e.g., one or more optical sensors 164 and / or one or more depth camera sensors 175), such as for tracking user gestures (e.g., hand gestures and / or air gestures) as input. In some embodiments, the one or more input devices are integrated with the computer system. In some embodiments, the one or more input devices are separate from the computer system. In some implementations, air gestures are gestures detected without the user touching an input element that is part of the device (or independently of an input element that is part of the device) and based on the detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tapping gesture that includes a hand moving a predetermined amount and / or speed in a predetermined posture, or a shaking gesture that includes a predetermined speed or amount of rotation of a part of the user's body)).

[0062] A quick press of the push button optionally disengages the touchscreen 112 from its lock or optionally initiates a process of unlocking the device using gestures on the touchscreen, as described in U.S. Patent Application 11 / 322,549 (i.e., U.S. Patent No. 7,657,849), filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," the entire contents of which are incorporated herein by reference. A long press of the push button (e.g., 206) optionally powers the device 100 on or off. The function of one or more buttons is optionally user-customizable. The touchscreen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0063] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0064] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In an exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.

[0065] Touchscreen 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. Touchscreen 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that from Apple Inc. (Cupertino, California). and iPod The technology used.

[0066] In some embodiments of the touchscreen 112, the touch-sensitive display optionally resembles a multi-touch touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touchscreen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.

[0067] The touch-sensitive display in some embodiments of the touchscreen 112 is described in the following applications: (1) U.S. Patent Application 11 / 381,313, filed May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. Patent Application 10 / 840,862, filed May 6, 2004, “Multipoint Touchscreen”; (3) U.S. Patent Application 10 / 903,964, filed July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. Patent Application 11 / 048,264, filed January 31, 2005, “Gestures For Touch Sensitive Input Devices”; and (5) U.S. Patent Application 11 / 038,590, filed January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input”. (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, “Multi-Functional Hand-Held Device”. The full text of all these applications is incorporated herein by reference.

[0068] Touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. Users optionally use any suitable object or accessory such as a stylus, finger, etc., to interact with touchscreen 112. In some embodiments, the user interface is designed to operate primarily through finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0069] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. Optionally, the touchpad is a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0070] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0071] The device 100 may optionally also include one or more optical sensors 164. Figure 1AAn optical sensor 164 is shown coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing images of the user to be optionally acquired for video conferencing while the user views other video conferencing participants on the touchscreen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lenses and sensor within the device housing), allowing a single optical sensor 164 to be used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.

[0072] The device 100 optionally also includes one or more depth camera sensors 175. Figure 1A A depth camera sensor is shown coupled to a depth camera controller 169 in I / O subsystem 106. Depth camera sensor 175 receives data from the environment to create a 3D model of an object (e.g., a face) within the scene from a viewpoint (e.g., the depth camera sensor). In some embodiments, in conjunction with imaging module 143 (also referred to as a camera module), depth camera sensor 175 may optionally be used to determine depth maps of different portions of an image captured by imaging module 143. In some embodiments, the depth camera sensor is located at the front of device 100, such that user images with depth information are optionally acquired for video conferencing while the user views other video conferencing participants on a touchscreen display, and selfies with depth map data are captured. In some embodiments, depth camera sensor 175 is located at the rear of the device, or both the rear and front of device 100. In some embodiments, the position of depth camera sensor 175 may be changed by the user (e.g., by rotating a lens and sensor within the device housing), such that depth camera sensor 175 is used in conjunction with a touchscreen display for both video conferencing and still image and / or video image acquisition.

[0073] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1AA contact strength sensor is shown coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100, opposite to the touchscreen display 112 located on the front of device 100.

[0074] The device 100 may optionally also include one or more proximity sensors 166. Figure 1A A proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 may optionally be coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 may optionally perform as described in the following U.S. patent applications: No. 11 / 241,839, entitled "Proximity Detector In Handheld Device"; No. 11 / 240,788, entitled "Proximity Detector In Handheld Device"; No. 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and No. 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, the proximity sensor is turned off and the touchscreen 112 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0075] The device 100 may optionally also include one or more tactile output generators 167. Figure 1AA haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.

[0076] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may be coupled to an input controller 160 in an I / O subsystem 106. The accelerometer 168 may optionally perform as described in the following U.S. patent publications: U.S. Patent Publication No. 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication No. 20060017692, entitled "Methods and Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 100 may optionally include, in addition to the accelerometer 168, a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for acquiring information about the location and orientation (e.g., portrait or landscape) of device 100.

[0077] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 ( Figure 1A ) or 370 ( Figure 3 Storage device / global internal state 157, such as Figure 1A and Figure 3 As shown in the figure. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, indicating what applications, views or other information occupy various areas of the touch screen display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and position information relating to the device's position and / or orientation.

[0078] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0079] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to… (Trademark of Apple Inc.) The same or similar and / or compatible multi-pin (e.g., 30-pin) connectors used in Apple Inc. devices.

[0080] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., touchpads or physical click-based rotary dials). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to single-point contact (e.g., single-finger contact) or multi-point simultaneous contact (e.g., "multi-touch" / multiple-finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0081] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0082] The touch / motion module 130 optionally detects gesture input performed by the user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0083] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0084] In some implementations, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like to specify the graphic to be displayed, and, if necessary, also receives coordinate data and other graphic attribute data, and then generates screen image data for output to the display controller 156.

[0085] The haptic feedback module 133 includes various software components for generating instructions which are used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0086] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0087] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to phone 138 for use in location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services, such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0088] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0089] • Contacts module 137 (sometimes called address book or contact list);

[0090] • Telephone module 138;

[0091] • Video conferencing module 139;

[0092] • Email client module 140;

[0093] • Instant Messaging (IM) module 141;

[0094] Fitness support module 142;

[0095] • Camera module 143 for still images and / or video images;

[0096] • Image management module 144;

[0097] • Video player module;

[0098] Music player module;

[0099] • Browser module 147;

[0100] • Calendar module 148;

[0101] • Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets acquired by the user, and user-created widgets 149-6;

[0102] • Widget creator module 150 for creating user-created widgets 149-6;

[0103] • Search module 151;

[0104] • Video and music player module 152, which combines a video player module and a music player module;

[0105] • Notes module 153;

[0106] • Map module 154; and / or

[0107] • Online video module 155.

[0108] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0109] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., in the application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0110] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 137, modify entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0111] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0112] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0113] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving an instant message, and viewing a received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0114] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes executable instructions for creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (executive devices); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing music for fitness activities; and displaying, storing, and transmitting fitness data.

[0115] In conjunction with the touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0116] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0117] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet according to user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as links to attachments and other files on web pages.

[0118] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0119] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is optionally a microapplication downloaded and used by a user (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or a user-created microapplication (e.g., user-created widget 149-6). In some embodiments, the widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! widgets).

[0120] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget creator module 150 can optionally be used by the user to create widgets (e.g., to convert user-specified portions of a webpage into widgets).

[0121] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0122] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0123] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the note-taking module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0124] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data related to shops and other points of interest at or near a specific location, and other location-based data) according to user instructions.

[0125] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions for performing the following actions: allowing users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Further descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.

[0126] Each of the above modules and applications corresponds to an executable set of instructions for performing one or more of the functions described above and the methods described in this patent application (e.g., computer-implemented methods and other information processing methods as described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs (such as computer programs (e.g., including instructions)), processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. For example, a video player module may optionally be combined with a music player module into a single module (e.g., Figure 1A (e.g., video and music player module 152). In some embodiments, memory 102 optionally stores subgroups of the aforementioned modules and data structures. Additionally, memory 102 optionally stores other modules and data structures not described above.

[0127] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on device 100 can be optionally reduced.

[0128] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main menu, home menu, or root menu. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0129] Figure 1B This is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0130] Event classifier 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0131] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0132] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral device interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.

[0133] In some implementations, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other implementations, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0134] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0135] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0136] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0137] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.

[0138] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.

[0139] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0140] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.

[0141] In some implementations, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other implementations, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit or a higher-level object from which application 136-1 inherits methods and other properties. In some implementations, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0142] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event based on the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0143] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a longitudinal orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).

[0144] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 112, and lifting off the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0145] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0146] In some implementations, the definition of the corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0147] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0148] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0149] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and delaying) the sub-event to the corresponding hit view. In some implementations, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with that tag retrieves the tag and executes a predefined procedure.

[0150] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined procedure.

[0151] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on touch-sensitive display.

[0152] In some implementations, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.

[0153] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0154] Figure 2A portable multifunction device 100 with a touchscreen 112 is shown according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some specific embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0155] Device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.

[0156] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic outputs for a user of device 100.

[0157] Figure 3This is a block diagram of an exemplary multi-functional device with a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the reference above). Figure 1A The tactile output generator 167 and sensor 359 (e.g., optical sensor, accelerometer, proximity sensor, touch sensor and / or contact intensity sensor (similar to the one mentioned above)) are described. Figure 1A The contact strength sensor 165 is described above. The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100. Figure 1A The memory 370 stores programs, modules, and data structures similar to those in the memory 102 of the portable multifunction device 100, or subsets thereof. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 (… Figure 1A The memory 102 may optionally not store these modules.

[0158] Figure 3Each of the elements described above is optionally stored in one or more memory devices of the previously mentioned memory devices. Each of the modules described above corresponds to a set of instructions for performing the functions described above. The modules or computer programs described above (e.g., instruction sets or including instructions) need not be implemented as separate software programs (such as computer programs (e.g., including instructions)), processes, or modules, and therefore various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subgroup of the modules and data structures described above. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0159] Now let’s turn our attention to the implementation of the user interface, which is optionally implemented on, for example, a portable multifunction device 100.

[0160] Figure 4A An exemplary user interface for an application menu on a portable multifunction device 100 according to some embodiments is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof:

[0161] • Signal strength indicator 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0162] • Time 404;

[0163] Bluetooth indicator 405;

[0164] • Battery status indicator 406;

[0165] • Tray tray 408 with icons for commonly used applications, such as:

[0166] The telephone module 138 has an icon 416 labeled "telephone", which optionally includes an indicator 414 indicating the number of missed calls or voicemail messages.

[0167] The email client module 140 has an icon 418 labeled "Mail", which optionally includes an indicator 410 for the number of unread emails;

[0168] The browser module 147 has an icon 420 labeled "Browser"; and

[0169] The video and music player module 152 (also known as the iPod module 152, a trademark of Apple Inc.) with an icon 422 labeled "iPod"; and

[0170] • Icons of other applications, such as:

[0171] Icon 424 of the οIM module 141, labeled "Message";

[0172] The icon 426 of the calendar module 148 is labeled "Calendar";

[0173] The icon 428 of the image management module 144 is labeled "photo";

[0174] The icon 430 of the camera module 143 is labeled "camera";

[0175] The icon 432 of the online video module 155 is labeled "Online Video";

[0176] The icon 434 labeled "Stock Market" in the Stock Market widget 149-2;

[0177] The icon 436 of the map module 154 is labeled "map";

[0178] The weather widget 149-1 has icon 438 labeled "weather";

[0179] The alarm clock widget 149-4 has an icon 440 labeled "clock";

[0180] The icon 442 of the fitness support module 142 is labeled "fitness support";

[0181] The icon 444 labeled "Notes" in the Notes module 153; and

[0182] The icon 446, labeled "Settings," is used to set the settings of the device 100 and its various applications 136.

[0183] It should be pointed out that, Figure 4A The icon labels shown are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels may be optionally used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0184] Figure 4B A touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)) is shown. Figure 3 Devices such as tablets or touchpads (e.g., 355) Figure 3An exemplary user interface on the device 300. The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 359) for detecting the intensity of contact on the tactile surface 451 and / or one or more tactile output generators 357 for generating tactile outputs for the user of the device 300.

[0185] While some examples of input on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given below, in some implementations the device detects input on a touch-sensitive surface separate from the display, such as... Figure 4B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a spindle (e.g., on the display (e.g., 450)). Figure 4B The spindle corresponding to 453 in the middle (e.g., Figure 4B (452 in the example). According to these embodiments, the device detects the position corresponding to the corresponding position on the display (e.g., in the example). Figure 4B In the middle, 460 corresponds to 468 and 462 corresponds to 470) is in contact with the touch-sensitive surface 451 (e.g., Figure 4B (460 and 462 in the text). Thus, when the touch-sensitive surface (e.g., ...) Figure 4B 451) and the display of a multi-functional device (e.g., Figure 4B When 450 is separated from the touch-sensitive surface, user input detected by the device on that touch-sensitive surface (e.g., touches 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may optionally be used for other user interfaces described herein.

[0186] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger touch, single-finger tap, finger swipe), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0187] Figure 5AAn exemplary personal electronic device 500 is illustrated. Device 500 includes a body 502. In some embodiments, device 500 may include components relative to devices 100 and 300 (e.g., Figures 1A to 4B Some or all of the features described herein. In some embodiments, device 500 has a touch-sensitive display 504, referred to below as touchscreen 504. As an alternative to or complement to touchscreen 504, device 500 has a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., touch). The one or more intensity sensors of touchscreen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on the intensity of the touch, meaning that touches of different intensities can invoke different user interface operations on device 500.

[0188] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, filed May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is incorporated herein by reference in its entirety.

[0189] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) may be physical. Examples of physical input mechanisms include push-buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watch straps, bangles, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow a user to wear device 500.

[0190] Figure 5B An exemplary personal electronic device 500 is depicted. In some embodiments, device 500 may include a reference retrieval system. Figure 1A , Figure 1B and Figure 3 Some or all of the components described herein. Device 500 has a bus 512 that operatively couples I / O portion 514 to one or more computer processors 516 and memory 518. I / O portion 514 may be connected to display 504, which may have touch-sensitive components 522 and optionally have an intensity sensor 524 (e.g., a contact intensity sensor). Furthermore, I / O portion 514 may be connected to communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 may optionally be a rotatable input device or a pressable input device and a rotatable input device. In some examples, input mechanism 508 may optionally be a button.

[0191] In some examples, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to the I / O section 514.

[0192] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, may cause the computer processors to perform the techniques described below, including processes 1500 and 1600. Figures 15 to 16A computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. Personal electronic devices are not limited to... Figure 5B It can be the components and configurations, or it can include other components or additional components in a variety of configurations.

[0193] As used herein, the term "power indication" refers optionally to the power indication in devices 100, 300, and / or 500 ( Figure 1A , Figure 3 and Figures 5A to 5B A user-interactive graphical user interface object displayed on a screen. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) optionally each constitute a functional representation.

[0194] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other positional marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3 The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of enabling direct interaction with user interface elements on the touchscreen display) Figure 1A The touch-sensitive display system 112 or Figure 4AIn some embodiments of the touchscreen 112, a touch detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that delivers the user-expected interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0195] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted away, before or after contact begins to move, before contact ends, before or after contact intensity is detected to increase and / or before or after contact intensity decreases). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some implementations, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the feature intensity and one or more thresholds is used to determine whether to perform one or more actions (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.

[0196] Figure 6 A set of exemplary hand gestures according to some implementation schemes are shown. The user interface in these figures is used to illustrate including Figures 15 to 16 The process described below is the process in the middle.

[0197] Figure 6 A set of exemplary hand gestures that can be detected by a computer system 600 is shown. The computer system 600 includes an input mechanism 602a (e.g., a watch crown), an input mechanism 602b (e.g., a side button), and a display screen 602c. In some embodiments, the computer system 600 includes one or more components of devices 100, 300, and / or 500. In some embodiments, the computer system 600 is a wearable device, such as a watch. In some embodiments, input mechanisms 602a and 602b may include those described above. Figure 5AThe input mechanisms 506 and 608 are described as having one or more components and / or features. In some embodiments, the display 602c is a touch-sensitive display and includes one or more components and / or features as described above with respect to touchscreen 504.

[0198] like Figure 6 As shown, the computer system 600 is worn around the user's wrist and away from the hand 622. The computer system 600 includes one or more accelerometers, gyroscopes, and / or biometric sensors to detect various hand gestures and / or movements (e.g., tilting). Figure 6 At this location, one or more biometric sensors in the biometric sensor suite include optical sensors and / or heart rate sensors used by the computer system 600 to detect various hand gestures. In some embodiments, the computer system 600 may use one or more sensors different from the optical sensors / heart rate sensors to detect hand gestures.

[0199] Figure 6 An exemplary hand gesture is shown that can be detected by a computer system 600 (e.g., via an optical sensor / heart rate sensor). Figure 6 As shown, hand gestures include a neutral gesture / position 610, a clenched gesture 620, a double clenched gesture 630, a pinching gesture 640, and a double pinching gesture 650. While these hand gestures (e.g., 610, 620, 630, 640, and 650) are discussed throughout this application, hand gestures (e.g., multi-finger tapping gestures and / or three (four, five) clenched / pinched gestures) and one or more combinations of these hand gestures are contemplated as being detectable by computer system 600 (e.g., via an optical sensor / heart rate sensor) and / or used to perform one or more of the operations described below. Therefore, the hand gestures provided herein are for illustrative purposes only, and the embodiments described herein are not limited to these specific hand gestures. Furthermore, the hand gestures described herein are not directed at (or captured by) one or more cameras of computer system 600. Additionally, the hand gestures described herein do not contact display screen 602c and / or are not performed in front of display screen 602c (e.g., so that the hand gestures can be detected by computer system 600).

[0200] like Figure 6As shown, a neutral hand gesture / position 610 is a hand gesture / position in which the fingertips of hand 622 do not touch any part of hand 622. A clenched gesture 620 is a hand gesture in which one or more fingers of hand 622 are touching another part of the user's hand, such that the palm of the user's hand is touching when the user forms a fist. A double clenched gesture 630 is a combination (and / or sequence) of neutral gesture / position 610 and clenched gesture 620, in which the fingers of hand 622 close to form a first clenched gesture (e.g., first part 630a), open to form a neutral gesture / position (e.g., second part 630b), and close again to form a second clenched gesture (e.g., third part 630c). Therefore, a double clenched gesture is a gesture that includes multiple (e.g., two) instances (e.g., first part 630a and third part 630c) of clenched gestures detected within a predetermined time period (e.g., 0 seconds to 2 seconds). A clamping gesture 640 is a hand gesture in which one or more fingers of hand 622 are touching each other (e.g., two fingers). A clamping gesture differs from a clenching gesture because a fist is not formed when a clamping gesture is made, whereas a fist is formed when a clenching gesture is made. A double clamping gesture 650 is a combination (and / or sequence) of a neutral gesture / position 610 and a clamping gesture 640, wherein the fingers of hand 622 touch to form a first clamping gesture (e.g., first part 650a), open to form a neutral gesture / position (e.g., second part 650b), and close again to form a second clamping gesture (e.g., third part 650c). Therefore, a double clamping gesture includes a plurality (e.g., two) instances (e.g., first part 650a and third part 650c) of clamping gestures detected at a period detected within a predetermined time period (e.g., 0 seconds–2 seconds). Figure 6 As shown, double-grip gestures 630 and 620 do not include multiple instances of clamping gesture 640, and double-clamping gestures 650 and 640 do not include multiple instances of clamping gesture 620. In some embodiments, when two clamping gestures are detected within a predetermined time period, the computer system 600 registers (or detects) the two gestures as a double-clamping gesture. In some embodiments, when no two clamping gestures are detected within the predetermined time period, the computer system 600 registers (or detects) the two gestures as two separate clamping gestures (e.g., does not register a double-clamping gesture). In some embodiments, when two gripping gestures are detected within a predetermined time period, the computer system 600 registers (or detects) the two gestures as a double-griping gesture. In some embodiments, when no two gripping gestures are detected within the predetermined time period, the computer system 600 registers (or detects) the two gestures as two separate gripping gestures (e.g., does not register a double-griping gesture).

[0201] Figures 7A to 7AAExemplary user interfaces for navigating using hand gestures are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 15 to 16 The process described below is the process in the middle.

[0202] Figures 7A to 7G An exemplary user interface is shown for responding to incoming alarms (e.g., timer alarms) using hand gestures. Figure 7A The illustration shows a computer system 600 displaying a clock user interface 710 on a display screen 602c, which includes the current time (e.g., 10:09). While the computer system 600 is displaying the clock user interface 710, the hand 622 is in a neutral position (e.g., ...). Figure 6 (610 in the middle). Figure 7A At this point, computer system 600 receives incoming alarms from (for example, a timer application installed on computer system 600). Figure 7B As shown, in response to receiving an incoming alarm from a timer application, computer system 600 displays a timer user interface 712. The timer user interface 712 includes a stop control 712a (e.g., when activated, it causes computer system 600 to stop outputting an audible sound associated with the incoming alarm), a repeat control 712b (e.g., when activated, it causes computer system 600 to repeat the timer associated with the incoming alarm), and an indication that the timer has completed (e.g., "Timer Complete"). Figure 7C At that location, computer system 600 detects clamping gestures 750c. However, as... Figure 7D As shown, the computer system 600 maintains the display of the timer user interface 712 and does not perform any operation in response to the clamping gesture 750c because the computer system 600 is not operating in a hand gesture navigation mode (e.g., and / or accessibility mode). In some embodiments, the computer system 600 does not detect the clamping gesture 750c because the computer system 600 is not operating in a hand gesture navigation mode.

[0203] exist Figure 7D At that location, the computer system detected a double-clenched hand gesture (600d). Figure 7E Upon detecting a double-clenched gesture 750d, the computer system 600 begins operation in hand gesture navigation mode. Figure 7E At this point, computer system 600 detects a clamping gesture 750e (e.g., a second clamping gesture). For example... Figure 7FAs shown, in response to detecting a clamping gesture 750e, the computer system 600 displays a focus indicator around the stop control 712a because the computer system 600 is operating in hand gesture navigation mode. The computer system 600 displays the focus indicator around the stop control 712a to indicate that the stop control 712a can be activated in response to the computer system 600 detecting a specific hand gesture. Conversely, the computer system 600 does not... Figure 7F A focus indicator is displayed around the repeating control 712b. Therefore, the repeating control 712b cannot be activated in response to the computer system 600 detecting a specific hand gesture (e.g., before the repeating control 712b is displayed along with the focus indicator). Figure 7F At this location, the computer system 600 detects a clenched fist gesture 750f. Figure 7G In response to detecting a clenched gesture 750f, computer system 600 activates stop control 712a. Upon activation of stop control 712a, computer system 600 stops displaying the timer user interface 712, stops outputting audible sounds associated with incoming alarms, and redisplays the clock user interface 710 (e.g., as shown in the image). Figure 7G (as shown in the image).

[0204] Figures 7G to 7J An exemplary user interface for responding to incoming alarms (e.g., incoming calls) using hand gestures is shown. As discussed above, Figure 7G The computer system 600 is shown displaying a clock user interface 710 when operating in hand gesture navigation mode. Figure 7G At that location, computer system 600 receives an alarm corresponding to an incoming telephone call. For example... Figure 7H As shown, in response to receiving an alarm, computer system 600 displays a telephone user interface 714 including a call identifier 714a (e.g., “JOHN APPLESEED Incoming Call”), which indicates that computer system 600 is receiving an incoming call from John Appleseed. The telephone user interface 714 also includes a reject control 714b (e.g., when activated, it causes computer system 600 to reject the call), an answer control 714c (e.g., when activated, it causes computer system 600 to answer the call), and an additional options control 714d (e.g., when activated, it causes computer system 600 to display additional options for responding to the incoming call). Figure 7H As shown, computer system 600 displays hand gesture notification 716, which instructs the user to answer a telephone call by providing a double-clenched gesture (e.g., instead of providing multiple gestures (e.g., using such gestures as "about"). Figures 7C to 7GOne or more techniques discussed (such as one or more gripping gestures and clenching gestures to navigate to the answer control 714c) are used to answer telephone calls. In some embodiments, the computer system 600 displays hand gesture notification 716 because it is determined that the computer system 600 is operating in a hand gesture navigation mode and / or the computer system 600 is operating in a context where a specific type of action (e.g., answering a telephone call, stopping an alarm, and / or replying to a text message) can be performed via a single hand gesture (e.g., a pre-defined hand gesture). In some embodiments, the computer system 600 displays other and / or different hand gesture notifications (e.g., from hand gesture 716) to inform the user that one or more other gestures can be used to perform the action.

[0205] exist Figure 7I At this location, computer system 600 detects a double-clenched hand gesture 750i. (For example...) Figure 7J As shown, in response to detecting a double-grip gesture 750i, computer system 600 replaces call identifier 714a with an elapsed time indicator 714e (which indicates that an incoming telephone call has been answered) and displays volume control 714f (e.g., when activated, it adjusts the volume level of one or more speakers of computer system 600). In other words, in response to detecting a double-grip gesture 750i, computer system 600 answers an incoming telephone call. In some embodiments, computer system 600 performs the same actions as when an incoming alarm is being received (e.g., and / or has been received within a predetermined time period) in response to detecting a double-grip gesture (e.g., as per [reference to...]). Figures 7I to 7L The different operations discussed (e.g., regarding...) Figures 7D to 7E and Figures 7L to 7M (As discussed).

[0206] Figures 7I to 7U An exemplary user interface is shown for navigating the user interface using hand gestures and cursor movement. Specifically, Figures 7I to 7U An exemplary scenario is shown where hand gestures and cursor movement are used to end a fitness tracker session. Figure 7K A computer system 600 is shown displaying a fitness user interface 718 that includes a list of fitness metrics. When the fitness user interface 718 is displayed, the computer system 600 detects… Figure 7L The double-clenched hand gesture at the location is 750l. (Example) Figure 7M As shown, in response to detecting a double-grip gesture 750l, computer system 600 displays a menu 732 including digital hardware operation controls 732a, a cursor movement control 732b, interactive controls 732c, and additional option controls 732d. Menu 732 includes controls indicating the currently focused element (e.g., ...). Figure 7MThe control identifier 708 (“numeric crown”) is a black box surrounding the digital hardware operation control 732a. In response to detecting activation of the digital hardware operation control 732a, the computer system 600 begins operation in a digital hardware operation mode (e.g., different hand gesture navigation modes). In response to detecting activation of the interactive control 732c, the computer system 600 replaces menu 732 with a menu having controls that, when activated, cause the computer system 600 to perform different operations corresponding to various gestures detected at a location on the computer system 600 (e.g., as described below regarding…). Figure 7Z (As discussed below). In response to detecting the activation of the additional option control 732d, the computer system 600 replaces menu 732 with a menu that includes the additional control (e.g., as described below regarding...). Figure 7Z (As discussed).

[0207] like Figure 7M As shown, in response to detecting a double-clenched gesture 750l, the computer system 600 displays a focus indicator around the digital hardware operation control 732a to indicate that the digital hardware operation control 732a can be activated in response to the computer system 600 detecting one or more hand gestures. Figure 7M At this location, the computer system detects clamping gestures at 600m. (For example...) Figure 7N As shown, in response to the detection of a clamping gesture 750m, the computer system 600 moves the focus indicator to the right, causing it to revolve around the movement cursor control 732b instead of the digital hardware operation control 732a. It is noteworthy that, in response to the detection of a clamping gesture 750m, the computer system 600 moves the focus indicator from one control to the next on the user interface (e.g., in terms of position). Figure 7N At this point, the computer system 600 detects the clenched gesture 750n and simultaneously displays a focus indicator around the moving cursor control 732b.

[0208] like Figure 7O As shown, in response to detecting a clenched hand gesture 750n, the computer system 600 stops displaying the menu 732 and displays a cursor 742 at the location shown on the fitness user interface 718. Figure 7O In response to detecting a clenched hand gesture 750n, the computer system 600 begins operating in a cursor movement mode (e.g., another hand gesture mode). Figure 7P At this point, computer system 600 detects (e.g., via one or more accelerometers and / or gyroscopes, via sensors different from optical sensors / heart rate sensors, and / or via sensors different from those detecting one or more hand gestures) that computer system 600 is tilting to the left (e.g., moving) (e.g., 750p). Figure 7PIn response to the detection that the computer system 600 is tilting to the left, the computer system 600 moves the cursor 742 to the left based on the amount and / or speed of tilt detected by the computer system 600. Figure 7P As shown, cursor 742 moves toward the left edge of the fitness user interface 718. Figure 7P At this point, it is determined that cursor 742 is positioned at (e.g., and / or near) the left edge of the fitness user interface 718 (for example, for a predetermined time period). Figures 7Q to 7R As shown, because the cursor 742 is positioned on the left edge of the fitness user interface 718, the computer system 600 displays an animation of sliding the fitness user interface 718 away from the right edge of the display screen 602c and sliding the fitness control user interface from the left edge of the display screen onto the display screen 602c (e.g., sliding to the right). Therefore, because the cursor 742 is positioned at (e.g., and / or near) this location, the computer system 600 replaces the currently displayed user interface with another user interface.

[0209] like Figure 7R As shown, the fitness control user interface 720 includes a lock control 720a (e.g., when activated, it causes the computer system 600 to ignore touch input detected on the computer system 600), a new control 720b (e.g., when activated, it causes the computer system 600 to initiate a new fitness tracker), an end control 720c (e.g., when activated, it causes the computer system 600 to stop the fitness tracker currently tracking the fitness activity), and a pause control 730d (e.g., when activated, it causes the computer system 600 to pause the fitness tracker currently tracking the fitness activity). Figure 7R As shown, hand 622 is in a neutral position (e.g., Figure 6 (610 in the middle), and the cursor 742 is positioned above the end control 720c. In Figure 7S At this point, a certain period of time has passed since the cursor 742 was first positioned above the end control 720c (and / or at the current position of the cursor 742) (e.g., and the hand 622 has been held in a neutral position). Figure 7S As shown, computer system 600 displays a cursor 742 with an indicator 744. The indicator 744 indicates the amount of time before an operation corresponding to the position of cursor 742 (e.g., an activation operation) is performed, such as activating an end control 720c and / or displaying a menu including one or more controls for performing the operation corresponding to the position of cursor 742 (e.g., ...). Figure 7M(Menu 732). Therefore, when cursor 742 is displayed at a location corresponding to a user interface object such as a control, computer system 600 displays an animation of the size of indicator 744 and / or fills cursor 742 for a certain period of time (e.g., a period of time indicating the remaining time before an operation will be performed). Figure 7S At this point, the indicator 744, filling approximately half of the cursor 742, indicates that approximately half the time has elapsed before the computer system 600 can perform the operation corresponding to the position of the cursor 742 (e.g., since the cursor 742 was positioned above the end control 720c). (Review) Figures 7Q to 7R The computer system 600 does not display the indication with cursor 742, nor does it display it on the user interface and / or at the edge of the display, regardless of how long the computer system 600 is displayed in a particular location. Therefore, in some embodiments, the computer system 600 displays the animation including indication 744 only when the cursor 742 is moved over a user interface object (e.g., a selectable and / or (activatable) user interface object, or a user interface object that can be activated via one or more gestures detected on display screen 602c).

[0210] exist Figure 7S At this point, more time has passed since the cursor 742 was first positioned above the end control 720c (e.g., and the hand 622 has been held in a neutral position). Figure 7T As shown, computer system 600 has updated instruction 744 so that instruction 744 has filled all cursors 742. Figure 7T At this point, ensure that cursor 742 is positioned above end control 720c for a predetermined time period (e.g., 1-5 seconds). Figure 7U Since the cursor 742 has been positioned above the end control 720c for a predetermined period of time, the computer system 600 activates the end control 720c. Once the end control 720c is activated, the computer system 600 displays a completion user interface 770 indicating that the fitness tracker has ended.

[0211] Figures 7U to 7AA An exemplary user interface is shown for navigating the user interface using hand gestures and cursor movement. Specifically, Figures 7U to 7AA This illustrates that computer system 600 enters a cursor movement operation mode in response to a detected hand gesture (e.g., instead of activating the cursor movement control 732b on menu 732 via one or more hand gestures, as described above). Figure 7N The exemplary scenario described above. Figure 7U A computer system 600 is shown displaying a clock user interface 710, which includes the current time (e.g., 10:09). Figure 7WAt this point, computer system 600 detects that it is (or has been) being shaken (e.g., via one or more accelerometers and / or gyroscopes) (e.g., 750W). Figure 7X In response to detecting that the computer system 600 is being (or has been) shaken, the computer system 600 begins operating in a cursor-moving mode (e.g., switching from operating in a cursor-moving mode to operating in a cursor-moving mode). In response to detecting that the computer system 600 is being (or has been) shaken, the computer system 600 displays cursor 742. Figure 7Y At this point, computer system 600 detects that computer system 600 is tilting (e.g., moving) to the right (e.g., 750 y). Figure 7Y In response to the detection that the computer system 600 is tilting to the right, the computer system 600 moves the cursor 742 to the right based on the amount and / or speed of tilt detected by the computer system 600. Figure 7Y As shown, cursor 742 moves toward the right edge of the clock user interface 710. Figure 7Y At this point, it is determined that the cursor 742 is positioned at (e.g., and / or near) the right edge of the clock user interface 710. Figure 7Z As shown in 7AA, because the cursor 742 is positioned on the left edge of the fitness user interface 718, the computer system 600 displays an animation of sliding the clock user interface away from the left edge of the display screen 602c (e.g., away from the edge on which the cursor 742 is positioned), and displays the analog clock user interface 780 sliding from the right side of the display screen 602c and moving toward the left edge of the display screen 602c.

[0212] Figures 8A to 8J Exemplary user interfaces for navigating using hand gestures are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 15 to 16 The process described below is the process in the middle.

[0213] Specifically, Figures 8A to 8J The computer system 600 displays a menu 732 in response to the cursor 742 being positioned (e.g., displayed) above a user interface object (e.g., instead of activating the user interface object, as described above). Figures 7I to 7U The exemplary scenario described above. Figure 8A A computer system 600 is shown displaying a clock user interface 810, which includes application controls 810a-810d (e.g., each of which, when activated, causes the computer system 600 to launch an application). Figure 8A At this point, computer system 600 detects that it is being (or has been) shaken (e.g., 850a) (e.g., using the above-mentioned...). Figures 7U to 7AA (One or more of the aforementioned technologies). In Figure 8B In response to the detection that the computer system 600 is (or has been) being shaken, the computer system 600 begins to operate in a moving cursor mode and displays cursor 742 on the clock user interface 810.

[0214] exist Figure 8C The cursor 742 is already in the same position (for example, it is in...). Figure 8B The specified time period is displayed at the location shown in the image. However, in... Figure 8C At this point, computer system 600 does not display the animation of cursor 742 filling because cursor 742 is not located at a position corresponding to a user interface object (e.g., when computer system 600 detects one or more inputs on a user interface object, and in some embodiments, when the computer system operates in normal operating mode and / or does not detect hand gestures, a selectable and / or activated user interface object and / or an activated user interface object) (e.g., using about Figures 7I to 7U (One or more of the aforementioned technologies). In Figure 8C At that point, computer system 600 detects that it is tilting downwards (e.g., 850c) (e.g., using about Figures 7I to 7U (Description of one or more technologies). For example... Figure 8D As shown, in response to the detection that the computer system 600 is tilting downwards, the computer system 600 moves the cursor 742 above the application icon 810d.

[0215] like Figures 8E to 8G As shown, computer system 600 displays an animated cursor 744 that fills cursor 742 for a certain period of time because cursor 742 is displayed above application icon 810d (e.g., optional user interface object) (e.g., using the above regarding...). Figures 7I to 7U (One or more technologies discussed). Figure 8G As shown, indicator 744 has completely filled cursor 742. Figure 8G At this point, it is confirmed that cursor 742 has been displayed above application icon 810d, and that cursor 742 has been positioned above application icon 810d for a predetermined period of time (e.g., 1 second to 5 seconds).

[0216] like Figure 8H As shown, because the cursor 742 has been positioned above the application icon 810d for a predetermined period of time, the computer system 600 displays the menu 732 (e.g., and in some embodiments, selecting the application icon 810d and / or the location of the application icon 810d (e.g., making it possible to perform an operation using the location of the application icon 810d and / or the application icon 810d) without activating the application icon 810d). Furthermore, in Figure 8HAt this point, computer system 600 is displaying cursor 742 positioned above interactive control 732c. In some embodiments, one or more settings control whether computer system 600 activates the user interface object when it is determined that cursor 742 has been positioned above the corresponding user interface object for a predetermined period of time, whether computer system 600 displays a menu when it is determined that cursor 742 has been positioned above the corresponding user interface object for a predetermined period of time, or both. In some embodiments, computer system 600 displays menu 732 on the display at a location further away from where cursor 742 is positioned above application icon 810d (so as not to cover the user interface object on which cursor 742 is hovering).

[0217] like Figure 8I As shown, computer system 600 displays an indication 744 that the cursor 742 is filled (e.g., using a command like "About"). Figures 8E to 8G (One or more technologies discussed). Figure 8I At this point, it is confirmed that cursor 742 has been detected above interactive control 732c for a predetermined duration. Figure 8J Since it is determined that cursor 742 has been detected over the selectable user interface object for a predetermined period of time, computer system 600 activates interactive control 732c. In response to the activation of interactive control 732c, computer system 600 displays additional controls, including tap control 832a. It is worth noting that when it is determined that cursor 742 has been detected over the selectable user interface object for a predetermined period of time and menu 732 is displayed, computer system 600 activates the user interface object (and does not redisplay menu 732, regardless of one or more settings of computer system 600).

[0218] exist Figure 8J At this location, computer system 600 displays a cursor above tap control 832a. In some embodiments, computer system 600 activates tap control 832a when it is determined that cursor 742 has been detected above tap control 832a. In some embodiments, in response to detecting activation of tap control 832a, computer system 600 detects and / or executes an operation that would be performed if a tap gesture was detected at the location where cursor 742 is displayed, causing menu 732 to be displayed (e.g., in...). Figures 8G to 8I (in Chinese). For example, in Figure 8J In response to detecting the activation of a tap on control 832a, computer system 600 launches the application icon 810d corresponding to the application (e.g., Figure 10FThe application is a user interface. In some implementations, other gesture controls may be displayed and activated, wherein the computer system 600 performs an operation (e.g., a long press gesture, a drag gesture) if a corresponding gesture (e.g., a long press gesture, a drag gesture) is detected at the location where the cursor 742 is displayed to display the menu 732 (e.g., different operations if a tap gesture has been detected) (e.g., a gesture to display the clock face menu, a gesture to move the application icon from one position to another on the clock user interface 810, a gesture to remove the application icon from the clock user interface 810).

[0219] Figures 9A to 9H Exemplary user interfaces for navigating using hand gestures are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 15 to 16 The process described below is the process in the middle.

[0220] Specifically, Figures 9A to 9H The computer system 600 is shown responding to the detection of a hand gesture (e.g., using the method described above). Figures 7A to 7G An exemplary scenario of navigation via a user interface (similar to the aforementioned techniques). Figure 9A The diagram illustrates a media user interface 910 displayed by computer system 600 when not operating in hand gesture navigation mode. The media user interface 910 includes a back control 910a (e.g., when activated, it causes computer system 600 to display the previously displayed user interface), a reverse control 910b (e.g., when activated, it configures computer system 600 to play back a previous media item in the media item list), a pause control 910c (e.g., when activated, it causes computer system 600 to pause playback of a media item), a forward control 910d (e.g., when activated, it configures computer system 600 to play back the next media item in the media item list), and an additional options control 910e (e.g., when activated, it causes computer system 600 to display the media item not currently in the list). Figure 9A The system displays one or more additional controls, a queue control 910f (e.g., when activated, it causes the computer system 600 to display a list of queued media items), and a connection control 910g (e.g., when activated, it causes the computer system 600 to display a user interface for connecting the computer system 600 to one or more external computer systems). Figure 9A At this location, the computer system 600 detects a double-clenched hand gesture 950a.

[0221] exist Figure 9B In response to the detection of a double-clenched gesture 950a, the computer system 600 begins operating in a hand gesture navigation mode (e.g., using the gestures described above). Figures 7D to 7E (One or more of the aforementioned technologies). For example... Figure 9B As shown, in response to detecting a double-grip gesture 950a, the computer system 600 displays a focus indicator around the forward control 910d (e.g., using the method described above). Figure 7D (One or more of the techniques discussed). In some implementations, the computer system 600 displays a focus indicator around a control on the media user interface 910 that is different from and / or not a forward control 910d (e.g., controls 910a-910d or 910e-910g) in response to detecting a double-grip gesture 950a.

[0222] exist Figure 9B At this point, computer system 600 detected a double-clamp gesture 950b. (For example...) Figure 9C As shown, in response to detecting a double-clamp gesture 950b, the computer system 600 moves the focus indicator to the left, causing it to appear around the pause control 910c instead of the forward control 910d. Figure 9C In response to detecting a double-clamp gesture 950b, the computer system 600 moves a focus indicator relative to the forward control 910d around a previous control that the computer system 600 has identified as the media user interface 910. In some embodiments, the computer system 600 determines the previous control by identifying the control on the media user interface 910 in the direction of travel from the position of the forward control 910d (e.g., the previous control surrounded by the focus indicator) toward the starting line on the media user interface 910 (e.g., the position on the line near and / or at the top of the media user interface 910 and / or the position on the line having the furthest left / right and near the top of the media user interface 910 optional user interface objects (e.g., controls)). In some embodiments, in Figure 9C At this point, the pause control 910c is determined to be the preceding control of the forward control 910d because the pause control 910c is adjacent to the forward control 910d (e.g., right next to or adjacent to it with no other control between them) and is closer to the start line of the media user interface 910 than the forward control 910d (or, in some embodiments, any other control among the other adjacent controls (e.g., queue control 910f, additional options control 910e)). Figure 9C At this location, the computer system 600 detects a clenched hand gesture 950c.

[0223] exist Figure 9D In response to detecting a clenched hand gesture 950c, computer system 600 activates pause control 910c and pauses media playback (e.g., "What Matters" by a pop singer). Figure 9DAs shown, in response to the detection of a squeeze gesture 950c, the pause control 910c is replaced by the play control 910h (e.g., when activated, it initiates playback of the media item). Figure 9D As shown, in response to detecting a clenched gesture 950c, the computer system 600 maintains the focus indicator in the same position on the media user interface 910. Therefore, in some embodiments, when a clenched gesture 950c is detected and / or when a control is activated in response to a detected hand gesture, the computer system 600 does not move the focus control, regardless of whether the control is replaced by another control. Figure 9E At this location, the computer system 600 detects clamping gestures 950e.

[0224] like Figure 9F As shown, in response to detecting a clamping gesture 950e, the computer system 600 moves the focus indicator to the right, causing it to appear around the forward control 910d instead of the playback control 910h. Figure 9F In response to detecting a clamping gesture 950e, the computer system 600 moves the focus indicator relative to the playback control 910h around the control that the computer system 600 has determined to be the next control on the media user interface 910. In some embodiments, the computer system 600 determines the next control by identifying the control on the media user interface 910 in the direction of travel from the position of the forward control 910d (e.g., the previous control surrounded by the focus indicator) toward the end line on the media user interface 910 (e.g., the position on the line near and / or at the top of the media user interface 910 and / or the position on the line having the furthest left / right and near the bottom of the media user interface 910, respectively). In some embodiments, in Figure 9F At this point, the forward control 910d is determined to be the next control after the playback control 910h because the forward control 910d is adjacent to the forward control 910h (e.g., next to or adjacent to each other without another control between the two controls) and is closer to the start line of the media user interface 910 than the playback control 910h.

[0225] It is worth noting that, Figure 9B The double clamping gesture 950b includes Figure 9EMultiple instances of the clamping gesture 950e. Furthermore, in response to detecting a double clamping gesture 950b, the computer system 600 performs an operation opposite to the operation performed by the computer system 600 in response to detecting the clamping gesture 950e (e.g., navigating to a previous control). Therefore, in some embodiments, the computer system 600 performs an opposite operation in response to detecting gestures comprising multiple instances of each other. In some embodiments, due to the connectivity of different gestures (e.g., similarity between different gestures), opposite gestures help the user navigate the user interface more easily. Return to Figure 9F The computer system 600 detects clamping gestures 950f.

[0226] like Figure 9G As shown, in response to detecting a clamping gesture 950f, the computer system 600 moves the focus indicator downwards, causing it to appear around the additional option control 910e instead of the forward control 910d. Here, the computer system 600 moves the focus indicator downwards because the additional option control 910e is the next control. Figure 9G In this case, computer system 600 does not display a focus indicator around the connection control 910g because the connection control 910g is not adjacent to the forward control 910d (e.g., therefore the connection control 910g is not determined as the next control). In some embodiments, computer system 600 moves the focus around the user interface in a pattern based on a specific layout of controls on a particular user interface. (See previous section.) Figure 9A In some embodiments, the computer system 600 moves the focus indicator around the media user interface 910 sequentially (or in reverse order) according to the identification order of the controls (910a, 910b, 910c, 910d, 910e, 910f, and 910g). In some embodiments, when the focus indicator is around the previous control (e.g., 910g), the computer system 600 moves the focus indicator around the first control (e.g., 910a) in response to detecting a clamping gesture (or a gesture to move the next control). In some embodiments, when the focus indicator is around the first control (e.g., 910a), the computer system 600 moves the focus indicator around the previous control (e.g., 910g) in response to detecting a double clamping gesture (or a gesture to move the previous control). Return to Figure 9G The computer system 600 detects a clenched gesture 910g when the focus indicator is around the additional options control 950e.

[0227] like Figure 9H As shown, in response to detecting a clenched grip 950g, the computer system 600 displays the additional options user interface 920 and stops displaying the media user interface 910. Figure 9HAs shown, in response to detecting a clenching gesture 950g, computer system 600 displays a focus indicator around controls (e.g., controls located on the start line) of the additional options user interface 920. In some embodiments, in response to detecting one or more hand gestures (e.g., clamping and / or double-clamping gestures) that move the focus indicator around the additional options user interface 920, computer system 600 moves the focus indicator around different controls included in the additional options user interface 920. In some embodiments, computer system 600 moves the focus indicator around different controls included in the additional options user interface 910 in a different mode (e.g., vertical mode) than the mode in which computer system 600 moves the focus indicator around controls included in the media user interface 920 (e.g., a serpentine mode). Figures 9A to 9G The above).

[0228] Figures 10A to 10F Exemplary user interfaces for navigating using hand gestures are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 15 to 16 The process described below is the process in the middle.

[0229] Specifically, Figures 10A to 10F The computer system 600 is shown responding to the detection of a hand gesture (e.g., using the method described above). Figures 7A to 7G An exemplary scenario of navigation via a user interface (similar to the aforementioned techniques). Figure 10A The diagram illustrates a computer system 600 displaying a media user interface 1010 when not operating in hand gesture navigation mode. The media user interface 1010 includes application icons 1010a-1010d (e.g., when activated, they cause the computer system 600 to display a user interface for the corresponding activated application icon). Figure 10A At this point, computer system 600 is operating in a hand navigation mode (e.g., as mentioned above regarding...). Figures 7A to 7G as well as Figures 9A to 9H (As discussed above). In some implementations, computer system 600 uses the above-mentioned... Figures 7A to 7G and Figures 9A to 9H The described one or more technologies navigate via a media user interface 1010. Figure 10B At this location, the computer system 600 detects a double-clenched hand gesture 1050b.

[0230] like Figure 10C As shown, in response to the detection of a double-clenched gesture 1050b, the computer system 600 displays a menu 732 including controls 732a-732d (e.g., as described above regarding...). Figure 7D (As discussed). Menu 732 includes controls that identify the currently focused area (e.g., Figure 10CThe control identifier 708 (“Number Crown”) is located within the black border surrounding the digital hardware operation control 732a. Figure 10C As shown, computer system 600 displays a focus indicator around digital hardware operation control 732a and application icon 1010c because application icon 1010c is selected before double-grip gesture 1050b is detected and / or computer system 600 can perform an operation at the location of application icon 1010c (e.g., activation of application icon 1010c, as described above regarding...). Figure 8J (As discussed). Figure 10C At this point, the computer system detected a double-clenched gesture 1050c.

[0231] like Figure 10D As shown, in response to detecting a double-grip gesture 1050c, the computer system 600 stops displaying menu 732 (e.g., when the computer system 600 has been operating in hand gesture navigation mode). Figure 10D At this location, the computer system 600 detects clamping gestures 1050d. For example... Figure 10E As shown, in response to detecting a clamping gesture 1050d, the computer system 600 moves the focus indicator down and to the left, such that the focus indicator is displayed around the application icon 1010d but not around the application icon 1010c. The computer system 600 displays the focus indicator around the application icon 1010d because it is determined that the application icon 1010d is the next control (e.g., in response to detecting the clamping gesture 1050d). In some embodiments, the above description is used... Figures 9A to 9H The determination is made using one or more techniques discussed. Figure 10E At this location, computer system 600 detects a clenched fist gesture 1050e. Figure 10F In response to detecting a clenched gesture 1050e, computer system 600 activates application icon 1010d and displays the application (e.g., respirator application) corresponding to application icon 1010d (e.g., breathing application icon).

[0232] Figures 11A to 11H Exemplary user interfaces for navigating using hand gestures are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 15 to 16 The process described below is the process in the middle. Specifically, Figures 11A to 11H An exemplary scenario is illustrated where a computer system 600 automatically scrolls through a user interface and performs various operations in response to detecting one or more hand gestures. In some embodiments, when the computer system uses, as shown in the example regarding... Figures 11A to 11HWhen one or more of the aforementioned technologies are operating in a digital hardware operating mode (e.g., a hand gesture navigation mode), the computer system 600 automatically scrolls through the user interface. In some embodiments, the computer system 600 begins operating in a digital hardware operating mode in response to activation of the digital hardware operating control 732a. In some embodiments, the computer system 600 automatically scrolls through the user interface and / or performs the same operation as if it were performing an operation in response to the computer system 600 detecting input on an input mechanism (e.g., a rotatable input). In some embodiments, the computer system 600 automatically scrolls through the user interface while the computer system is operating in an automatic scrolling mode (e.g., as discussed below with respect to 1412).

[0233] Figure 11A A computer system 600 is shown displaying a text message user interface 1110. The text message user interface 1110 includes a text message area 1120 and one or more controls, including an animated image control 1110a. (Example...) Figure 11A As shown, text message area 1120 includes text message 1120a (“Hey, where are you?”). For example... Figure 11A As shown, the computer system 600 is displaying a focus indicator around the animated image control 1110a, which is shown at the bottom of the text message user interface 1110. Figure 11A At this location, the computer system 600 detects a double-clenched hand gesture 1150a.

[0234] like Figure 11B As shown, in response to the detection of a double-grip gesture 1150a, the computer system 600 displays a menu 732 including digital hardware operation controls 732a. Figure 11B In some embodiments, the computer system 600 displays menu 732 at the top of the text messaging user interface 1110. In some implementations, the computer system 600 displays menu 732 at the top of the text messaging user interface 1110 because a focus indicator is displayed around the animated image control 1110a, and the animated image control 1110a is displayed at the bottom of the text messaging user interface 1110. In some implementations, the computer system 600 displays menu 732 at the bottom (or another area of ​​the user interface) in response to determining that a selected control (e.g., a control with a focus indicator around it) is not displayed at the bottom of the media user interface 1110. Figure 11B At this point, the computer system 600 detects the double-grip gesture 1150b, and at the same time, a focus indicator is displayed around the digital hardware operation control 732a.

[0235] like Figure 11CAs shown, in response to detecting a double-grip gesture 1150b, the computer system 600 begins operation in a digital hardware operating mode (e.g., and / or an auto-scrolling mode). When operating in the digital hardware operating mode, the computer system 600 performs operations as if receiving one or more inputs (e.g., rotation input, press input, slide input) at the input mechanism of the computer system 600. (Display) Figure 11C At some point after the user interface is accessed, the computer system 600 performs an operation consistent with the detection of one or more inputs on the input mechanism 602a, even when no input and / or hand gestures are detected (e.g., hand 622 is in a neutral position). Figure 11D As shown, when performing an operation, the computer system 600 moves the focus indicator from around the animated image control 1110a to around the language control 1110b (e.g., no input is detected on the input mechanism 602a), which is consistent with the operation when input is detected on the input mechanism 602a. In the display... Figure 11D At some point after the user interface is accessed, the computer system 600 performs another operation consistent with the detection of one or more inputs on the input mechanism 602a, even when no input and / or hand gestures are detected (e.g., hand 622 is in a neutral position). Figure 11E As shown, when an operation is performed, the computer system 600 scrolls the text message user interface 1110 and moves the focus indicator from around the language control 1110b to around the response control 1110c. In the display... Figure 11E At some point after the user interface is accessed, the computer system 600 performs another operation consistent with the detection of one or more inputs on the input mechanism 602a, while no input and / or hand gestures are detected (e.g., hand 622 is in a neutral position). The computer system 600 scrolls the text message user interface 1110 (e.g., displays a new control) and moves the focus indicator from around the response control 1110c to around the response control 1110e. Figure 11F At this point, computer system 600 detects the clamping gesture 1150f, and simultaneously, a focus indicator is displayed around the response control 1110e. For example... Figure 11G As shown, in response to the detection of a clamping gesture 1150f, the computer system 600 stops automatically performing operations consistent with one or more inputs detected on the input mechanism 602a (e.g., and / or pausing scrolling). In some embodiments, in response to the detection of an additional clamping gesture, the computer system 600 resumes performing operations consistent with one or more inputs detected on the input mechanism 602a.

[0236] exist Figure 11G At this point, computer system 600 detects a clenched hand gesture 1150g, and a focus indicator is displayed around the response control 1110e. For example... Figure 11GAs shown, in response to the detection of a clenched gesture 1150g, the response control 1110e is activated. Figure 11G In response to detecting a clenched gesture 1150g, computer system 600 inserts a reply message 1120b (“On the way”) corresponding to reply control 1110e into text message area 1120. In some embodiments, computer system 600 sends reply message 1120b to one or more external computer systems as part of a text message session.

[0237] Figures 12A to 12J Exemplary user interfaces for navigating using hand gestures are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 15 to 16 The process described below is the process in the middle.

[0238] Specifically, Figures 12A to 12J An exemplary scenario is illustrated where a computer system 600 automatically scrolls through a user interface and performs various operations in response to detecting one or more hand gestures. In some embodiments, when the computer system uses, as shown in the example regarding... Figures 12A to 12J When one or more of the aforementioned technologies are operating in a digital hardware operating mode (e.g., a hand gesture navigation mode), the computer system 600 automatically scrolls through the user interface. In some embodiments, the computer system 600 begins operating in a digital hardware operating mode in response to activation of the digital hardware operating control 732a. In some embodiments, the computer system 600 automatically scrolls through the user interface and / or performs the same operation as if it were performing an operation in response to the computer system 600 detecting input on an input mechanism (e.g., a rotatable input). In some embodiments, the computer system 600 automatically scrolls through the user interface while the computer system is operating in an automatic scrolling mode (e.g., as discussed below with respect to 1412).

[0239] Figure 12A A computer system 600 is shown displaying a clock-side user interface 1210. The clock-side user interface 1210 includes optional controls 1210a-1210e. Figure 12A Currently, computer system 600 is operating in automatic scrolling mode. For example... Figures 12A to 12C As shown, computer system 600 automatically (e.g., as indicated by hand 622 being in a neutral position) moves the focus indicator between selectable controls 1210a-1210c (e.g., when computer system 600 does not detect input) (e.g., at a first speed). Figure 12C At this point, computer system 600 detects the clamping gesture 1250c, and simultaneously a focus indicator is displayed around the selectable control 1210c. For example... Figure 12DAs shown, in response to detecting a clamping gesture 1250c, the computer system 600 stops automatically moving the focus indicator between selectable controls and continues to display the focus indicator around the selectable control 1210c (e.g., as indicated by the hand 622 being in a neutral position). Figure 12E At this point, computer system 600 detects the clamping gesture 1250e, and simultaneously a focus indicator is displayed around the selectable control 1210c. For example... Figures 12F to 12G As shown, in response to detecting a clamping gesture 1250e, the computer system 600 restores the focus indicator (e.g., around the optional control 1210d, and then around the optional control 1210e).

[0240] exist Figure 12H At this point, computer system 600 moves the focus indicator to the left, so that the focus indicator is displayed around the selectable control 1210d instead of around 1210e. Figure 12H At this point, computer system 600 moves the focus indicator from optional control 1210e to optional control 1210d because 1210e is identified as the previous optional control on the clock-side user interface 1210. Therefore, in Figure 12H At this point, computer system 600 reverses the direction of movement of the focus indicator around optional user interface objects of clock-side user interface 1210. In some embodiments, computer system 600 displays the focus indicator around optional control 1210c when a predetermined amount of time has elapsed. Figure 12H At this location, the computer system 600 detects double-clamping gestures for 1250 hours. Figure 12G At that point, in response to detecting a double-clamp gesture 1250h, the computer system 600 reverses the movement direction of the focus indicator around the selectable user interface object of the clock-side user interface 1210. For example... Figure 12I As shown, in response to the detection of a double-pinch gesture 1250h, the computer system 600 displays a focus indicator around the optional control 1210e (for example, instead of displaying a focus indicator around the optional control 1210c, the computer system 600 would already be displaying a focus indicator around the optional control if the double-pinch gesture 1250h was not detected). Figure 12I At this point, computer system 600 detects the clenched hand gesture 1250i and simultaneously displays a focus indicator around the selectable control 1210e. For example... Figure 12J As shown, in response to the detection of a clenched gesture 1250j, the computer system 600 activates the optional control 1210e. (As...) Figure 12J As shown, in response to the detection of a clenched gesture 1250j, the computer system 600 displays a calendar application user interface 1220, the user interface of which corresponds to the optional control 1210e.

[0241] Figures 13A to 13G Exemplary user interfaces for navigating using hand gestures are shown according to some embodiments. The user interfaces in these figures are used to illustrate including... Figures 15 to 16 The process described below is the process in the middle.

[0242] Specifically, Figures 13A to 13G One or more exemplary settings are shown for controlling one or more hand gesture navigation modes (e.g., as described above regarding...). Figures 7A to 7AA , Figures 8A to 8J , Figures 9A to 9H , Figures 10A to 10F , Figures 11A to 11H and Figures 12A to 12J The above). Figure 13A The computer system 600 is shown displaying one or more settings including accessibility setting controls 1312a, 1312b, and 1312c. In some embodiments, in response to detecting input on accessibility setting control 1312a, the computer system 600 displays one or more of accessibility settings 1322a-1322f (e.g., such as...). Figure 13B (As shown).

[0243] like Figure 13B As shown, accessibility settings 1312a-1312f include a hand gesture navigation control 1322h. In some embodiments, when the hand gesture navigation control 1322h is closed (e.g., inactive), the computer system 600 does not perform any operation in response to detecting a hand gesture. In some embodiments, when the hand gesture navigation control 1322h is open (e.g., active), the computer system 600 performs any operation in response to detecting a hand gesture. In some embodiments, in response to detecting a gesture toward the hand gesture navigation control 1322h, the computer system 600 displays... Figure 13C User interface.

[0244] like Figure 13CAs shown, computer system 600 displays hand gesture navigation controls 1314a-1314h. The hand gesture navigation controls include a main settings control 1314a, a hand gesture control 1314b, a cursor movement control 1314c, a movement style control 1314d, and one or more appearance settings controls 1314e-1314g, as well as a hand gesture confirmation control 1314h. In response to detecting input to the main settings control 1314a, computer system 600 turns the hand gesture navigation mode on / off (e.g., as described above with respect to hand gesture navigation control 1322h). In response to detecting input to the hand gesture control 1314b, computer system 600 toggles the on / off state of the hand gesture control 1314b. When the hand gesture control 1314b is on, computer system 600 is configured to perform one or more operations in response to detecting a hand gesture. When the hand gesture control 1314b is off, the computer system 600 is not configured to perform one or more operations in response to the detection of a hand gesture. In response to the detection of input to the motion cursor control 1314c, the computer system 600 toggles the motion cursor control on / off. When the motion cursor control 1314c is on, the computer system 600 can be configured to operate in motion cursor mode (e.g., as described above regarding...). Figures 7I to 7U and Figures 8A to 8J (As described). When the cursor movement control 1314c is closed, the computer system 600 cannot be configured to operate in cursor movement mode (e.g., in response to a shake gesture and / or in response to a detected...). Figure 7N Input on the cursor movement control 732b in the menu 732. In response to detecting input to the movement style control 1314d, the computer system 600 switches between different movement options for the movement style control 1314d. The different movement options include one or more of automatic and manual movement (e.g., the focus indicator on the displayed user interface). In response to detecting input to one or more appearance setting controls 1314e-1314g, the computer system 600 changes the menu 732 (e.g., as about...). Figure 7N The cursor 742 (for example, as mentioned above) Figures 7A to 7AA The above) and instruction 744 (e.g., as regarding Figures 7A to 7AAOne or more appearances, such as the colors and / or controls of one or more user interface objects displayed. In response to input to the hand gesture confirmation control 1314h, the computer system 600 toggles the hand gesture confirmation control 1314h on / off. When the hand gesture confirmation control 1314h is on, the computer system 600 is configured to confirm one or more operations (e.g., payment transactions) in response to the detection of one or more hand gestures. When the hand gesture confirmation control 1314h is off, the computer system 600 is not configured to confirm one or more operations in response to the detection of one or more hand gestures.

[0245] like Figure 13D As shown, computer system 600 displays one or more setting controls (e.g., setting controls 1316a-1313e) for changing one or more operations performed by computer system 600 in response to the detection of a corresponding hand gesture. In some embodiments, in response to the detection of one or more gestures toward learning control 1316e, computer system 600 displays... Figure 13E User interface.

[0246] like Figure 13E As shown, computer system 600 displays user interface 1318. User interface 1318 may include one or more instructions (e.g., with graphical representation) instructing the user how to perform one or more gestures. In some embodiments, in Figure 13C At this point, computer system 600 displays a gesture in response to detecting a gesture toward the moving cursor control 1314c. Figure 13F The user interface. In some implementations, when the user interface 1318 is displayed, the computer system 600 will provide feedback on whether the user performed gestures correctly and / or incorrectly during the learning mode session.

[0247] like Figure 13F As shown, computer system 1300 displays one or more controls for controlling one or more aspects of the cursor movement mode, such as sensitivity (e.g., 1322b) (e.g., the manner and / or amount by which the cursor moves in response to detected movement (e.g., tilt) of computer system 600), activity time (e.g., 1322c) (e.g., a predetermined amount of time during which the cursor must be positioned above a user interface object to perform an operation), and / or indication (e.g., ... Figure 7NThe indication 744 represents the amount of time, the movement tolerance (e.g., 1322d) (e.g., the amount of movement required after the cursor has moved from its position corresponding to the user interface object, before indication 744 is reset and / or the animation stops displaying), and the dwell control (e.g., 1322e) (e.g., the predetermined amount of time the user must be positioned above the user interface object to perform an operation and / or whether the computer system can initiate an operation in cursor movement mode via one or more shake inputs). Figure 13G At this point, computer system 600 displays one or more settings (e.g., 1324a-1324c) for changing the color of the moving cursor.

[0248] Figure 14 This shows several menu controls that can be displayed on menu 732 (e.g., as mentioned above regarding...). Figure 7N (As described above). Menu 732 may include a hierarchical structure of controls, such as about... Figure 14 The control hierarchy described above. In some embodiments, in response to detecting input to interactive control 1410, computer system 600 displays one or more of the following: tap control gesture 1410a (e.g., as described above regarding 832a), movement cursor control 1410b (e.g., as described above regarding movement cursor control 732b), and digital hardware operation control 1410 (e.g., as described above regarding digital hardware operation 732a). In some embodiments, in response to detecting input to system control 1414, computer system 600 displays icon grid 1414a (e.g., when activated, it displays a grid including multiple application icons), control center 1414b (e.g., when activated, it displays one or more device controls, such as a Wi-Fi control for toggling Wi-Fi settings, a Bluetooth control for toggling Bluetooth settings, and a mute control for toggling sound output on / off), and settings 1414c (e.g., when activated, it displays one or more settings for configuring the computer system, as described above regarding...). Figures 13A to 13GThe computer system 600 displays one or more of the following: the base control 1414d (e.g., when activated, it displays one or more controls for navigating to one or more applications (e.g., opening an application and / or an application running in the background)). In some embodiments, in response to detecting input to the additional option control 1416 (e.g., 732d), the computer system 600 displays the wallet control 1416a (e.g., when activated, it causes the computer system 600 to display one or more user interface objects for initiating a payment transaction), the side key 1416b (e.g., when activated, it causes the computer system 600 to perform one or more actions consistent with the pressing of the input mechanism 602b, such as shutting down the computer system 600), and the gesture mode 1416c (e.g., when activated, it causes the computer system 600 to begin operating in gesture mode). In some embodiments, in response to detecting input to the auto-scroll control 1412, the computer system 600 begins operating in auto-scroll mode (e.g., as described above regarding...). Figures 11A to 11H and Figures 12A to 12J (As described above). In some implementations, in response to detecting input to the exit control 1418, the computer system 600 stops displaying the menu 732 (e.g., as described above regarding...). Figure 7N (As described above). In some embodiments, menu 732 includes... Figure 14 One or more controls not shown. In some embodiments, the controls of menu 732 are in a position relative to... Figure 14 The hierarchical structure described herein is different from the hierarchical structure.

[0249] Figure 15 This is a flowchart illustrating a method 1500 for using a computer system according to some embodiments. Method 1500 is performed at a computer system (e.g., 100, 300, 500) (e.g., a wearable device (e.g., a smartwatch)) communicating with display generating components (e.g., a display controller, a touch-sensitive display system) and optical sensors (e.g., a heart rate sensor). In some embodiments, the computer system communicates with one or more sensors (e.g., one or more biometric sensors (e.g., optical sensors (e.g., heart rate sensors), cameras), gyroscopes, accelerometers)). In some embodiments, the computer system communicates with one or more input devices (e.g., touch-sensitive surfaces, microphones). In some embodiments, the computer system communicates with one or more output devices (e.g., one or more speakers, one or more microphones). Some operations in method 1500 may optionally be combined, some operations may optionally be changed in order, and some operations may optionally be omitted.

[0250] As described below, Method 1500 provides an intuitive way to navigate a user interface using hand gestures. This method reduces the cognitive burden on users navigating the user interface using hand gestures, thereby creating a more efficient human-computer interface. For battery-powered computing devices, it enables users to navigate the user interface faster, saves power more effectively, and increases the time interval between battery charging sessions.

[0251] The computer system displays (1502) a user interface including a first user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d), a second user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d), a third user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d), and an indication (e.g., a focus indicator, such as on 712a-712b, 910a-910h, or 1010a-1010d) of the selection of the first user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d) via a display generation component. (As discussed in 2b, 910a-910h, or 1010a-1010d) (e.g., visual indications (e.g., highlighting (e.g., the boundary of the first user interface), displaying text of the first user interface object, enlarging the first user interface object)) (e.g., not indicating that the second and third user interface objects are selected). In some embodiments, the first, second, and third user interface objects are different from each other. In some embodiments, the first user interface object is displayed between the second and third user interface objects. In some embodiments, the first, second, and third user interface objects are different user interface objects.

[0252] In some implementations, when displaying a user interface including a first user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d) (e.g., when the computer system is in a first operating mode (e.g., a first accessibility mode)), a second user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d), a third user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d), and an indication that the first user interface object has been selected (e.g., a focus indicator, as discussed around 712a-712b, 910a-910h, or 1010a-1010d), the computer system at least via (e.g., using) optics Sensors (e.g., and / or one or more other sensors, such as gyroscopes, accelerometers) are used to detect (1504) hand gestures (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) (e.g., first hand gestures) (e.g., those that do not touch the device (e.g., do not point to an object on the user interface and / or any device that communicates with the device)) (e.g., hand gestures that are not detected by one or more cameras of the device) (e.g., hand gestures that occur when the user's wrist is in a single position and / or does not move, hand gestures that are detected when the computer system is worn on the wrist (e.g., the user's wrist)).

[0253] In some implementations, in response to (1506) detecting a hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor (and when a user interface including a first user interface object, a second user interface object, and a third user interface object is displayed) (e.g., and when the computer system is operating in a first operating mode), and based on determining the hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, ... 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) are first-type gestures (e.g., 610, 620, 630, 640, 650) (e.g., types of hand gestures) (e.g., finger tapping / clamping gestures (e.g., gestures in which two or more fingers touch each other)) (and in some embodiments, first-type gestures are non-finger tapping / clamping gestures (e.g., clenching gestures (e.g., making a fist), multi-clenching gestures (e.g., gestures including multiple clenching gestures), finger extension gestures (e.g., gestures in which one or more fingers do not touch parts of the hand), multi-finger extension gestures (e.g., gestures including multiple extensions)). (gestures of the first user interface object) and / or any combination thereof, the computer system displays (1508) via the display generation component an indication (e.g., a focus indicator, such as those discussed around 712a-712b, 910a-910h, or 1010a-1010d) that a second user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d) has been selected (e.g., a new user interface object has been selected) (e.g., a focus indicator, such as those discussed around 712a-712b, 910a-910h, or 1010a-1010d) (e.g., visual indications (e.g., highlighting (e.g., the boundary of the second user interface), displaying text of the second user interface, enlarging the second user interface)) (e.g., stopping the display of an indication that a first user interface object has been selected and / or not displaying an indication that a third user interface object has been selected) The indication of selection (e.g., not indicating that the first and third user interface objects are selected). In some embodiments, the indication of the second user interface object is not previously displayed before the first indication of the first user interface object is displayed. In some embodiments, based on determining that the hand gesture is a first type of gesture, the computer system moves the indication from the first user interface object (e.g., from being displayed around, near (e.g., above, below, beside) the first user interface object) (e.g., the indication that the first user interface object is selected) to the second user interface object (e.g., to being displayed around, near (e.g., above, below, beside) the second user interface object) (e.g., the indication that the second user interface object is selected).

[0254] In some implementations, in response to (1506) detecting a hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor (and when a user interface including a first user interface object, a second user interface object, and a third user interface object is displayed) (e.g., and when the computer system is operating in a first operating mode), and based on determining the hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950...), c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) are second-type gestures (e.g., 610, 620, 630, 640, 650) that are different from the first-type gestures (e.g., types of hand gestures) (e.g., multi-finger tapping / clamping gestures (e.g., gestures in which two or more fingers touch each other multiple times)). In some embodiments, the first-type gesture is a non-finger tapping / clamping gesture (e.g., a clenching gesture (e.g., a gesture of making a fist), a multi-clenching gesture (e.g., a gesture including multiple clenching gestures), a finger-extending gesture (e.g., a gesture in which one or more fingers do not touch a part of the hand), a multi-finger-extending gesture (e.g., a gesture in which one or more fingers do not touch a part of the hand), or a multi-finger-extending gesture (e.g., a gesture in which one or more fingers do not touch a part of the hand). (e.g., gestures including multiple outstretched gestures) and / or any combination thereof), the computer system displays (1510) via a display generation component an indication (e.g., a focus indicator, such as those discussed around 712a-712b, 910a-910h, or 1010a-1010d) that a third user interface object (e.g., 712a-712b, 910a-910h, or 1010a-1010d) has been selected (e.g., a previously selected user interface object) (e.g., a focus indicator, such as those discussed around 712a-712b, 910a-910h, or 1010a-1010d) (e.g., visual indications (e.g., highlighting (e.g., the boundary of the third user interface), displaying text of the third user interface, enlarging the third user interface)) (e.g., not indicating that the first user interface object and the second user interface object have been selected) (e.g., The display of the indication that a first user interface object is selected is stopped and / or the indication that a second user interface object is selected is not displayed. In some embodiments, an indication of a third user interface object is displayed before the first indication of the first user interface object is displayed. In some embodiments, based on the determination that the hand gesture is a second type of gesture, the computer system moves the indication from the first user interface object (e.g., from being displayed around, near (e.g., above, below, beside) the first user interface object) (e.g., the indication that the first user interface object is selected) to the third user interface object (e.g., to being displayed around, near (e.g., above, below, beside) the third user interface object) (e.g., the indication that the second user interface object is selected).In some implementations, at least two of the indications for selecting a first user interface object, a second user interface object, and / or a third user interface object are of different sizes. Choosing whether to display the indication for selecting a second or third user interface object based on the type of detected hand gesture provides the user with more control over the system and helps the user navigate the user interface without touching the computer system, which can lead to more effective control of the user interface for some users.

[0255] In some embodiments, hand gestures (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) are detected based on heart rate data determined using data detected via an optical sensor. In some embodiments, a second type of gesture (e.g., 630 or 650) is a gesture type of multiple instances of a first type of gesture (e.g., 620 or 640) (e.g., multi-finger tapping / pinching, multi-finger extending gestures, and / or multi-clenching gestures). In some embodiments, a second type of gesture includes at least one instance of a first type of gesture. Choosing whether to display an indication that a second user interface object has been selected or an indication that a third user interface object has been selected based on the type of detected hand gesture (where the second type of gesture is a gesture type that is multiple instances of the first type of gesture) gives the user more control over the system and helps the user navigate the user interface to perform similar actions (e.g., one hand gesture includes another hand gesture) without touching the computer system, such as selecting an object to the right versus selecting an object to the left. This can result in more effective control of the user interface for some users.

[0256] In some implementations, in response to the detection of a hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor and based on the determination that the hand gesture is a third type of gesture (e.g., 620, 630, 640, or 650) that is different from the first and second types of gestures (e.g., the type of hand gesture), the computer system performs (e.g., via a display generation component) an operation (e.g., an action) corresponding to the selection of a first user interface object (e.g., selecting a play / pause button, selecting an interaction on a menu, selecting a cancel button, selecting an answer / reject button for an incoming telephone call) (e.g., not displaying a menu that includes one or more optional options). In some implementations, in response to the detection of a hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor and based on the determination that the hand gesture is a fourth type of gesture (e.g., 620, 630, 640, or 650) that is different from the first, second, and third types of gestures (e.g., the type of hand gesture), the computer system displays a menu including one or more optional options (e.g., as described below with respect to method 1600) via a display generation component (e.g., not executing the option corresponding to the selection of a first user interface object). Choosing between performing an operation corresponding to the selection of a first user interface object based on the type of detected hand gesture, or displaying a menu with one or more optional options, provides the user with more control over the system and helps the user navigate the user interface without touching the computer system. This can lead to more effective control of the user interface for some users.

[0257] In some implementations, before detecting a hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e), the computer system (e.g., 600) is in a first operating mode (e.g., wherein the computer system performs the operations described above in response to detecting one or more of a first type of gesture, a second type of gesture, a third type of gesture, and / or a fourth type of gesture) (e.g., operations as described above regarding...). Figures 7A to 7IThe operation mode described above). In some embodiments, as part of performing an operation corresponding to the selection of a first user interface object (e.g., 1412 or 736a) (e.g., as described above regarding the operation mode). Figures 11A to 11H and Figures 12A to 12J The computer system will switch from (e.g., 600) a first operating mode to a second operating mode (e.g., as described above regarding...). Figures 7I to 7U (The aforementioned) (e.g., a mode in which the cursor moves automatically on the display, a mode in which the user interface scrolls automatically in an direction (e.g., up, down, right, left, tilt), and a mode in which the detection of one or more of a first type of gesture, a second type of gesture, a third type of gesture, and / or a fourth type of gesture causes the computer system to perform an operation different from those performed when the computer system is operating in a first operating mode).

[0258] In some implementations, when the computer system (e.g., 600) is in a second operating mode, the computer system detects a second hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor. In some implementations, in response to detecting a second hand gesture at least via an optical sensor (e.g., when the computer system is in a second operating mode instead of a first operating mode), and based on determining that the second hand gesture is a first type (e.g., 620, 630, 640, or 650) gesture, the computer system switches between an active state (e.g., a resumed state) and an inactive state (e.g., a paused state) and a first automatic scrolling operation (e.g., as described above regarding...). Figures 12C to 12EThe discussed (e.g., not reversing the direction of the automatic scrolling operation) (e.g., operations where one or more user interface objects are selected sequentially (e.g., without user input)) (e.g., switching the automatic scrolling operation between an active and inactive state is an operation different from displaying an indication that a second user interface object has been selected). In some embodiments, based on determining that the hand gesture is a first type of gesture and based on determining that the scrolling operation is in an inactive state (e.g., paused state, scroll paused state, scroll off state), the computer system transitions the automatic scrolling operation from an inactive state to an active state (e.g., resumes the scrolling operation). In some embodiments, based on determining that the hand gesture is a second type of gesture and based on determining that the scrolling operation is in an active state (e.g., scroll on state, resumed state), the computer system transitions the automatic scrolling operation from an active state to an inactive state (e.g., pauses the scrolling operation). In some embodiments, based on determining that the automatic scrolling operation is active, the computer system displays a notification indicating that the automatic scrolling operation is active. In some embodiments, based on determining that the automatic scrolling operation is inactive, the computer system displays a notification indicating that the automatic scrolling operation is inactive and / or stops displaying a notification indicating that the automatic scrolling operation is active. In some implementations, in response to the detection of a second hand gesture at least via an optical sensor (e.g., when the computer system is in a second operating mode instead of a first operating mode) and based on the determination that the second hand gesture is a second type of gesture, the computer system reverses the first direction of the first auto-scrolling operation (e.g., does not switch the auto-scrolling operation between active and inactive states) (e.g., reversing the first direction of the auto-scrolling operation is an operation different from displaying an indication that a third user interface object has been selected). In some implementations, as part of reversing the direction of the auto-scrolling operation, the computer system selects user interface objects in a reverse order to the order in which the user interface was previously selected and / or reverses the interface scrolling in one direction. Choosing whether to switch the first auto-scrolling operation between active and inactive states or reverse the first direction of the auto-scrolling operation based on the type of detected hand gesture provides the user with more control over the system and helps the user navigate the user interface without touching the computer system, which can result in more efficient control of the user interface for some users.

[0259] In some implementations, in response to the detection of a second hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor: if the second hand gesture is determined to be a third type of gesture (e.g., 610, 620, 630, 640, or 650), the computer system performs a second operation (e.g., changing the speed of the automatic scrolling operation); and if the second hand gesture is determined to be a fourth type of gesture (e.g., 610, 620, 630, 640, or 650), the computer system performs a third operation different from the second operation (e.g., selecting the user interface at the current position of the user interface object highlighted by the scrolling operation when the second hand gesture is detected, ending the automatic scrolling mode). Choosing between performing a first or second action based on the type of detected hand gesture provides the user with more control over the system and helps the user navigate the user interface without touching the computer system, which can lead to more effective control of the user interface for some users.

[0260] In some implementations, when the computer system (e.g., 600) is in a second operating mode, the computer system performs a second automatic scrolling operation that scrolls a sequence of multiple interface objects in a second direction (e.g., as described above regarding...). Figures 12G to 12H (As discussed above). In some implementations, when the second automatic scrolling operation is performed, the computer system detects the end of the sequence of multiple user interface objects (e.g., detects that the scroll position is at or near the last user interface object in the sequence) and / or detects the boundary of the user interface (e.g., as described above regarding...). Figures 12G to 12H (As discussed above). In some implementations, in response to detecting the end of a sequence of multiple user interface objects, the computer system performs a third automatic scrolling operation that scrolls the sequence of multiple interface objects upwards in a third direction, different from (e.g., opposite to) the second direction (e.g., as discussed above). Figures 12G to 12H (As discussed). Automatically executing a third auto-scrolling operation in response to the detection of the end of a sequence of multiple user interface objects, scrolling the sequence upwards in a third direction different from the second, allows the computer system to provide the user with a method for automatically navigating the user interface without requiring additional user input to restart / reset auto-navigation after it has cycled through the user interface objects.

[0261] In some implementations, in response to the detection of a second hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor and based on the determination that the second hand gesture is a fourth type of gesture (e.g., 610, 620, 630, 640, or 650), the computer system switches from a second operating mode to a third operating mode (and / or terminates / exits the second operating mode). In some implementations, the third operating mode is the first operating mode. Switching the computer system from the second operating mode to the third operating mode based on the determination that the second hand gesture is a fourth type of gesture provides the user with more control over the system and helps the user navigate the user interface without touching the computer system, which can result in more effective control of the user interface for some users.

[0262] In some implementations, based on the determination that a second hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) is a first-type gesture (e.g., 610, 620, 630, 640, or 650), the computer system displays a notification (e.g., scroll paused, scroll stopped) indicating the status of the automatic scrolling operation (e.g., 716). Displaying the notification indicating the status of the automatic scrolling operation based on the determination that the second hand gesture is a first-type gesture provides the user with visual feedback in response to the detection that an operation has been performed by a hand gesture. This can prevent the user from mistakenly performing multiple gestures that might cause the computer system to perform unnecessary and / or unintentional operations.

[0263] In some embodiments, a third type of gesture (e.g., 610, 620, 630, 640, or 650) is a gesture type of multiple instances of a fourth type of gesture (e.g., 610, 620, 630, 640, or 650) (e.g., multi-finger tapping / pinching, multi-finger extending gestures, and / or multi-clenching gestures). In some embodiments, a third type of gesture includes at least one instance of a fourth type of gesture. In some embodiments, a fourth type of gesture (e.g., 610, 620, 630, 640, or 650) does not include multiple instances of a first type of gesture (e.g., 610, 620, 630, 640, or 650) (and / or a second type of gesture), and a third type of gesture (e.g., 610, 620, 630, 640, or 650) does not include multiple instances of a second type of gesture (e.g., 610, 620, 630, 640, or 650) (and / or a first type of gesture) (e.g., as described above regarding...). Figure 6 (As discussed). In some embodiments, the fourth type of gesture does not include the first type of gesture and / or the second type of gesture. In some embodiments, the third type of gesture does not include the first type of gesture and / or the second type of gesture.

[0264] In some implementations, in response to the detection of a second hand gesture (e.g., 750e, 750f, 750i, 750l, 750m, 750n, 950a, 950b, 950c, 950e, 950f, 950g, 1050b, 1050c, 1050d, or 1050e) at least via an optical sensor and based on the determination that a notification (e.g., a notification corresponding to a telephone call, email, text message, or voicemail message) was received within a threshold time period and based on the determination that the second hand gesture is a fourth type of gesture, the computer system performs an action related to the notification (e.g., answering a telephone call, responding to / opening a text message and / or email corresponding to the notification, playing a voicemail message) (e.g., as described above regarding...). Figure 7H (As discussed). Determining that a notification is received within a threshold time period and performing a notification-related action based on determining that the second hand gesture is of type four provides the user with more control over the system and helps the user navigate the user interface without touching the computer system (e.g., causing the computer system to perform notification-related actions), which can lead to more effective control of the user interface for some users.

[0265] In some implementations, one or more optional options (e.g., 732a-732d, 832a, or 1410-1418) include a first optional user interface object (e.g., 732a-732d, 832a, or 1410-1418) for changing the operations that one or more hand gestures can cause the computer system to perform. In some implementations, selection of the first optional user interface object causes the computer system to display multiple settings. In some implementations, each setting controls the operations that one or more hand gestures can cause the computer system to perform when the computer system detects one or more hand gestures (e.g., as described above regarding...). Figures 8A to 8J (As discussed). Displaying a menu that includes a first optional user interface object for changing what the computer system can do with one or more hand gestures allows the computer system to provide the user with the option to perform an operation without requiring more complex hand gestures. This reduces the number of hand gestures required to perform an operation and allows the user to customize the hand gestures that enable the operation to be performed.

[0266] In some implementations, one or more alternatives (e.g., 732a-732d, 832a, or 1410-1418) include a second alternative for switching the computer system to a fourth operating mode and a third alternative for switching the computer system to a fifth operating mode different from the fourth operating mode. In some implementations, selecting a second optional option (e.g., 732a-732d, 832a, or 1410-1418) causes the computer system to switch to a fourth operating mode (e.g., auto-scrolling mode, cursor mode (e.g., as discussed herein with respect to method 1600)). In some implementations, selecting a third optional option (e.g., 732a-732d, 832a, or 1410-1418) causes the computer system to switch to a fifth operating mode (e.g., auto-scrolling mode, cursor mode (e.g., as discussed herein with respect to method 1600)). A menu displaying a second optional option for switching the computer system to a fourth operating mode and a third optional option for switching the computer system to a fifth operating mode different from the fourth operating mode provides the user with more control over the computer system by allowing the user to switch the computer system to a different operating mode (e.g., without providing more complex hand gestures), and also reduces the number of hand gestures required to perform the operation.

[0267] In some embodiments, one or more optional options (e.g., 732a-732d, 832a, or 1410-1418) include a fourth optional option (e.g., 1416) for displaying one or more additional optional options (e.g., an option corresponding to an operation performed when input is detected at a location (e.g., a second location) of an optional user interface object; an option for performing an operation such as turning the computer system on / off, displaying a different menu and / or user interface; an option corresponding to an operation performed in response to the detection of input via one or more input devices communicating with the computer system). In some embodiments, selection of the fourth optional option causes the computer system (e.g., 600) to display one or more additional optional options that were not previously displayed before the selection of the fourth optional option (e.g., each additional optional option causes the computer system to perform one or more operations (e.g., different operations) when selected). In some embodiments, in response to the detection of the fourth optional option, the computer system stops displaying one or more optional options that were displayed before the detection of the selection of the fourth optional option. Displaying a menu that includes a fourth optional option for showing one or more additional options gives the user more control over the computer system by allowing the user to choose to display additional optional options that were not previously displayed, reduces the number of hand gestures required to perform additional operations, and reduces the number of optional user interface options displayed when the menu is initially displayed, thus eliminating clutter in the user interface.

[0268] In some implementations, based on determining that the corresponding user interface object (e.g., the selected user interface object, the user interface object surrounded by the focus indicator) is located at a first position on the user interface, the menu is displayed at the first position (e.g., as described above regarding...). Figure 11B (as discussed); and determine that, depending on whether the corresponding user interface object is in a first position on the user interface, the menu is displayed in a second position different from the first position (e.g., as mentioned above regarding...). Figure 11B (As discussed). Automatically selecting a location to display a menu based on the location of the corresponding user interface object allows a computer system to avoid displaying the menu at locations where the user might be interested in the user interface object (e.g., the selected user interface object), and also reduces the amount of input the user would need to make to move the menu without affecting the display of the user interface object.

[0269] In some implementations, when a third menu (e.g., 732) including one or more optional options (e.g., 1410a-732d, 732a, or 832-1418) is displayed (e.g., a menu including one or more optional options, a second menu including one or more optional options), the computer system detects a third hand gesture (e.g., as described above) at least via an optical sensor. Figures 10A to 10E(As discussed above). In some implementations, in response to the detection of a third hand gesture at least via an optical sensor (e.g., as mentioned above...), Figures 10A to 10E (As discussed) and based on the determination that the third hand gesture is a fourth type of gesture, the computer system stops displaying a third menu containing one or more optional options. Stopping the display of a menu containing one or more optional options based on the determination that the third hand gesture is a fourth type of gesture provides the user with more control over the system and helps the user navigate the user interface without touching the computer system, which can lead to more efficient control of the user interface for some users.

[0270] In some embodiments, after displaying a user interface including a first user interface object (e.g., 732a-732d, 832a, or 1410-1418), the computer system detects a fourth hand gesture at least via an optical sensor. In some embodiments, in response to detecting the fourth hand gesture via at least an optical sensor and determining that the fourth hand gesture is a fifth type of gesture (e.g., and in some embodiments, the fifth type of gesture is the same as the fourth type of gesture) that is different from the first type of gesture and the second type of gesture (and / or the first type of gesture), the computer system transitions the computer system from an inactive state (e.g., sleep state, hibernation state, low-power mode state, state where the display generating component is less bright than the active display generating component) to an active state (e.g., wake-up state, full-power state) (or transitions the computer system from an active state to an inactive state) (e.g., as described above regarding...). Figure 14 (As discussed in 1416b). The determination that the fourth hand gesture is the fifth type of gesture transitions the computer system from an inactive to an active state, giving the user more control over the system without touching it. This can lead to more effective control of the user interface for some users.

[0271] In some embodiments, when an indication that a second user interface object has been selected is displayed via a display generation component, the computer system detects a fifth hand gesture at least via an optical sensor. In some embodiments, in response to detecting a fourth hand gesture at least via an optical sensor: based on determining that the fifth hand gesture is a first type of gesture, the computer system displays an indication that a fourth first user interface object (e.g., the next selected user interface object) has been selected via the display generation component; and based on determining that the fifth hand gesture is a second type of gesture, the computer system displays an indication that a first user interface object (e.g., the previously selected user interface object) has been selected via the display generation component. In some embodiments, when an indication that a third user interface object has been selected is displayed via a display generation component, the computer system detects the corresponding hand gesture at least via an optical sensor. In some embodiments, in response to detecting a corresponding hand gesture and based on determining that the corresponding hand gesture is a first type of gesture, the computer system displays an indication that a fourth user interface object, different from the first, second, and third user interface objects, has been selected. In some embodiments, in response to detecting a corresponding hand gesture and based on determining that the corresponding hand gesture is a second type of gesture, the computer system displays an indication that a first user interface object has been selected. Choosing whether to display a fourth indication that a first user interface object has been selected, based on the type of hand gesture detected at the same time as the indication that a second user interface object has been selected, or to display an indication that a first user interface object has been selected, provides the user with more control over the system and helps the user navigate the user interface without touching the computer system (e.g., navigating the user interface consistently in the same way). This can result in more effective control of the user interface for some users.

[0272] In some implementations, when displaying a user interface including a first user interface object, the computer system detects a request to switch the computer system from a first operating mode to a fourth operating mode (e.g., as described above regarding...). Figures 13A to 13G and Figure 14(As discussed) (e.g., operating modes in which the computer system does not perform operations in response to the detection of a hand gesture and / or modes in which the computer system does not detect one or more hand gestures). In some embodiments, in response to detecting a request to switch the computer system from a first operating mode to a fourth operating mode, the computer system switches the computer system from the first operating mode to the fourth operating mode (e.g., as described above with respect to 7A to 7C). In some embodiments, while the computer system is in the fourth operating mode and while displaying a user interface including a first user interface object, a second user interface object, and a third user interface object, and an indication that the first user interface object has been selected, the computer system detects a sixth hand gesture at least via an optical sensor (e.g., as described above with respect to 7A to 7C). In some implementations, in response to the detection of a sixth hand gesture at least via an optical sensor (e.g., and when the computer system is in a fourth operating mode) (e.g., as described above with respect to 7A to 7C): Based on the determination that the hand gesture is a first type of gesture, the computer system continues to display an indication that a first user interface object has been selected (without displaying an indication that a second user interface object has been selected and / or an indication that a third user interface object has been selected via a display generation component and / or does not perform any operation) (e.g., as described above with respect to 7A to 7C); based on the determination that the hand gesture is a second type of gesture, the computer system continues to display an indication that a first user interface object has been selected (without displaying an indication that a third user interface object has been selected and / or an indication that a second user interface object has been selected via a display generation component and / or does not perform any operation) (e.g., as described above with respect to 7A to 7C). The continued display of the indication that a first user interface object has been selected based on the determination that the hand gesture is a first type of gesture (e.g., and when the computer system is in a fourth operating mode) provides the user with more control over the system to control the computer system detection and / or execution of operations related to the hand gesture in response to its detection.

[0273] It should be noted that the above is relative to method 1500 (e.g., Figure 15 The details of the process described herein also apply in a similar manner to the methods described herein. For example, method 1500 optionally includes one or more features of the various methods described herein with reference to method 1600. For example, method 1500 can be executed when a computer system switches from an operating mode in which the computer system operates using the technology of method 1600. For the sake of brevity, these details will not be repeated below.

[0274] Figure 16This is a flowchart illustrating a method for navigation using a computer system according to some embodiments. Method 1600 is performed at a computer system (e.g., 100, 300, 500) (e.g., a wearable device (e.g., a smartwatch)) communicating with display generating components (e.g., a display controller, a touch-sensitive display system). In some embodiments, the computer system communicates with one or more sensors (e.g., one or more biometric sensors (e.g., a heart rate sensor, a camera), a gyroscope, an accelerometer)). In some embodiments, the computer system communicates with one or more input devices (e.g., a touch-sensitive surface, a microphone). In some embodiments, the computer system communicates with one or more output devices (e.g., one or more speakers, one or more microphones). Some operations in method 1500 are optionally combined, some operations are optionally changed in order, and some operations are optionally omitted.

[0275] As described below, Method 1600 provides an intuitive way to navigate a user interface using hand gestures. This method reduces the cognitive burden on users navigating the user interface using hand gestures, thereby creating a more efficient human-computer interface. For battery-powered computing devices, it enables users to navigate the user interface faster, saves power more effectively, and increases the time interval between battery charging sessions.

[0276] The computer system displays (1602) a user interface (e.g., 710, 720, 810 or 780) via a display generation component. The user interface includes optional user interface objects (e.g., 720c, 732c, 810d or 832a) and a cursor (e.g., 742) displayed at a first position on the user interface (e.g., a position different from the position of the user interface object) (e.g., user interface object, non-optional user interface object).

[0277] When an optional user interface object (e.g., 720c, 732c, 810, or 780) and a cursor (e.g., 742) are displayed at a first location on the user interface (e.g., 710, 720, 810, or 780), the computer system detects (1604) a request to move the cursor (e.g., 742) from the first location on the user interface to a second location (e.g., a location different from the first location) (e.g., via a hand gesture (e.g., wrist tilt and / or one or more other hand gestures as described above with respect to method 1500)). In some embodiments, the request to move the cursor is detected based on movement of the computer system, movement of one or more devices associated with the computer system (e.g., a mouse), and / or one or more inputs / gestures (e.g., swipe gestures) detected on the display generation components of the computer system.

[0278] In response to (1606) detecting a request to move the cursor from the first position to the second position (e.g., 750p, 850c, 850d or...), Figure 7Y or Figure 7Z (The cursor is tilted in the middle), and the computer system displays (1608) the cursor at the second position (e.g., moves the cursor from the first position to the second position). In some embodiments, the computer system moves the cursor from the first position to the second position as part of displaying the cursor at the second position. In some embodiments, the computer system moves the cursor according to the direction of movement of the computer system.

[0279] In response to (1606) detecting a request to move the cursor from the first position to the second position (e.g., 750p, 850c, 850d or...), Figure 7Y or Figure 7Z The computer system displays (1610) an animation (e.g., 744) of a visual indication (e.g., 744) that provides how long the cursor (e.g., 742) needs to be in the second position to perform an operation, based on the determination that the second position corresponds to the position of an optional user interface object (e.g., 720c, 732c, 810d, or 832a) (and / or corresponds to an area of ​​the display (e.g., an edge)). This animation indicates how long the cursor (e.g., 742) needs to be in the second position to perform an operation. The animation may include actions such as filling, changing color, blinking, counting down / up, increasing / decreasing size, and fading in / out. The visual indication (e.g., 742) is updated within a certain time period (e.g., a time period less than or equal to a threshold time period) (e.g., when the cursor / visual indication is displayed in the second position) (e.g., no operation including selecting an optional user interface object is performed). In some embodiments, while the animation is being displayed (e.g., lasting less than a certain time period), the computer system detects a request to move the cursor, and in response to detecting the second request, the computer system stops displaying the animation. In some embodiments, after the animation is displayed, the computer system redisplays the cursor. In some implementations, the cursor is a visual indicator and / or includes a visual indicator.

[0280] In response to (1606) detecting a request to move the cursor from the first position to the second position (e.g., 750p, 850c, 850d or...), Figure 7Y or Figure 7ZIf the second position does not correspond to a position of an optional user interface object (e.g., 720c, 732c, 810d, or 832a) (and / or does not correspond to an area of ​​the display (e.g., an edge)), the computer system abandons displaying the (1612) animation (e.g., an animation providing an indication (e.g., 744)) (and continues to display the cursor) (e.g., does not perform actions including selecting an optional user interface object). In some embodiments, as part of displaying the animation providing a visual indication, the computer system replaces the display of the cursor with a visual indication. In some embodiments, when displaying the animation, the computer system detects a request to move the cursor from the second position to a third position, and in response to detecting the request to move the cursor from the third position to the second position, displays the cursor at the third position (e.g., does not display a visual indication and / or replaces the visual indication with the cursor). In some embodiments, the third position is different from the second position and / or the first position.

[0281] In some implementations, in response to detecting a request to move the cursor (e.g., 742) from a first position to a second position (e.g., 750p, 850c, 850d, or...), Figure 7Y or Figure 7Z The computer system performs an operation (e.g., regarding the tilt) and determines the position of the second position corresponding to the position of an optional user interface object (e.g., 720c, 732c, 810d, or 832a) and the cursor (e.g., 742) is displayed at the second position for a period of time exceeding a first threshold (e.g., a non-zero period of time (e.g., 1 second to 5 seconds)) (e.g., a period of time longer than or equal to the time the animation is displayed). Figures 7I to 7U The discussion, as mentioned above, is about Figures 8A to 8J (As discussed). In some implementations, as part of the execution of operations, the computer system selects an optional user interface object (e.g., regarding...). Figures 7I to 7U The discussion, as mentioned above, is about Figures 8A to 8J (As discussed) (e.g., highlighting an optional user interface object, launching a menu corresponding to the optional user interface object, launching an application corresponding to the optional user interface object). Automatically performing operations including selecting an optional user interface object based on determining that a second position corresponds to the position of an optional user interface object and that the cursor is displayed at the second position for more than a first threshold time period allows the computer system to automatically select a user interface object when predetermined conditions are met. Performing operations including selecting an optional user interface object based on determining that a second position corresponds to the position of an optional user interface object and that the cursor is displayed at the second position for more than a first threshold time period provides the user with greater control over the computer system by giving the user the ability to select a user interface object after a certain time period has elapsed since the cursor has been positioned at a certain location (e.g., by the user).

[0282] In some implementations, in response to detecting a request to move the cursor (e.g., 742) from a first position to a second position and based on determining that the second position corresponds to the position of an optional user interface object (e.g., 720c, 732c, 810d, or 832a) and the cursor (e.g., 742) being displayed at the second position for a period of time exceeding a second threshold (e.g., a non-zero period of time (e.g., 1 second to 5 seconds)) (e.g., a period of time longer than or equal to the time the animation was displayed), the computer system performs an operation (e.g., as per...). Figures 7I to 7U (As discussed). In some implementations, performing the operation includes launching an application corresponding to an optional user interface object (e.g., as discussed in the context of...). Figures 7I to 7U (As discussed). Automatically executing operations, including launching an application corresponding to the optional user interface object, based on determining that a second position corresponds to the position of an optional user interface object and that the cursor is displayed at the second position for more than a first threshold time period allows the computer system to automatically launch the application corresponding to the optional user interface object when predetermined conditions are met. Executing operations, including launching an application corresponding to the optional user interface object, based on determining that a second position corresponds to the position of an optional user interface object and that the cursor is displayed at the second position for more than a first threshold time period provides the user with greater control over the computer system by giving the user the ability to launch the application corresponding to the optional user interface object after a certain time period has elapsed since the cursor has been positioned at a certain location (e.g., by the user).

[0283] In some implementations, in response to detecting a request to move the cursor from a first position to a second position and determining that the second position does not correspond to the position of an optional user interface object (e.g., 720c, 732c, 810d, or 832a), the operation is abandoned (e.g., as described above regarding...). Figure 8C (For example, abandoning the display of a menu including one or more user interface objects via a display generation component to perform one or more operations at a second location, or abandoning the launch of an application corresponding to an optional user interface object). In some embodiments, the computer system abandons the operation based on the determination that the second location does not correspond to the location of an optional user interface object and the cursor is displayed and / or not displayed for a threshold time period. Selecting not to perform an operation based on the determination that the second location does not correspond to the location of an optional user interface object gives the computer system the ability not to perform an operation when specified conditions are not met and helps the computer system avoid performing unnecessary and / or unintended operations.

[0284] In some implementations, in response to detecting a request to move the cursor (e.g., 742) from a first position to a second position and determining that the cursor (e.g., 742) has not been displayed at the second position for more than a third threshold time period (e.g., a non-zero time period (e.g., 1 second to 5 seconds)) (e.g., a time period longer than or equal to the time the animation was displayed), the computer system abandons the operation (e.g., as...). Figure 8D (As shown). In some implementations, if the second position corresponds to and / or does not correspond to a position of an optional user interface object and the cursor is not displayed at the second position for more than a threshold time period, the computer system abandons the operation. Selecting not to perform an operation based on determining that the cursor is not displayed at the second position for more than a third threshold time period gives the computer system the ability not to perform an operation when specified conditions are not met and helps the computer system avoid performing unnecessary and / or unintended operations.

[0285] In some implementations, in response to a request to move the cursor (e.g., 742) from a first position to a second position and based on the determination that the second position corresponds to (e.g., is) a position in the area where the generated component is displayed (e.g., Figure 7Q 742 or Figure 7YIf the cursor is displayed at the second position for more than a fourth threshold time period (e.g., a non-zero time period, such as 1 second to 5 seconds, or a time period longer than or equal to the time the animation was displayed), the computer system performs a second operation. This second operation includes displaying a second user interface (e.g., 780 or 720) that is different from the user interface (e.g., 710 or 718) (and stopping the display of the user interface) (while continuing to display the cursor) (not displaying the menu) (e.g., a different operation, such as selecting a user interface object). In some embodiments, as part of displaying a second user interface different from the user interface, the computer system displays an animation of the second user interface (e.g., a sliding animation, a fade animation, a fading animation, and / or a sliding animation) instead of the user interface. In some embodiments, the animation includes sliding the user interface from the screen to the area and / or sliding a second user interface object on the screen to the area. In some embodiments, based on the determination that the second position corresponds to a position in the area of ​​the display and the cursor not being displayed at the second position for more than a threshold time period, the computer system abandons the second operation including displaying the second user interface and / or continues to display the user interface. In some embodiments, the second threshold time period is the same time period as the threshold time period. In some implementations, the second threshold time period is a different threshold time period from the threshold time period. In some implementations, the area does not include one or more optional user interface objects. In some implementations, the computer system displays an animation based on determining that the second position corresponds to (e.g., is) a position in the area. In some implementations, the computer system abandons displaying the animation based on determining that the second position corresponds to (e.g., is) a position in the area. In some implementations, the computer system displays an animation based on determining that the second position does not correspond to (e.g., is) a position in the area. In some implementations, the computer system abandons displaying the animation based on determining that the second position does not correspond to (e.g., is) a position in the area. Automatically performing a second operation, including displaying a second user interface different from the user interface, based on determining that the second position corresponds to a position in the area where the generated component is displayed and the cursor is displayed at the second position for more than a fourth threshold time period, allows the computer system to display another user interface when the cursor is in a specific area where the generated component is displayed. Performing a second operation, including displaying a second user interface different from the user interface, based on determining that the second position corresponds to a position in the area where the generated component is displayed and the cursor is displayed at the second position for more than a fourth threshold time period, provides the user with more control over the computer system by giving the user the ability to display another user interface when predetermined conditions are met.

[0286] In some implementations, the computer system performs an operation based on the determination that the second position corresponds to the position of an optional user interface object (e.g., 720c, 732c, 810d, or 832a) and the cursor (e.g., 742) is displayed at the second position for a period of time exceeding a fifth threshold (e.g., a non-zero period of time) (e.g., 1 second to 5 seconds) (e.g., when the cursor is displayed at the second position) (e.g., a period of time longer than or equal to the time the animation is displayed). In some embodiments, performing an operation includes displaying a menu (e.g., 732) (e.g., a user interface object) comprising one or more optional options (e.g., 1410a-732d and / or 732-1418) via a display generation component to perform one or more operations (e.g., one or more different operations) at a second location (e.g., an operation performed when a tap input (e.g., and / or a first type of input) (e.g., a single tap input / multiple tap input) is detected at the second location (e.g., launching an application), an operation performed when a press and hold input (e.g., and / or a second type of input different from the first type of input) is detected at the second location (e.g., bringing up a menu for deleting a user interface object), an operation performed when a swipe input (e.g., a third type of input different from the first and second types of input) is detected at the second location, and an operation to initiate an automatic scrolling operation at the second location). In some embodiments, the menu is displayed simultaneously with optional user interface objects and / or a cursor. In some embodiments, when the menu is displayed, one or more portions of the user interface (e.g., excluding optional user interface objects and / or a cursor) are de-displayed. In some embodiments, the menu is displayed on the user interface at a location that does not include an optional user interface object and / or the cursor (e.g., top / bottom). In some embodiments, based on determining that an optional user interface object and / or the cursor is displayed at a first corresponding location, the menu is displayed at a second corresponding location (e.g., on the side of the display opposite to the first corresponding location, different from the first location) (not displayed at the first corresponding location). In some embodiments, based on determining that an optional user interface object and / or the cursor is not displayed at the first corresponding location, the menu is displayed at the first corresponding location (not displayed at the second corresponding location). Automatically performing the operation of displaying the menu based on determining that the second location corresponds to the location of an optional user interface object and that the cursor is displayed at the second location for more than a first threshold time period allows the computer system to automatically display a menu that includes one or more optional options for performing one or more operations at the second location when predetermined conditions are met.The operation of displaying a menu that includes one or more optional options for performing one or more operations is performed by determining that the second position corresponds to the position of an optional user interface object and that the cursor is displayed at the second position for a period of time exceeding a first threshold. This provides the user with more control over the computer system by giving the user the ability to display a menu to perform one or more operations after the cursor has been (e.g., by the user) positioned at a certain position for a certain period of time.

[0287] In some embodiments, one or more optional options (e.g., 732a-732d) include a first optional option (e.g., 832a, 1410a, 1410b, 1410c, or 1412) for performing a selection operation, and wherein selection of the first optional option causes the computer system to perform the selection operation (e.g., an action performed when a tap is received at the second location) at a second location (e.g., on an optional user interface object). In some embodiments, in response to detecting selection of the first optional option, the computer system performs a selection operation including launching an application associated with the optional user interface object. In some embodiments, in response to detecting selection of the first optional option, the computer system performs a selection operation that includes highlighting (e.g., highlighting) the optional user interface object displayed at the second location. Displaying a menu that includes a first optional option for performing a selection operation provides the user with more control over the computer system by allowing the user to make the computer system perform a selection operation (e.g., without providing more complex hand gestures), and also reduces the number of hand gestures used to perform the operation.

[0288] In some implementations, one or more optional options include a second optional option (e.g., 832a) for performing an operation (e.g., launching an application in response to input (e.g., tapping, pressing and holding, swiping), moving an optional user interface object displayed at a second location in response to input (e.g., tapping, pressing and holding, swiping), displaying a notification (e.g., a pop-up) corresponding to the optional user interface object displayed at the second location), the operation corresponding to (e.g., as) an operation performed (and / or to be performed) in response to input (e.g., tapping, pressing and holding, swiping) (and / or gesture) detected at the second location (e.g., by the computer system), and wherein selection of the second optional option causes the computer system to perform an operation corresponding to the operation performed in response to input being detected at the second location (e.g., no input is detected at the second location). The menu that displays a second optional option for performing an operation corresponding to an operation performed in response to input detected at the second location provides the user with more control over the computer system (e.g., without providing more complex hand gestures and without requiring the user to touch the computer system and / or display generated components) by allowing the user to make the computer system perform an operation in response to input detected at the second location, and also reduces the number of hand gestures used to perform the operation.

[0289] In some embodiments, one or more optional options include a third optional option (e.g., 732a, 1410c, or 1412) for performing an automatic scrolling operation (e.g., as described with respect to method 1500), and wherein selection of the third optional option causes the computer system to automatically scroll through a plurality of optional user interface objects, including optional user interface objects. In some embodiments, as part of the automatic scrolling through a plurality of optional user interface objects, the computer system automatically scrolls from a second position (e.g., from the optional user interface object displayed at the second position). In some embodiments, as part of the automatic scrolling through a plurality of optional user interface objects, the computer system sequentially emphasizes (e.g., highlights) one or more optional user interface objects (e.g., as described with respect to method 1500). Displaying a menu including a third optional option for performing an automatic scrolling operation provides the user with more control over the computer system by allowing the user to make the computer system automatically scroll through a plurality of optional user interface objects (e.g., without providing more complex hand gestures), and also reduces the number of hand gestures used to perform the operation.

[0290] In some embodiments, one or more optional options include a fourth optional option (e.g., 1416) for displaying one or more additional options (e.g., an option corresponding to an operation performed when input is detected at a location (e.g., a second location) of an optional user interface object; an option for performing an operation such as turning the computer system on / off, displaying a different menu and / or user interface; and / or an option corresponding to an operation performed in response to the detection of input via one or more input devices communicating with the computer system). In some embodiments, selection of the fourth optional option causes the computer system to display one or more additional options that were not previously displayed before the selection of the fourth optional option was detected (e.g., ...). Figure 8J (as shown in 832a and the menu). In some embodiments, in response to the detection of a fourth optional option, the computer system stops displaying one or more optional options that were displayed before the selection of the fourth optional option was detected. Displaying a menu that includes a fourth optional option for displaying one or more additional optional options provides the user with more control over the computer system by allowing the user to select to display additional optional options that were not previously displayed, reduces the number of hand gestures used to perform additional operations, and reduces the number of selectable user interface options that were not previously displayed before the selection of the fourth optional option was detected.

[0291] In some implementations, one or more additional options (e.g., 1414 or 1416) that were not previously displayed are included before a selection of a fourth alternative option (e.g., 1414 or 1416) is detected. Figure 8J The 832a and menu displayed include corresponding additional options (e.g., 1414a-1414d or 1416a-1416c), wherein selecting the corresponding additional option does not cause the computer system to perform an operation not performed in the second location (e.g., an exit operation, a close menu operation). In some embodiments, one or more optional options do not include a selection operation that, when selected, causes the computer system to perform an operation not performed in the second location (e.g., not initiated in the second location, not performed in the second location, and / or not involving the second location) (e.g., any operation). A menu displaying one or more additional options that were not previously displayed before the detection of a selection of a fourth optional option includes the corresponding additional options, which provide the user with more control over the computer system by allowing the user to select an option to perform an operation not performed in the second location, and reduce the number of hand gestures used to perform an operation not performed in the second location.

[0292] In some embodiments, the computer system (e.g., 600) communicates with one or more input devices (e.g., 602a-602c). In some embodiments, one or more optional options include a fifth optional option (e.g., 732a, 1410c, or 1416b) for performing an operation corresponding to (e.g., as) an operation (e.g., pressing (e.g., single and / or multiple presses) on a hardware input device to display a menu, initiate a payment operation, initiate shutdown of the computer system, initiate use of voice assistance; rotate the hardware input device to move the cursor from one location to another; and / or rotate the hardware input device to zoom, scroll, and / or adjust one or more parts of the user interface), which is (and / or will be) performed in response to the detection of input via one or more input devices (e.g., hardware buttons (e.g., the crown of a smartwatch, a rotatable hardware button, a pressable / depressable hardware button), hardware sliders). In some embodiments, selection of the fifth optional option causes the computer system to perform an operation corresponding to an operation performed in response to the detection of input via one or more input devices (e.g., no input detected at one or more input devices). The menu displays a fifth optional option for performing an action corresponding to an action detected in response to input via one or more input devices. This provides the user with more control over the computer system by allowing the user to select an action corresponding to an action detected in response to input via one or more input devices (e.g., without providing more complex hand gestures and without requiring the user to touch one or more input devices), and also reduces the number of hand gestures used to perform the action.

[0293] In some implementations, one or more optional options include a sixth optional option (e.g., 732a-732d or 1410-1418) and a seventh optional option (e.g., 732a-732d or 1410-1418) (e.g., different from the sixth optional option). In some implementations, when the menu is displayed (e.g., 732), the computer system detects a request to move the cursor from a second position to a third position on the user interface (e.g., 750p, 850c, 850d, or...). Figure 7Y or Figure 7Z(The tilt in the text). In some embodiments, the second request to move the cursor is detected based on movement of the computer system, movement of one or more devices associated with the computer system (e.g., mouse), and / or one or more inputs / gestures (e.g., swipe gestures) detected on the display generation component of the computer system. In some embodiments, in response to detecting a request to move the cursor from a second position to a third position on the user interface, the computer system: performs a first operation corresponding to the sixth optional option (e.g., not performing the second operation) based on determining that the third position corresponds to the position of the sixth optional option and that the cursor is displayed at the third position for more than a sixth threshold time period (e.g., a non-zero time period (e.g., 1 second to 5 seconds)) (e.g., a time period longer than or equal to the time the animation is displayed); and performs a second operation different from the first operation (e.g., not performing the first operation) based on determining that the third position corresponds to the position of the sixth optional option and that the cursor is not displayed at the third position for more than a third threshold time period. In some embodiments, the computer system abandons performing the first operation based on determining that the third position corresponds to the position of the seventh optional option and that the cursor is not displayed at the third position for more than a third threshold time period. In some implementations, the third threshold time period is the same as the threshold time period. In other implementations, the third threshold time period is a different threshold time period. Performing different operations based on the cursor position displayed within the threshold time period provides the user with more control over the computer system, allowing different operations to be performed based on the cursor's position.

[0294] In some implementations (e.g., when a cursor is displayed at a second position and the second position corresponds to an optional user interface object), one or more operations include causing the computer system (e.g., 600) to display an operation of the user interface of an application corresponding to the optional user interface object (e.g., 720c, 732c, 810d, or 832a) (e.g., as described above regarding...). Figure 8J (As discussed) (e.g., not exiting the first mode).

[0295] In some implementations (e.g., when a cursor is displayed at a second position and the second position corresponds to an optional user interface object), one or more operations include operations that cause the computer system to switch from a first operating mode to a second operating mode (e.g., as discussed above with respect to 732a and / or 1412) (e.g., an operating mode in which an operation (e.g., an auto-scroll operation, a motion pointer operation) is initiated (and / or starts from) the second position) (e.g., not launching an application corresponding to an optional user interface object).

[0296] In some implementations, after displaying a menu that includes one or more optional options (e.g., 732) (e.g., displaying one or more menu objects when displaying a menu), the computer system detects a request to display a second user interface that is different from the user interface (e.g., as described above regarding...). Figures 7I to 7U and Figures 8A to 8J (As shown). In some embodiments, in response to detecting a request to display a second user interface, the computer system displays the second user interface (and when operating in the first operating mode). In some embodiments, the second user interface does not include a menu and includes a second optional user interface object (e.g., 732a-732d or 1410-1418) (e.g., which is different from the optional user interface object) and a cursor (e.g., 744) (e.g., as described above regarding...). Figures 7I to 7U and Figures 8A to 8J (As shown). In some implementations, when a second optional user interface object and a cursor are displayed, the computer system detects a request to move the cursor from a fourth position on the second user interface to a fifth position on the second user interface (e.g., as described above regarding...). Figures 7I to 7U and Figures 8A to 8J (As shown). In some embodiments, the third request to move the cursor is detected based on movement of the computer system, movement of one or more devices associated with the computer system (e.g., a mouse), and / or one or more inputs / gestures (e.g., swipe gestures) detected on the display generation component of the computer system. In some embodiments, in response to detecting a request to move the cursor from a fourth position on the second user interface to a fifth position on the second user interface, the computer system displays the cursor at the fifth position on the second user interface (e.g., as described above regarding...). Figures 7I to 7U and Figures 8A to 8J (As shown). In some embodiments, in response to detecting a request to move the cursor from a fourth position on the second user interface to a fifth position on the second user interface, the computer system, based on determining that the fifth position corresponds to the position of a second optional user interface object and that the cursor is displayed at the second position for a period of time exceeding a seventh threshold (e.g., a non-zero period of time (e.g., 1 second to 5 seconds)) (e.g., a period of time longer than or equal to the time the animation is displayed), redisplays a menu (e.g., 732) including one or more optional options via a display generation component (e.g., as described above regarding...). Figures 7I to 7U and Figures 8A to 8J(As shown) (e.g., when displaying a menu, one or more menu objects are displayed). In some implementations, the menu is abandoned if it is determined that the fifth position does not correspond to the position of the second optional user interface object and / or the cursor is not displayed at the second position for more than a seventh threshold time period. Automatically selecting to redisplay the menu based on determining that the fifth position corresponds to the position of the second optional user interface object and that the cursor is displayed at the second position for more than a seventh threshold time period allows the computer system to provide a consistent menu to the user when the same set of defined conditions is met for different optional user interface objects.

[0297] In some implementations, when it is determined that the computer system (e.g., 600) has tilted in the corresponding direction (e.g., 750p, 850c, 850d, or...), Figure 7Y or Figure 7Z A request to move the cursor (e.g., 742) from a first position to a second position is detected when the computer system is tilted (e.g., one side of the computer system (e.g., the right side, left side) is higher / lower than the other side of the computer system). In some embodiments, the determination is made using data detected via one or more gyroscopes communicating with the computer system. In some embodiments, the corresponding direction is from the higher side of the computer system to the lower side. In some embodiments, the cursor (e.g., 742) moves in the direction corresponding to the corresponding direction.

[0298] In some embodiments, the computer system displays a third user interface before displaying a user interface including optional user interface objects (e.g., 720c, 732c, 810d, or 832a) and a cursor (e.g., 742) displayed at a first location on the user interface. In some embodiments, when the third user interface (e.g., 710, 720, 810, or 780) is displayed (and in some embodiments, the user interface includes optional user interface objects), the computer system detects a request to enter a first operating mode (e.g., 750w, 850a). In some embodiments, in response to detecting a request to enter the first operating mode, the computer system transitions the computer system (e.g., 600) from a second operating mode (e.g., different from the first operating mode) to the first operating mode. In some embodiments, in response to detecting a request to enter the first operating mode, the computer system displays a cursor (e.g., at the first location) (e.g., and / or a user interface including optional user interface objects and a cursor displayed at the first location on the user interface). In response to a detected request to enter the first operating mode, a cursor is displayed to provide the user with feedback on the operation of the computer system in the first operating mode (e.g., one or more user actions may cause the computer system to perform one or more specific operations because the computer system is in the first operating mode).

[0299] In some implementations, as part of detecting a request to enter a first operating mode, the computer system detects that the computer system has been (or is being) shaken (e.g., 750w, 850a) (e.g., when the computer system is in a second operating mode). Switching the computer system from the second operating mode to the first operating mode in response to detecting that the computer system has been shaken provides the user with more control over the computer system by allowing the user to switch between modes by shaking the computer system (e.g., without providing input on the device's display generation component).

[0300] In some implementations, when the computer system is in a first mode, the computer system detects that the computer system (e.g., 600) has been shaken (e.g., 750w, 850a). In some implementations, in response to detecting that the computer system has been shaken, the computer system switches the computer system from the first operating mode to a third operating mode (and, in some implementations, the third operating mode is the second operating mode) (e.g., the first operating mode as described above with respect to method 1500). Switching the computer system from the first operating mode to the third operating mode in response to detecting that the computer system has been shaken provides the user with more control over the computer system by allowing the user to switch between modes by shaking the computer system (e.g., without providing input on the display generation component of the device).

[0301] In some implementations, as part of transitioning the computer system (e.g., 600) from a first operating mode to a third operating mode, the computer system stops displaying the cursor (e.g., 742). In some implementations, the computer system stops displaying the cursor in response to detecting that the computer system has been shaken (e.g., when the computer system is in the first mode). Stopping the cursor display in response to detecting that the computer system has been shaken provides feedback to the user that the computer system is not operating in the first operating mode (e.g., and / or one or more user actions cannot cause the computer system to perform one or more specific operations because the computer system is not in the first operating mode).

[0302] In some implementations, when the computer system is in a third operating mode and when the cursor is displayed (e.g., 742), the computer system detects a request to move the cursor from the sixth position to the seventh position. In some implementations, in response to detecting a movement of the cursor from the sixth position (e.g., 750p, 850c, 850d, or...), the computer system... Figure 7Y or Figure 7Z In some implementations, in response to a request to move the cursor from the sixth position (e.g., 750p, 850c, 850d, or...), the computer system displays the cursor at the seventh position. Figure 7Y or Figure 7Z The request to tilt the cursor to the seventh position (e.g., when the computer system is in the third operating mode) and, based on determining that the seventh position corresponds to the position of an optional user interface object and that the cursor is displayed at the seventh position for more than an eighth threshold time period (e.g., a non-zero time period (e.g., 1 second to 5 seconds)) (e.g., a time period longer than or equal to the time the animation is displayed), the computer system abandons the operation. Abandoning the operation based on determining that the seventh position corresponds to the position of an optional user interface object and that the cursor is displayed at the seventh position for more than an eighth threshold time period (e.g., when the computer system is in the third operating mode) provides the user with more control over the computer system to control when certain hand gestures will perform certain operations.

[0303] In some embodiments, in response to detecting a request to move the cursor (e.g., 742) from a first position to a second position and based on determining that the second position corresponds to the position of an optional user interface object (e.g., 720c, 732c, 810d, or 832a) and the cursor (e.g., 742) is displayed at the second position for more than a ninth threshold time period (e.g., a non-zero time period) (e.g., 1 second to 5 seconds) (e.g., when the cursor is displayed at the second position) (e.g., a time period longer than the time for displaying an animation), the computer system performs an operation. In some embodiments, as part of performing the operation, based on determining that one or more settings of the computer system (e.g., user settings) are in a first state, the computer system displays a second menu including one or more optional options (e.g., user interface objects) to perform one or more operations (e.g., one or more different operations) at the second position without launching a second application corresponding to the optional user interface object. In some embodiments, as part of performing the operation, based on determining that one or more settings of the computer system (e.g., user settings) are in a first state, the computer system launches a second application corresponding to the optional user interface object without displaying a second menu including one or more optional options (e.g., user interface objects) to perform one or more operations (e.g., one or more different operations) at the second position. Choosing whether to launch a second application and / or display a second menu when certain conditions are met provides users with more control over the computer system, allowing them to change whether one or more actions will result in launching an application and / or displaying a menu.

[0304] It should be noted that the above is relative to method 1600 (e.g., Figure 16The details of the process described herein also apply in a similar manner to the methods described herein. For example, method 1600 optionally includes one or more features of the various methods described herein with reference to method 1500. For example, method 1600 can be executed when a computer system switches from an operating mode in which the computer system operates using the technology of method 1500. For the sake of brevity, these details will not be repeated below.

[0305] For purposes of explanation, the foregoing description has been given by reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible based on the teachings above. These embodiments were chosen and described in order to best explain the principles of these techniques and their practical applications. Others skilled in the art will thus be able to best utilize these techniques and the various embodiments with various modifications suitable for the particular intended use.

[0306] While this disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of this disclosure and examples as defined by the claims.

[0307] As described above, one aspect of this technology involves collecting and using data from various sources to improve hand gesture detection. This disclosure anticipates that, in some instances, this collected data may include personal information that uniquely identifies or can be used to contact or locate specific individuals. Such personal information may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0308] This disclosure recognizes that the use of such personal information data in the techniques of this invention can benefit users. For example, personal information data can be used to improve the detection of a user's hand gestures. Thus, using such personal information data enables users to calculate their control over the data, which can improve the detection of hand gestures. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into a user's overall health status or can be used as positive feedback for individuals using technology to pursue health goals.

[0309] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as data collection and / or use change. Personal information from users should be collected for the entity's lawful and reasonable purposes and not shared or sold outside of these lawful uses. Furthermore, such collection / sharing should be conducted only after obtaining informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and processes. Furthermore, such entities may be subject to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Additionally, policies and practices should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including specific considerations regarding jurisdiction. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0310] Regardless of the foregoing, this disclosure also contemplates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure contemplates providing hardware and / or software components to prevent or block access to such personal information data. For example, with regard to user interface management (e.g., including navigation), the inventive technology can be configured to allow users to opt-in or opt-out at any time during or after service registration to participate in the collection of personal information data. In another example, users may choose not to provide heart rate data and / or other data that can be used to improve hand gesture detection. In yet another example, users may choose to limit the length of time heart rate data and / or other data is retained. In addition to providing "opt-in" and "opt-out" options, this disclosure also envisions providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0311] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0312] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be rendered inoperable due to the absence of all or part of such personal information data. For example, hand gestures can be detected based on non-personal information data or an absolute minimum amount of personal information (such as content requested by a device associated with a user, other non-personal information available through data detection services, or publicly available information).

Claims

1. A method for navigating a user interface using hand gestures, the method comprising: At the computer system that communicates with the display generation components and optical sensors: The display generation component displays a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected; When displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, hand gestures are detected via at least the optical sensors, wherein the computer system is in a first operating mode before the hand gestures are detected; In response to the detection of the hand gesture via at least the optical sensor: Based on the determination that the hand gesture is a first type of gesture, an indication that the second user interface object has been selected is displayed via the display generation component; Based on the determination that the hand gesture is a second type of gesture different from the first type of gesture, an indication that the third user interface object has been selected is displayed via the display generation component; and Based on the determination that the hand gesture is a third type of gesture, different from the first type of gesture and the second type of gesture, an operation corresponding to the selection of the first user interface object is performed, wherein performing the operation corresponding to the selection of the first user interface object includes switching the computer system from the first operating mode to the second operating mode; When the computer system is in the second operating mode, a second hand gesture is detected via at least the optical sensor; as well as In response to the detection of the second hand gesture via at least the optical sensor: Based on determining that the second hand gesture is the first type of gesture, a first automatic scrolling operation is switched between an active and inactive state; and Based on the determination that the second hand gesture is the second type of gesture, the first direction of the first automatic scrolling operation is reversed.

2. The method of claim 1, wherein the hand gesture is detected based on heart rate data determined using data detected via the optical sensor.

3. The method according to any one of claims 1 to 2, wherein the second type of gesture is a gesture type that is a plurality of instances of the first type of gesture.

4. The method according to any one of claims 1 to 2, further comprising: In response to the detection of the hand gesture via at least the optical sensor: Based on the determination that the hand gesture is a fourth type of gesture, different from the first type of gesture, the second type of gesture, and the third type of gesture, a menu including one or more optional options is displayed via the display generation component.

5. The method according to claim 4, further comprising: In response to the detection of the second hand gesture via at least the optical sensor: Based on the determination that the second hand gesture is the third type of gesture, the second operation is performed; as well as Based on the determination that the second hand gesture is the fourth type of gesture, a third operation, different from the second operation, is performed.

6. The method of claim 1, wherein the first automatic scrolling operation scrolls a sequence of multiple user interface objects in a second direction, and wherein the method further comprises: When performing the first automatic scrolling operation, the end of the sequence of the plurality of user interface objects is detected; as well as In response to detecting the end of the sequence of the plurality of user interface objects, a second automatic scrolling operation is performed, the second automatic scrolling operation scrolling the sequence of the plurality of user interface objects upward in a third direction different from the second direction.

7. The method according to claim 4, further comprising: In response to the detection of the second hand gesture via at least the optical sensor: Based on the determination that the second hand gesture is the fourth type of gesture, the computer system is switched from the second operating mode to the third operating mode.

8. The method according to claim 1, further comprising: Based on the determination that the second hand gesture is the first type of gesture, a notification indicating the status of the first automatic scrolling operation is displayed.

9. The method of claim 4, wherein the third type of gesture is a gesture type that is a plurality of instances of the fourth type of gesture.

10. The method of claim 4, wherein the fourth type of gesture does not include multiple instances of the first type of gesture, and the third type of gesture does not include multiple instances of the second type of gesture.

11. The method of claim 4, further comprising: The third hand gesture is detected via at least the aforementioned optical sensor; as well as In response to the detection of the third hand gesture via at least the optical sensor: Based on the determination that the notification was received within a threshold time period and based on the determination that the third hand gesture is the fourth type of gesture, an action related to the notification is performed.

12. The method of claim 4, wherein the one or more optional options include a first optional user interface object for changing the operation that one or more hand gestures can cause the computer system to perform, wherein selection of the first optional user interface object causes the computer system to display a plurality of settings, wherein each setting controls the operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

13. The method according to claim 4, wherein: The one or more optional options include a second optional option for switching the computer system to a fourth operating mode and a third optional option for switching the computer system to a fifth operating mode different from the fourth operating mode; Selecting the second optional option causes the computer system to switch to the fourth operating mode; and Selecting the third optional option causes the computer system to switch to the fifth operating mode.

14. The method of claim 4, wherein the one or more optional options include a fourth optional option for displaying one or more additional optional options, and wherein selection of the fourth optional option causes the computer system to display the one or more additional optional options that were not previously displayed prior to selection of the fourth optional option.

15. The method of claim 4, wherein the menu: Based on determining the first position of the corresponding user interface object on the user interface, it is displayed at the first position; and Based on the determination that the corresponding user interface object is not in the first position on the user interface, it is displayed in a second position different from the first position.

16. The method of claim 4, further comprising: When the menu, which includes one or more optional options, is displayed, a third hand gesture is detected via at least the optical sensor; as well as In response to the detection of the third hand gesture via at least the optical sensor: Based on the determination that the third hand gesture is the fourth type of gesture, the display of the menu including the one or more optional options is stopped.

17. The method according to any one of claims 1 to 2, further comprising: After the user interface including the first user interface object is displayed, a fourth hand gesture is detected via at least the optical sensor; as well as In response to the detection of the fourth hand gesture via at least the optical sensor: Based on the determination that the fourth hand gesture is a fifth type of gesture, different from the first type of gesture and the second type of gesture, the computer system is switched from an inactive state to an active state.

18. The method according to any one of claims 1 to 2, further comprising: When the indication that the second user interface object has been selected is displayed via the display generation component, a fifth hand gesture is detected via at least the optical sensor; as well as In response to the detection of the fifth hand gesture via at least the optical sensor: Based on the determination that the fifth hand gesture is the first type of gesture, an indication that a fourth user interface object has been selected is displayed via the display generation component; and Based on the determination that the fifth hand gesture is the second type of gesture, the indication that the first user interface object has been selected is displayed via the display generation component.

19. The method according to any one of claims 1 to 2, further comprising: When displaying the user interface including the first user interface object, a request to switch the computer system from the first operating mode to the fourth operating mode is detected; In response to detecting a request to switch the computer system from the first operating mode to the fourth operating mode, the computer system is switched from the first operating mode to the fourth operating mode. When the computer system is in the fourth operating mode and when the user interface including the first user interface object, the second user interface object and the third user interface object and the indication that the first user interface object is selected are displayed, a sixth hand gesture is detected via at least the optical sensor; as well as In response to the detection of the sixth hand gesture via at least the optical sensor: Based on the determination that the sixth hand gesture is the first type of gesture, the indication that the first user interface object has been selected continues to be displayed; and Based on the determination that the sixth hand gesture is the second type of gesture, the indication that the first user interface object has been selected continues to be displayed.

20. A computer system for navigating a user interface using hand gestures, wherein the computer system is configured to communicate with a display generation component and an optical sensor, the computer system comprising: One or more processors; as well as The memory stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: The display generation component displays a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected; When displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, hand gestures are detected via at least the optical sensors, wherein the computer system is in a first operating mode before the hand gestures are detected; In response to the detection of the hand gesture via at least the optical sensor: Based on the determination that the hand gesture is a first type of gesture, an indication that the second user interface object has been selected is displayed via the display generation component; Based on the determination that the hand gesture is a second type of gesture different from the first type of gesture, an indication that the third user interface object has been selected is displayed via the display generation component; and Based on the determination that the hand gesture is a third type of gesture, different from the first type of gesture and the second type of gesture, an operation corresponding to the selection of the first user interface object is performed, wherein performing the operation corresponding to the selection of the first user interface object includes switching the computer system from the first operating mode to the second operating mode; When the computer system is in the second operating mode, a second hand gesture is detected via at least the optical sensor; as well as In response to the detection of the second hand gesture via at least the optical sensor: Based on determining that the second hand gesture is the first type of gesture, a first automatic scrolling operation is switched between an active and inactive state; and Based on the determination that the second hand gesture is the second type of gesture, the first direction of the first automatic scrolling operation is reversed.

21. The computer system of claim 20, wherein the hand gesture is detected based on heart rate data determined using data detected via the optical sensor.

22. The computer system according to any one of claims 20 to 21, wherein the second type of gesture is a gesture type that is a plurality of instances of the first type of gesture.

23. The computer system according to any one of claims 20 to 21, wherein the one or more programs further include instructions for performing the following operations: In response to the detection of the hand gesture via at least the optical sensor: Based on the determination that the hand gesture is a fourth type of gesture, different from the first type of gesture, the second type of gesture, and the third type of gesture, a menu including one or more optional options is displayed via the display generation component.

24. The computer system of claim 23, wherein the one or more programs further include instructions for performing the following operations: In response to the detection of the second hand gesture via at least the optical sensor: Based on the determination that the second hand gesture is the third type of gesture, the second operation is performed; and Based on the determination that the second hand gesture is the fourth type of gesture, a third operation, different from the second operation, is performed.

25. The computer system of claim 20, wherein the first automatic scrolling operation scrolls a sequence of a plurality of user interface objects in a second direction, and wherein the one or more programs further include instructions for: During the execution of the first automatic scrolling operation, the end of the sequence of the plurality of user interface objects is detected; and In response to detecting the end of the sequence of the plurality of user interface objects, a second automatic scrolling operation is performed, the second automatic scrolling operation scrolling the sequence of the plurality of user interface objects upward in a third direction different from the second direction.

26. The computer system of claim 23, wherein the one or more programs further include instructions for performing the following operations: In response to the detection of the second hand gesture via at least the optical sensor: Based on the determination that the second hand gesture is the fourth type of gesture, the computer system is switched from the second operating mode to the third operating mode.

27. The computer system of claim 23, wherein the one or more programs further include instructions for performing the following operations: Based on the determination that the second hand gesture is the first type of gesture, a notification indicating the status of the first automatic scrolling operation is displayed.

28. The computer system of claim 23, wherein the third type of gesture is a gesture type that is a plurality of instances of the fourth type of gesture.

29. The computer system of claim 23, wherein the fourth type of gesture does not include multiple instances of the first type of gesture, and the third type of gesture does not include multiple instances of the second type of gesture.

30. The computer system of claim 23, wherein the one or more programs further include instructions for performing the following operations: Detecting a third hand gesture via at least the aforementioned optical sensor; and In response to the detection of the third hand gesture via at least the optical sensor: Based on the determination that the notification was received within a threshold time period and based on the determination that the third hand gesture is the fourth type of gesture, an action related to the notification is performed.

31. The computer system of claim 23, wherein the one or more optional options include a first optional user interface object for changing the operation that the computer system can perform when the one or more hand gestures are detected by the computer system, wherein selection of the first optional user interface object causes the computer system to display a plurality of settings, wherein each setting controls the operation that the one or more hand gestures can perform when the one or more hand gestures are detected by the computer system.

32. The computer system according to claim 23, wherein: The one or more optional options include a second optional option for switching the computer system to a fourth operating mode and a third optional option for switching the computer system to a fifth operating mode different from the fourth operating mode; Selecting the second optional option causes the computer system to switch to the fourth operating mode; and Selecting the third optional option causes the computer system to switch to the fifth operating mode.

33. The computer system of claim 23, wherein the one or more optional options include a fourth optional option for displaying one or more additional optional options, and wherein selection of the fourth optional option causes the computer system to display the one or more additional optional options that were not previously displayed prior to selection of the fourth optional option.

34. The computer system of claim 23, wherein the menu: Based on determining the first position of the corresponding user interface object on the user interface, it is displayed at the first position; and Based on the determination that the corresponding user interface object is not in the first position on the user interface, it is displayed in a second position different from the first position.

35. The computer system of claim 23, wherein the one or more programs further include instructions for performing the following operations: When displaying the menu including one or more optional options, a third hand gesture is detected via at least the optical sensor; and In response to the detection of the third hand gesture via at least the optical sensor: Based on the determination that the third hand gesture is the fourth type of gesture, the display of the menu including the one or more optional options is stopped.

36. The computer system according to any one of claims 20 to 21, wherein the one or more programs further include instructions for performing the following operations: After displaying the user interface including the first user interface object, a fourth hand gesture is detected via at least the optical sensor; and In response to the detection of the fourth hand gesture via at least the optical sensor: Based on the determination that the fourth hand gesture is a fifth type of gesture, different from the first type of gesture and the second type of gesture, the computer system is switched from an inactive state to an active state.

37. The computer system according to any one of claims 20 to 21, wherein the one or more programs further include instructions for performing the following operations: When the indication that the second user interface object has been selected is displayed via the display generation component, a fifth hand gesture is detected via at least the optical sensor; and In response to the detection of the fifth hand gesture via at least the optical sensor: Based on the determination that the fifth hand gesture is the first type of gesture, an indication that a fourth user interface object has been selected is displayed via the display generation component; and Based on the determination that the fifth hand gesture is the second type of gesture, the indication that the first user interface object has been selected is displayed via the display generation component.

38. The computer system according to any one of claims 20 to 21, wherein the one or more programs further include instructions for performing the following operations: When displaying the user interface including the first user interface object, a request to switch the computer system from the first operating mode to the fourth operating mode is detected; In response to detecting a request to switch the computer system from the first operating mode to the fourth operating mode, the computer system is switched from the first operating mode to the fourth operating mode. When the computer system is in the fourth operating mode and when the user interface including the first user interface object, the second user interface object and the third user interface object and the indication that the first user interface object is selected are displayed, a sixth hand gesture is detected via at least the optical sensor; as well as In response to the detection of the sixth hand gesture via at least the optical sensor: Based on the determination that the sixth hand gesture is the first type of gesture, the indication that the first user interface object has been selected continues to be displayed; and Based on the determination that the sixth hand gesture is the second type of gesture, the indication that the first user interface object has been selected continues to be displayed.

39. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with display generating components and optical sensors, the one or more programs including instructions for: The display generation component displays a user interface including a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected; When displaying the user interface including the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, hand gestures are detected via at least the optical sensors, wherein the computer system is in a first operating mode before the hand gestures are detected; In response to the detection of the hand gesture via at least the optical sensor: Based on the determination that the hand gesture is a first type of gesture, an indication that the second user interface object has been selected is displayed via the display generation component; Based on the determination that the hand gesture is a second type of gesture different from the first type of gesture, an indication that the third user interface object has been selected is displayed via the display generation component; and Based on the determination that the hand gesture is a third type of gesture, different from the first type of gesture and the second type of gesture, an operation corresponding to the selection of the first user interface object is performed, wherein performing the operation corresponding to the selection of the first user interface object includes switching the computer system from the first operating mode to the second operating mode; When the computer system is in the second operating mode, a second hand gesture is detected via at least the optical sensor; as well as In response to the detection of the second hand gesture via at least the optical sensor: Based on determining that the second hand gesture is the first type of gesture, a first automatic scrolling operation is switched between an active and inactive state; and Based on the determination that the second hand gesture is the second type of gesture, the first direction of the first automatic scrolling operation is reversed.

40. The computer-readable storage medium of claim 39, wherein the hand gesture is detected based on heart rate data determined using data detected via the optical sensor.

41. The computer-readable storage medium according to any one of claims 39 to 40, wherein the second type of gesture is a gesture type that is a plurality of instances of the first type of gesture.

42. The computer-readable storage medium according to any one of claims 39 to 40, wherein the one or more programs further include instructions for performing the following operations: In response to the detection of the hand gesture via at least the optical sensor: Based on the determination that the hand gesture is a fourth type of gesture, different from the first type of gesture, the second type of gesture, and the third type of gesture, a menu including one or more optional options is displayed via the display generation component.

43. The computer-readable storage medium of claim 42, wherein the one or more programs further include instructions for performing the following operations: In response to the detection of the second hand gesture via at least the optical sensor: Based on the determination that the second hand gesture is the third type of gesture, the second operation is performed; and Based on the determination that the second hand gesture is the fourth type of gesture, a third operation, different from the second operation, is performed.

44. The computer-readable storage medium of claim 39, wherein the first automatic scrolling operation scrolls a sequence of a plurality of user interface objects in a second direction, and wherein the one or more programs further include instructions for: During the execution of the first automatic scrolling operation, the end of the sequence of the plurality of user interface objects is detected; and In response to detecting the end of the sequence of the plurality of user interface objects, a second automatic scrolling operation is performed, the second automatic scrolling operation scrolling the sequence of the plurality of user interface objects upward in a third direction different from the second direction.

45. The computer-readable storage medium of claim 42, wherein the one or more programs further include instructions for: In response to the detection of the second hand gesture via at least the optical sensor: Based on the determination that the second hand gesture is the fourth type of gesture, the computer system is switched from the second operating mode to the third operating mode.

46. ​​The computer-readable storage medium of claim 39, wherein the one or more programs further include instructions for: Based on the determination that the second hand gesture is the first type of gesture, a notification indicating the status of the first automatic scrolling operation is displayed.

47. The computer-readable storage medium of claim 42, wherein the third type of gesture is a gesture type that is a plurality of instances of the fourth type of gesture.

48. The computer-readable storage medium of claim 42, wherein the fourth type of gesture does not include multiple instances of the first type of gesture, and the third type of gesture does not include multiple instances of the second type of gesture.

49. The computer-readable storage medium of claim 42, wherein the one or more programs further include instructions for: Detecting a third hand gesture via at least the aforementioned optical sensor; and In response to the detection of the third hand gesture via at least the optical sensor: Based on the determination that the notification was received within a threshold time period and based on the determination that the third hand gesture is the fourth type of gesture, an action related to the notification is performed.

50. The computer-readable storage medium of claim 42, wherein the one or more optional options include a first optional user interface object for changing the operation that one or more hand gestures can cause the computer system to perform, wherein selection of the first optional user interface object causes the computer system to display a plurality of settings, wherein each setting controls the operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

51. The computer-readable storage medium according to claim 42, wherein: The one or more optional options include a second optional option for switching the computer system to a fourth operating mode and a third optional option for switching the computer system to a fifth operating mode different from the fourth operating mode; Selecting the second optional option causes the computer system to switch to the fourth operating mode; and Selecting the third optional option causes the computer system to switch to the fifth operating mode.

52. The computer-readable storage medium of claim 42, wherein the one or more optional options include a fourth optional option for displaying one or more additional optional options, and wherein selection of the fourth optional option causes the computer system to display the one or more additional optional options that were not previously displayed prior to selection of the fourth optional option.

53. The computer-readable storage medium of claim 42, wherein the menu: Based on determining the first position of the corresponding user interface object on the user interface, it is displayed at the first position; and Based on the determination that the corresponding user interface object is not in the first position on the user interface, it is displayed in a second position different from the first position.

54. The computer-readable storage medium of claim 42, wherein the one or more programs further include instructions for: When displaying the menu including one or more optional options, a third hand gesture is detected via at least the optical sensor; and In response to the detection of the third hand gesture via at least the optical sensor: Based on the determination that the third hand gesture is the fourth type of gesture, the display of the menu including the one or more optional options is stopped.

55. The computer-readable storage medium according to any one of claims 39 to 40, wherein the one or more programs further include instructions for: After displaying the user interface including the first user interface object, a fourth hand gesture is detected via at least the optical sensor; and In response to the detection of the fourth hand gesture via at least the optical sensor: Based on the determination that the fourth hand gesture is a fifth type of gesture, different from the first type of gesture and the second type of gesture, the computer system is switched from an inactive state to an active state.

56. The computer-readable storage medium according to any one of claims 39 to 40, wherein the one or more programs further include instructions for: When the indication that the second user interface object has been selected is displayed via the display generation component, a fifth hand gesture is detected via at least the optical sensor; and In response to the detection of the fifth hand gesture via at least the optical sensor: Based on the determination that the fifth hand gesture is the first type of gesture, an indication that a fourth user interface object has been selected is displayed via the display generation component; and Based on the determination that the fifth hand gesture is the second type of gesture, the indication that the first user interface object has been selected is displayed via the display generation component.

57. The computer-readable storage medium according to any one of claims 39 to 40, wherein the one or more programs further include instructions for performing the following operations: When displaying the user interface including the first user interface object, a request to switch the computer system from the first operating mode to the fourth operating mode is detected; In response to detecting a request to switch the computer system from the first operating mode to the fourth operating mode, the computer system is switched from the first operating mode to the fourth operating mode. When the computer system is in the fourth operating mode and when the user interface including the first user interface object, the second user interface object and the third user interface object and the indication that the first user interface object is selected are displayed, a sixth hand gesture is detected via at least the optical sensor; as well as In response to the detection of the sixth hand gesture via at least the optical sensor: Based on the determination that the sixth hand gesture is the first type of gesture, the indication that the first user interface object has been selected continues to be displayed; and Based on the determination that the sixth hand gesture is the second type of gesture, the indication that the first user interface object has been selected continues to be displayed.

58. A computer system configured to communicate with a display generating component and an optical sensor, the computer system comprising: Apparatus for performing the method according to any one of claims 1 to 19.

59. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and an optical sensor, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 19.

Images