Interaction between input device and electronic device

Abstract

The invention relates to interaction between an input device and an electronic device. Some embodiments described in the present disclosure relate to displaying additional controls and / or information when an input device, such as a stylus, hovers on a user interface displayed by an electronic device. Some embodiments described in the present disclosure relate to providing feedback regarding a gesture of an input device relative to a surface. Some embodiments of the present disclosure relate to performing contextual actions in response to input provided from an input device. Some embodiments of the present disclosure relate to providing handwriting input for conversion to font-based text using an input device.

Description

[0001] This application is a divisional application of the invention patent application with the application date of May 10, 2023, application number 202380051163.X, and invention name “Interaction between input device and electronic device”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 364,488, filed May 10, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] The present invention generally relates to electronic devices that interact with input devices, and user interaction with such devices. Background Art

[0005] In recent years, users' interactions with electronic devices have increased significantly. These devices can be devices such as computers, tablet computers, televisions, multimedia devices, mobile devices, etc.

[0006] In some cases, users want to interact with electronic devices that have input devices such as styluses. Enhancing these interactions can improve the user's experience with the device and reduce user interaction time, which is especially important when the input device is battery-powered.

[0007] It is understood that the use of personally identifiable information should be subject to privacy policies and practices that are generally recognized to meet or exceed industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly stated to users. Summary of the Invention

[0008] Some embodiments described in this disclosure relate to displaying additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device. Some embodiments described in this disclosure relate to providing feedback about the posture of an input device relative to a surface. Some embodiments described in this disclosure relate to performing contextual actions in response to input provided from an input device. Some embodiments described in this disclosure relate to providing handwriting input using an input device for conversion to font-based text. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

[0010] Figure 1Ais a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0011] Figure 1B is a block diagram illustrating example components for event handling according to some embodiments.

[0012] Figure 2 A portable multifunction device with a touch screen according to some embodiments is illustrated.

[0013] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0014] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is illustrated.

[0015] Figure 4B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is illustrated according to some embodiments.

[0016] Figure 5A A personal electronic device according to some embodiments is illustrated.

[0017] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0018] Figures 5C to 5D Example components of a personal electronic device with a touch-sensitive display and an intensity sensor according to some embodiments are illustrated.

[0019] Figures 5E to 5H Illustrated are exemplary components and user interfaces of a personal electronic device according to some embodiments.

[0020] Figure 5I A block diagram illustrating an exemplary architecture for a device according to some embodiments of the present disclosure is illustrated.

[0021] Figures 6A to 6BF Illustrated are exemplary ways in which an electronic device displays additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by the electronic device, according to some embodiments.

[0022] Figures 7A to 7G is a flowchart illustrating a method for displaying additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device, according to some embodiments.

[0023] Figures 8A to 8AFIllustrated are exemplary ways in which an electronic device provides feedback regarding the posture of an input device relative to a surface according to some embodiments.

[0024] Figures 9A to 9K is a flowchart illustrating a method of providing feedback about the posture of an input device relative to a surface according to some embodiments.

[0025] Figures 10A to 10AP Illustrated are exemplary ways in which an electronic device according to some embodiments performs a contextual action in response to input provided from an input device.

[0026] Figures 11A to 11H is a flowchart illustrating a method for performing a contextual action in response to input provided from an input device according to some embodiments.

[0027] Figures 12A to 12AT Illustrated are exemplary ways in which an electronic device according to some embodiments uses an input device to provide handwriting input for conversion to font-based text.

[0028] Figures 13A to 13K is a flowchart illustrating a method of providing handwriting input using an input device for conversion to font-based text according to some embodiments. DETAILED DESCRIPTION

[0029] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0030] There is a need for electronic devices that provide efficient methods for interacting with input devices (e.g., from a stylus or other input device). Such techniques can reduce the cognitive burden on users of such devices. Furthermore, such techniques can reduce processor power and battery power that would otherwise be wasted on redundant user input.

[0031] Although the following description uses the terms "first," "second," etc. to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first touch can be named a second touch and similarly, a second touch can be named a first touch without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0032] The terms used in the description of the various described embodiments herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various described embodiments and in the appended claims, the singular forms "a" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "comprises" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groupings.

[0033] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that," or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining," or "in response to determining that," or "upon detecting [stated condition or event]," or "in response to detecting [stated condition or event]," depending on the context.

[0034] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Equipment and Device. Other portable electronic devices, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0035] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0036] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a rendering application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0037] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports various applications with a user interface that is intuitive and clear to the user.

[0038] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and external ports 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contacts on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0039] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, the force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a surrogate for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the surrogate measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurement). In some embodiments, the surrogate measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functionality that would otherwise be inaccessible to the user on a smaller device with limited real estate, which is used to display an indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0040] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical property of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0041] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0042] Memory 102 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0043] Peripherals interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0044] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuitry 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via a short-range communication radio. The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11d), IEEE 802.11e, IEEE 802.11f, IEEE 802.11g, IEEE 802.11g), IEEE 802.11f, IEEE 802.11g ... 11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)) and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0045] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and sends the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and sends the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or sent to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0046] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / transmit electrical signals from / to the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons and / or rocker buttons), dials, slide switches, joysticks, click wheels, and the like. In some alternative embodiments, the input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in ).

[0047] A quick press of the push button optionally releases the lock on the touch screen 112 or optionally initiates the process of unlocking the device using gestures on the touch screen, as described in U.S. patent application Ser. No. 11 / 322,549, filed Dec. 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," (i.e., U.S. Pat. No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0048] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or transmits electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0049] The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact on the touch screen 112 (and any movement or interruption of that contact) and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0050] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® Touch Controller from Apple Inc. (Cupertino, California). and iPod The technology used in

[0051] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive touchpads described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, whereas a touch-sensitive touchpad does not provide visual output.

[0052] The touch-sensitive display in some embodiments of the touch screen 112 is described in the following applications: (1) U.S. patent application No. 11 / 381,313, filed May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. patent application No. 10 / 840,862, filed May 6, 2004, “Multipoint Touchscreen”; (3) U.S. patent application No. 10 / 903,964, filed July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. patent application No. 11 / 48,264, filed January 31, 2005, “Gestures For Touch Sensitive Input Devices”; (5) U.S. patent application No. 11 / 38,590, filed January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices”; (6) U.S. patent application Ser. No. 11 / 228,758, filed Sep. 16, 2005, “Virtual Input Device Placement On ATouch Screen User Interface”; (7) U.S. patent application Ser. No. 11 / 228,700, filed Sep. 16, 2005, “Operation Of A Computer With ATouch Screen Interface”; (8) U.S. patent application Ser. No. 11 / 228,737, filed Sep. 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. patent application Ser. No. 11 / 367,749, filed Mar. 3, 2006, “Multi-Functional Hand-Held Device”. All of these applications are incorporated herein by reference in their entirety.

[0053] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or appendage, such as a stylus, a finger, or the like, to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positioning or commands for performing the actions desired by the user.

[0054] In some embodiments, the device 100 is a portable computing system that communicates with a display generation component (e.g., via wireless communication, via wired communication). The display generation component is configured to provide a visual output, such as a display via a CRT display, a display via an LED display, or a display via image projection. In some embodiments, the display generation component is integrated with the computer system (e.g., an integrated display, and / or touch screen 112). In some embodiments, the display generation component is separate from the computer system (e.g., an external monitor, and / or a projection system). As used herein, "displaying" content includes displaying content (e.g., video data rendered or decoded by the display controller 156) by sending data (e.g., image data or video data) to an integrated or external display generation component via a wired or wireless connection to visually generate the content.

[0055] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0056] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0057] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor is shown coupled to the optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, facing away from the touch screen display 112 on the front of the device, enabling the touch screen display to be used as a viewfinder for still and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing the user to optionally capture an image of the user for video conferencing while viewing other video conference participants on the touch screen display. In some embodiments, the positioning of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), allowing a single optical sensor 164 to be used with the touch screen display for both video conferencing and still and / or video image acquisition.

[0058] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite touch screen display 112 located on the front of device 100.

[0059] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent application serial numbers: 11 / 241,839, entitled “Proximity Detector In Handheld Device”; 11 / 240,788, entitled “Proximity Detector In Handheld Device”; 11 / 620,702, entitled “Using Ambient Light Sensor To Augment Proximity Sensor Output”; 11 / 586,862, entitled “Automated Response To And Sensing Of User Activity In Portable Devices”; and 11 / 638,251, entitled “Methods And Systems For Automatic Configuration Of Peripherals,” which are hereby incorporated by reference in their entireties. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0060] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1A A tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile outputs on the device 100 that can be felt by the user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0061] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripherals interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally implements as described in the following U.S. Patent Publication Nos. 20050190059, entitled “Acceleration-based Theft Detection System for Portable Electronic Devices” and 20060017692, entitled “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0062] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: active application state, which indicates which application, if any, is currently active; display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; sensor state, which includes information obtained from the device's various sensors and input control devices 116; and position information relating to the device's position and / or posture.

[0063] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, and / or power management), and facilitates communication between various hardware components and software components.

[0064] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), and / or FireWire) are suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, and / or a wireless LAN). In some embodiments, the external ports are connected to (trademark of Apple Inc.) devices.

[0065] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single-finger contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multiple-finger contact). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touchpad.

[0066] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, a mouse "click" threshold for a touchpad or touchscreen can be set to any one of a large range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some embodiments, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0067] Contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0068] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0069] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from the application specifying the graphics to be displayed, along with coordinate data and other graphic attribute data if necessary, and then generates screen image data for output to the display controller 156.

[0070] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0071] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (eg, contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0072] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services, such as a weather widget, a local yellow pages widget, and a map / navigation widget).

[0073] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0074] ● Contacts module 137 (sometimes called address book or contact list);

[0075] ● Telephone module 138;

[0076] ● Video conferencing module 139;

[0077] ● Email client module 140;

[0078] Instant messaging (IM) module 141;

[0079] ●Fitness support module 142;

[0080] A camera module 143 for still and / or video images;

[0081] Image management module 144;

[0082] ●Video player module;

[0083] Music player module;

[0084] ●Browser module 147;

[0085] ●Calendar module 148;

[0086] A widget module 149 , which optionally includes one or more of the following: a weather widget 149 - 1 , a stock market widget 149 - 2 , a calculator widget 149 - 3 , an alarm clock widget 149 - 4 , a dictionary widget 149 - 5 , and other widgets acquired by the user, and a user-created widget 149 - 6 ;

[0087] A widget creator module 150 for forming user-created widgets 149-6;

[0088] ●Search module 151;

[0089] ● Video and music player module 152, which merges the video player module and the music player module;

[0090] ●Note module 153;

[0091] ● Map module 154; and / or

[0092] ●Online video module 155.

[0093] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0094] In combination with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132 and text input module 134, the contacts module 137 is optionally used to manage an address book or contact list (for example, stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140 or IM 141; and so on.

[0095] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contacts module 137, modify an entered phone number, dial the corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0096] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, the text input module 134, the contact module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0097] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, transmitting, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and transmit emails with still images or video images captured by camera module 143.

[0098] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, sending the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the instant messages sent and / or received optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages transmitted using SMS or MMS) and Internet-based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

[0099] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a fitness; and displaying, storing, and sending fitness data.

[0100] In combination with the touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132 and image management module 144, the camera module 143 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0101] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0102] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with user instructions, including searching for, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0103] In combination with the RF circuitry 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140, and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar entries and / or to-do items) in accordance with user instructions.

[0104] In conjunction with the RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, the widget module 149 is a mini-application that is optionally downloaded and used by a user (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or created by a user (e.g., user-created widget 149-6). In some embodiments, a widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., a Yahoo! widget).

[0105] In combination with the RF circuitry 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, the widget creator module 150 is optionally used by a user to create widgets (e.g., converting a user-specified portion of a web page into a widget).

[0106] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0107] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0108] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the note module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0109] In combination with the RF circuitry 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0110] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video. Additional descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,67, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0111] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0112] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touchpad. By using a touch screen and / or a touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0113] A predefined set of functions that are exclusively performed through the touch screen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, the touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0114] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0115] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0116] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by application 136-1, a state queue for enabling the user to return to a previous state or view of application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0117] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 sends information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0118] In some embodiments, event monitor 171 transmits requests at predetermined intervals to peripheral device interface 118. In response, peripheral device interface 118 sends event information. In other embodiments, peripheral device interface 118 sends event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0119] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0120] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0121] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events that are recognized as valid input is optionally determined based at least in part on the hit view of the initial touch that initiated the touch-based gesture.

[0122] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should handle the sub-events. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0123] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0124] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver 182.

[0125] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0126] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of the data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0127] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event based on the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0128] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0129] The event comparator 184 compares the event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes the definition of the event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, the sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0130] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler that is associated with the sub-event and the object that triggered the hit test.

[0131] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0132] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0133] In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0134] In some embodiments, when one or more specific sub-events of an event are recognized, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from delivering (and deferred delivery of) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag retrieves the flag and performs a predefined process.

[0135] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0136] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positioning of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and transmits the display information to graphics module 132 for display on a touch-sensitive display.

[0137] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0138] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touchpad, such as taps, drags, and / or scrolls; stylus input; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0139] Figure 2A portable multifunction device 100 with a touch screen 112 according to some embodiments is illustrated. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of the finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon does not select the corresponding application when the gesture corresponding to selection is a tap.

[0140] In some embodiments, the stylus 203 is an active device and includes one or more electronic circuits. For example, the stylus 203 includes one or more sensors and one or more communication circuits (such as the communication module 128 and / or the RF circuit 108). In some embodiments, the stylus 203 includes one or more processors and a power system (e.g., similar to the power system 162). In some embodiments, the stylus 203 includes an accelerometer (such as the accelerometer 168), a magnetometer, and / or a gyroscope that can determine the positioning, angle, position, and / or other physical characteristics of the stylus 203 (e.g., such as whether the stylus is put down, tilted toward or away from the device, and / or close to or away from the device). In some embodiments, the stylus 203 communicates with the electronic device (e.g., via the communication circuit, through a wireless communication protocol such as Bluetooth) and sends the sensor data to the electronic device. In some embodiments, the stylus 203 can determine (e.g., via an accelerometer or other sensor) whether the user is holding the device. In some embodiments, the stylus 203 can accept tap input (e.g., a single or double tap) from a user on the stylus 203 (e.g., received by an accelerometer or other sensor) and interpret the input as a command or request to perform a function or change to a different input mode.

[0141] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications optionally being executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0142] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding it in the depressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions via the microphone 113. The device 100 also optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0143] Figure 3 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above referenced device). Figure 1A Tactile output generator 167 as described above), sensor 359 (e.g., optical sensor, acceleration sensor, proximity sensor, touch sensitive sensor and / or contact intensity sensor (similar to the above reference Figure 1A310). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores data related to portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100( Figure 1A )'s memory 102 optionally does not store these modules.

[0144] Figure 3 Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the memory device mentioned previously.Each module in the above-mentioned modules corresponds to the instruction set for performing the function described above.Above-mentioned modules or programs (for example, instruction set) need not be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the subset of above-mentioned modules and data structure.In addition, memory 370 optionally stores additional modules and data structure not described above.

[0145] Attention is now turned to an embodiment of a user interface, optionally implemented on, for example, portable multifunction device 100 .

[0146] Figure 4A An exemplary user interface for an application menu on portable multifunction device 100 is illustrated according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0147] Signal strength indicators 402 for wireless communications such as cellular and Wi-Fi signals;

[0148] ●Time 404;

[0149] Bluetooth indicator 405;

[0150] Battery status indicator 406;

[0151] A tray 408 with icons for commonly used applications such as:

[0152] o An icon 416 labeled "Phone" for the phone module 138, which optionally includes an indicator 414 of the number of missed calls or voicemails;

[0153] o An icon 418 of the email client module 140 labeled "Mail," which optionally includes an indicator 410 of the number of unread emails;

[0154] o An icon 420 labeled "Browser" for the browser module 147; and

[0155] o An icon 422 labeled “iPod” for the video and music player module 152 (also referred to as the iPod (trademark of Apple Inc.) module 152); and

[0156] Icons for other apps, such as:

[0157] o Icon 424 labeled "Messages" of the IM module 141;

[0158] o An icon 426 labeled “Calendar” of the calendar module 148;

[0159] o Icon 428 labeled "Photos" of the image management module 144;

[0160] o An icon 430 labeled “Camera” of the camera module 143;

[0161] ○ An icon 432 labeled “Online Video” of the online video module 155;

[0162] ○ Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;

[0163] o An icon 436 labeled “Map” of the map module 154;

[0164] ○ Icon 438 labeled “Weather” of weather widget 149-1;

[0165] ○ Icon 440 labeled “Clock” of the alarm clock widget 149-4;

[0166] o An icon 442 labeled “Fitness Support” of the fitness support module 142;

[0167] o An icon 444 labeled "Notes" for the notes module 153; and

[0168] o An icon 446 of a settings application or module labeled “Settings” that provides access to settings for the device 100 and its various applications 136 .

[0169] It should be pointed out that Figure 4A The illustrated icon labels are merely exemplary. For example, the icon 422 of the video and music player module 152 is labeled "Music" or "Music Player." Other labels are optionally used for various application icons. In some embodiments, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0170] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is illustrated as being separate from a display 450 (eg, touch screen display 112). Figure 3 a tablet device or touchpad 355) (e.g., Figure 3 Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0171] Although some of the examples below will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B , 460 corresponds to 468 and 462 corresponds to 470 ) at contact with touch-sensitive surface 451 (e.g., Figure 4B 460 and 462 in FIG. 4. Thus, when the touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B When the user interface 450 in FIG. 1 is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0172] Additionally, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact) followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, when the cursor is at the location of the tap gesture, the tap gesture is optionally replaced by a mouse click (e.g., instead of detecting contact and then ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0173] Figure 5A An exemplary personal electronic device 500 is illustrated. The device 500 includes a body 502. In some embodiments, the device 500 may include a body 502 relative to the devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in . In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. Alternatively, or in addition to touch screen 504, device 500 also has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of the contact (e.g., touch) applied. The one or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to the touch based on the intensity of the touch, which means that touches of different intensities can invoke different user interface operations on device 500.

[0174] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0175] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0176] Figure 5B An exemplary personal electronic device 500 is depicted. In some embodiments, the device 500 may include a Figure 1A 、 Figure 1B and Figure 3Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. The I / O portion 514 can be connected to a display 504, which can have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, the I / O portion 514 can be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, the input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, the input mechanism 508 is optionally a button.

[0177] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to I / O portion 514.

[0178] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may, for example, cause the computer processors to perform the techniques described below, including processes 700, 900, 1100, and 1300 (Figures 7, 9, 11, and 13). A computer-readable storage medium may be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with an instruction execution system, apparatus, and device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory such as flash memory, solid-state drives, and the like. The personal electronic device 500 is not limited to Figure 5B components and configurations, but may include other components or additional components in a variety of configurations.

[0179] In addition, in the method described herein where one or more steps depend on having met one or more conditions, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions of the steps in the method of determining the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then those of ordinary skill will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on having met one or more conditions can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require a system or computer-readable medium to declare that the system or computer-readable medium includes instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether a possible situation has been met without explicitly repeating the steps of the method until all conditions of the steps in the method of determining the method have been met. Those of ordinary skill in the art will also understand that, similar to the method with a contingent step, a system or computer-readable storage medium can repeat the steps of the method as needed multiple times to ensure that all contingent steps have been performed.

[0180] As used herein, the term "indicator" refers to an indication that is optionally provided on device 100, 300, and / or 500 ( Figure 1A 、 Figure 3 and Figures 5A to 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.

[0181] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in FIG, the particular user interface element is adjusted according to the detected input. In the case of a touch screen display (e.g., a touch screen display) that enables direct interaction with user interface elements on the touch screen display Figure 1A touch-sensitive display system 112 or Figure 4AIn some implementations of the touch screen 112 in FIG, 2 , a contact detected on the touch screen acts as a “focus selector” such that when input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0182] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0183] Figure 5C Detecting multiple contacts 552A-552E on touch-sensitive display 504 using multiple intensity sensors 524A-524D is illustrated. Figure 5C Also included is an intensity graph that shows the current intensity measurements of intensity sensors 524A-524D relative to intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are both 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are both 7 intensity units. In some implementations, the cumulative intensity is the sum of the intensity measurements of multiple intensity sensors 524A-524D, which in this example is 32 intensity units. In some embodiments, each contact is assigned a corresponding intensity, which is a portion of the cumulative intensity. Figure 5DThe assignment of cumulative strengths to contacts 552A-552E based on their distances from the center of force 554 is illustrated. In this example, each of contacts 552A, 552B, and 552E is assigned a strength of the contact of 8 strength units of the cumulative strength, and each of contacts 552C and 552D is assigned a strength of the contact of 4 strength units of the cumulative strength. More generally, in some implementations, each contact j is assigned a corresponding strength Ij that is a fraction of the cumulative strength A according to a predefined mathematical function Ij=A·(Dj / ΣDi), where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i=1 to the last) from the center of force. Reference may be performed using an electronic device similar to or identical to device 100, 300, or 500. Figures 5C to 5D In some embodiments, the characteristic intensity of the contact is based on one or more intensities of the contact. In some embodiments, the intensity sensor is used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included in the Figures 5C to 5D To assist readers.

[0184] In some embodiments, a portion of a gesture is identified for use in determining the characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position and reaches an ending position where the contact strength increases. In this example, the characteristic strength of the contact at the ending position is optionally based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the ending position). In some embodiments, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic strength.

[0185] The intensity of the contact on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move the focus selector in accordance with the movement of the contact on the touch-sensitive surface, without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0186] An increase in contact feature intensity from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact feature intensity from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact feature intensity from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0187] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting an increase in the intensity of the contact (or multiple contacts) to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting an increase in the intensity of the corresponding contact to above the press input intensity threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below the press input intensity threshold, and the corresponding operation is performed in response to detecting a subsequent decrease in the intensity of the corresponding contact to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0188] Figures 5E to 5HDetection of a gesture is illustrated, the gesture comprising contact 562 with an intensity ranging from below Figure 5E Light press intensity threshold in L ”) increases to a value higher than Figure 5H The deep press intensity threshold in D ”). On the displayed user interface 570 including application icons 572A-572D displayed in the predefined area 574, when the cursor 576 is displayed over the application icon 572B corresponding to application 2, a gesture performed using the contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch-sensitive display 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines whether the intensity of the contact 562 is within the deep press intensity threshold (e.g., “IT D ”) reaches a peak above. Contact 562 is maintained on touch-sensitive surface 560. In response to detecting the gesture, and according to the intensity rising to the deep press intensity threshold (e.g., “IT D ”) above contact 562, displays scaled representations 578A-578C (e.g., thumbnails) of documents recently opened for Application 2, as shown in FIG. Figures 5F to 5I In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in the Figures 5E to 5H To assist readers.

[0189] In some embodiments, the display of representations 578A-578C includes animation. For example, representation 578A is initially displayed near application icon 572B, as shown in FIG. Figure 5F As the animation progresses, representation 578A moves upward and representation 578B is displayed near application icon 572B, as shown. Figure 5G Then, representation 578A moves upward, 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown in FIG. Figure 5H . Indicators 578A-578C form an array above icon 572B. In some embodiments, the animation progresses according to the intensity of contact 562, such as Figures 5F to 5G , where representations 578A-578C appear and decrease as the intensity of contact 562 moves toward a deep press intensity threshold (e.g., “IT D In some embodiments, the intensity according to which the animation progresses is the characteristic intensity of the contact. The reference may be performed using an electronic device similar to or identical to device 100, 300, or 500. Figures 5E to 5H The operation described.

[0190] Figure 5I A block diagram illustrating an exemplary architecture for device 580 according to some embodiments of the present disclosure is shown. Figure 5I In an embodiment, media content or other content is optionally received by device 580 via network interface 582, which is optionally a wireless connection or a wired connection. The one or more processors 584 optionally execute any number of programs stored in memory 586 or storage, which optionally include instructions for performing one or more of the methods and / or processes described herein (e.g., methods 700, 900, 1100, 1300, 1500, and 1700).

[0191] In some embodiments, the display controller 588 causes the various user interfaces of the present disclosure to be displayed on the display 594. Additionally, input to the device 580 is optionally provided by a remote control 590 via a remote control interface 592, which is optionally a wireless or wired connection. In some embodiments, input to the device 580 is provided by a multifunction device 591 (e.g., a smartphone) on which a remote control application is running that configures the multifunction device to simulate remote control functionality, as will be described in more detail below. In some embodiments, the multifunction device 591 corresponds to Figure 1A and Figure 2 Device 100, Figure 3 Device 300 and Figure 5A It should be understood that one or more of the devices 500 in Figure 5I The embodiments are not meant to limit the features of the device of the present disclosure, and other components that facilitate other features described in the present disclosure are optionally included. Figure 5I In some embodiments, the device 580 optionally corresponds to Figure 1A and Figure 2 The multifunctional device 100, Figure 3 Device 300 and Figure 5A One or more of the devices 500 in; the network interface 582 optionally corresponds to Figure 1A and Figure 2 RF circuit 108, external port 124 and peripheral device interface 118 in the Figure 3 One or more of the network communication interfaces 360 in; the processor 584 optionally corresponds to Figure 1A The processor 120 and Figure 3 One or more of the CPUs 310 in; the display controller 588 optionally corresponds to Figure 1A The display controller 156 and Figure 3 One or more of the I / O interfaces 330 in; the memory 586 optionally corresponds to Figure 1AThe memory 102 and Figure 3 One or more of the memories 370 in; the remote control interface 592 optionally corresponds to Figure 1A The peripheral device interface 118 and the I / O subsystem 106 (and / or its components) and Figure 3 One or more of the I / O interfaces 330 in; the remote control 590 optionally corresponds to and / or includes Figure 1A Speaker 111, touch-sensitive display system 112, microphone 113, optical sensor 164, contact intensity sensor 165, tactile output generator 167, other input control devices 116, accelerometer 168, proximity sensor 166 and I / O subsystem 106 and Figure 3 One or more of keyboard / mouse 350, touchpad 355, tactile output generator 357, and contact intensity sensor 359 in FIG. 4 and touch-sensitive surface 451 in FIG. 4 ; and display 594 optionally corresponds to Figure 1A and Figure 2 The touch-sensitive display system 112 and Figure 3 One or more of the displays 340 in .

[0192] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).

[0193] For ease of explanation, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting any of the following: contact intensity increasing above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing below the press input intensity threshold, and / or contact intensity decreasing below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0194] As used herein, an "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched (e.g., become open) on the device. In some embodiments, a downloaded application becomes an installed application using an installer that extracts program portions from the downloaded software package and integrates the extracted portions with the computer system's operating system.

[0195] As used herein, the term "open application" or "executing application" refers to a software application that has persisted state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or executing application is optionally any of the following types of applications:

[0196] The active app currently displayed on the display of the device on which the app is being used;

[0197] ● Background applications (or background processes), which are not currently displayed but for which one or more processes are being processed by one or more processors; and

[0198] ●Not running but stored in memory (volatile and non-volatile respectively)

[0199] A suspended or dormant application that contains status information that can be used to resume execution of the application.

[0200] As used herein, the term "closed application" refers to a software application that does not retain state information (e.g., the state information of a closed application is not stored in the device's memory). Therefore, closing an application includes stopping and / or removing the application's application process and removing the application's state information from the device's memory. Generally speaking, while in a first application, opening a second application does not close the first application. When the second application is displayed and the first application stops displaying, the first application becomes a background application.

[0201] Attention is now turned to an embodiment of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0202] User interface and associated processes

[0203] Hover events and controls

[0204] Users interact with electronic devices in many different ways, including using peripheral devices that communicate with these devices. In some embodiments, an electronic device receives an indication that a peripheral device (e.g., a stylus) is close to but not in contact with a surface (such as a touch-sensitive surface that communicates with an electronic device). The embodiments described herein provide a way for an electronic device to respond to such indications, and, for example, provide a visual preview or other indication of the interaction with the electronic device based on the current positioning of the input device relative to the surface, thereby enhancing the interaction with the device. Enhancing the interaction with the device reduces the amount of time required for the user to perform an operation, thereby reducing the power consumption of the device and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, this person is optionally referred to as a user of the device.

[0205] 6A to 6BF Illustrated are exemplary ways in which an electronic device, according to some embodiments, displays selectable options and / or information in response to detecting that an input device is hovering over a surface associated with the electronic device. The embodiments in these figures are used to illustrate the processes described below, including reference to Figures 7A to 7G Describe the process.

[0206] Figure 6A The electronic device 500 is illustrated as displaying a user interface 609 (e.g., via a display device and / or via a display generating component). In some embodiments, the user interface 609 is displayed via a display generating component. In some embodiments, the display generating component is a hardware component (e.g., including electronic components) that is capable of receiving display data and displaying a user interface. In some embodiments, examples of display generating components include a touch screen display (e.g., touch screen 504), a monitor, a television, a projector, an integrated, discrete, or external display device, or any other suitable display device that communicates with the device 500. In some examples, a surface (e.g., a touch-sensitive surface) communicates with the device 500. For example, in Figure 6A , device 500 includes a touch screen 504 that displays a user interface and detects touch or hover interactions with the device 500 .

[0207] In some embodiments, user interface 609 is a user interface of an application or a user interface in which media browsing, input, and interaction can be performed (e.g., for creating drawings, viewing drawings, modifying and / or interacting with font-based text and / or handwritten text, navigating content such as web-based content, and / or interacting with media content). In some embodiments, the application is an application installed on device 500.

[0208] exist Figure 6A , user interface 609 includes elements for media browsing and interaction. In some embodiments, device 500 communicates with an input device such as stylus 600. In some embodiments, device 500 is configured to receive an indication of contact between stylus 600 and a surface such as touch screen 504. In some embodiments, device 500 and / or stylus 600 is further configured to send and / or receive an indication of proximity between a surface (e.g., touch screen 504) and stylus 600. For example, icon 603 includes hover distance threshold 601. Although threshold 601 is illustrated as a line extending parallel to touch screen 504, it should be understood that such illustration is merely exemplary and is not limiting in any way. In some embodiments, a "hover event" as referred to herein includes a situation where a corresponding portion of an input device (e.g., the tip of the stylus 600) moves to a position less than a threshold distance (e.g., a threshold 601, such as 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) from a surface (e.g., touch screen 504) while not contacting the surface. In some embodiments, determining that the positioning of the corresponding portion of the input device relative to the projection of the surface (e.g., the vertical projection of the tip of the stylus 600) corresponds to the positioning of a user interface element (e.g., a selectable option, text, and / or graphical object) is referred to herein as the position of the input device corresponding to the user interface element (e.g., the tip of the stylus corresponds to the object). In addition, as referred to herein, displaying or modifying one or more portions of a user interface corresponding to a user interface object in response to a hover event optionally describes a hover event between the input device and the surface at a position in the user interface corresponding to the user interface object.

[0209] like Figure 6AAs shown, user interface 609 includes multiple interactive and non-interactive visual elements. For example, text 602 includes editable font-based text in a text input area (e.g., a search box). In response to a hover event including stylus 600 corresponding to the text input area, a text input cursor preview is displayed, as will be described later. In some embodiments, icon 604 can be selected to initiate one or more operations, such as a search query based on text 602, and is visually emphasized in response to the hover event, as will be described later. In some embodiments, media player 608 can be interactive to control the playback of corresponding media and is visually modified in response to the hover event, as will be described later. In some embodiments, link 610 can be selected to initiate the execution of one or more operations, such as the display of the linked web page content, and is visually emphasized in response to the hover event, as will be described later. In some embodiments, text 612 is non-editable text (e.g., text that is part of an image that includes an image of a football and an image of text 612), and a selection cursor is displayed in response to the hover event, as will be described later. In some embodiments, the corresponding element within the selectable options 614 can be selected to view the corresponding linked content and displayed along with additional selectable options to navigate the corresponding element in response to a hover event, as will be described later. The user interface 609 represents a view of the electronic device 500 from a top position (e.g., perpendicular to a plane coplanar with the touch screen 504), while the icon 603 represents a view of the electronic device 500 from a corresponding side of the electronic device 500 (e.g., parallel or nearly parallel to a plane coplanar with the touch screen 504). It should be understood that such representations are merely exemplary for illustrative purposes to indicate hover events and interactions as described herein and are not limiting in any way.

[0210] exist Figure 6B , stylus 600 moves to a position above touch screen 504, but beyond threshold 601, as seen in icon 603. As referred to herein, a corresponding position of stylus 600 exceeding threshold 601 is described as being outside the hover threshold, as referred to herein. Because stylus 600 exceeds threshold 601, device 500 does not modify user interface 609 in response to such placement of the stylus on touch screen 504.

[0211] exist Figure 6C, the stylus 600 moves to a position within the hover threshold 601 but not contacting the touch screen 504, i.e., a position corresponding to a search box including text 602. In response to the hover event, a text insertion preview cursor 690 is displayed. In some embodiments, while the stylus 600 remains within the hover threshold 601 and corresponds to a position within the text input area including text 602, the text insertion preview cursor 690 moves in the user interface based on the movement of the stylus 600. The text insertion preview cursor 690 optionally indicates a position in the user interface 609 where the text insertion cursor will be placed and / or positioned in response to the device 500 detecting that the stylus has touched down and contacted the touch screen 504. In Figure 6C , a text insertion preview cursor 690 is displayed at the end of the text 602 in the search box.

[0212] exist Figure 6D In the example, as shown by icon 603, the stylus 600 is Figure 6C The position of the text insertion preview cursor 690 shown contacts the touch screen 504. In response to this contact, the display of the text insertion preview cursor 690 stops, and the text insertion cursor 692 is displayed at the end of the text 602, as shown in FIG. Figure 6D In some embodiments, the text insertion preview cursor 690 and the text insertion cursor 692 are displayed with different visual appearances (e.g., different scales, colors, opacities, shadows, borders, and / or lighting effects) to distinguish the insertion of the preview and text insertion cursors. In some embodiments, after inserting the text insertion cursor 692, the stylus 600 is removed from the hover threshold (e.g., to a position exceeding the threshold 601). In response to the stylus 600 moving beyond the threshold 601, the display of the text insertion cursor 692 is maintained, as shown. Figure 6E In addition, Figure 6E , device 500 has detected text input (e.g., from an external keyboard, from a soft keyboard, and / or from voice input) and, in response, displays new text corresponding to the text input (e.g., ABC) at the location of text insertion cursor 692. In some embodiments, while text insertion cursor 692 is displayed, in response to stylus 600 moving into hover threshold 601, a text insertion preview cursor 690 is simultaneously displayed at the location of the stylus in user interface 609.

[0213] exist Figures 6F to 6H , the corresponding object moves into a threshold distance 601 of the touch screen 504 and then contacts the touch screen 504 . Figure 6FIllustrated is a hand and / or finger 605 positioned outside of a hover threshold 601. The hand and / or finger 605 is positioned at the location of a search box that includes text 602, but more generally, the device 500 does not modify the display of the text 602 or the user interface 609 because the hand and / or finger 605 is outside of the threshold distance 601 and / or because the hand and / or finger 605 is not a stylus 600. If the device 500 detects movement of the hand and / or finger 605 toward the touch screen 504 within the hover threshold 601, the device 500 optionally determines that the hand 605 is not an input device (e.g., not a stylus 600) and forgoes displaying the text insertion preview cursor 690 (and / or other modifications to the user interface 609). However, in Figure 6H , the device 500 detects the contact of the corresponding part of the hand 605 (e.g., a finger) after contacting the text 602 within the search box, and in response, the device 500 inserts the text insertion cursor 692 at the end of the text 602, as shown in FIG. Figure 6H In some embodiments, the text insertion cursor 692 is displayed at a first location in the search box (e.g., at the end of the text 602), and in response to detecting contact of the hand and / or finger 605 at a second location of the text 602 (e.g., in the middle of the text 602), display of the text insertion cursor 692 at the first portion stops, and the text insertion cursor 692 is displayed at the location of the contact corresponding to the second portion of the text 602.

[0214] exist Figure 6I , stylus 600 is positioned outside of hover threshold 601, as indicated by icon 603, corresponding to positioning of a content (e.g., text) input area including text 602, and device 500 does not modify display of user interface 609. In response to stylus 600 being held at a position corresponding to the content input area, as shown in FIG. Figure 6J As shown entering the hover threshold 601, the selectable option 621 is displayed with visual emphasis 618. In some embodiments, the visual emphasis 618 is displayed with a first visual appearance. Figure 6J , the stylus 600 is located at a position corresponding to a content input area including text 602, but optionally is not located at a position corresponding to a selectable option 621. In some embodiments, display of the selectable option 621 ceases in response to the stylus 600 moving beyond the hover threshold 601 (e.g., away from the touch screen 504) and / or in response to the stylus moving to a position outside the content input area of ​​the user interface 609 (even if remaining within the hover threshold 601). While the selectable option 621 is displayed, selection of the selectable option 621 optionally initiates performance of one or more operations associated with the content input area. For example, as Figure 6KAs shown, in response to selection of selectable option 621 (e.g., contact of stylus 600 with touch screen 504 at a location corresponding to selectable option 621), display of text 602 ceases. Additionally, in response to selection of the selectable option, visual emphasis 618 is displayed in a second visual appearance that is different from the first visual appearance (e.g., a different scale, color, opacity, shading, border, and / or lighting effect). It should be understood that Figures 6I to 6K The embodiments illustrated in are exemplary only, and in some embodiments, other selectable options are displayed in response to a hover event between the input device and the surface corresponding to the positioning of the user interface element.

[0215] exist Figures 6L to 6P , depicts the positioning of stylus 600 within or outside of hover threshold 601 and / or optionally corresponding to selectable options, and various responses of device 500. For example, in Figure 6L , stylus 600 is outside the hover threshold, represented by threshold 601, in glyph 603, at a position corresponding to the position of search icon 604. In some embodiments, even though the positioning of stylus 600 corresponds to search icon 604, device 500 does not display additional visual emphasis or elements while stylus 600 is outside the hover threshold.

[0216] exist Figure 6M , the stylus 600 moves to a position within the hover threshold 601, but does not correspond to the position of the search icon 604. For example, when the tip of the stylus 600 is outside the threshold distance (e.g., 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) of the search icon 604 and when the stylus 600 is within the threshold distance 601 of the touch screen 504, the device 500 forgoes displaying additional visual emphasis or visual elements associated with the search icon 604. In some embodiments, such as Figure 6N As shown, in response to movement of the stylus 600 within a threshold distance of the search icon 604, the device 500 displays a visual emphasis or change (e.g., a modified scale, color, opacity, shadow, border, and / or lighting effect) of the icon 604. For example, the visual emphasis 618 includes an area surrounding the search icon 604, optionally visually emphasized with a solid color and / or a semi-transparent color surrounding the search icon 604. In some embodiments, while the visual emphasis 618 is displayed and hovering over the search icon 604, movement of the stylus 600 results in or does not result in a modification to the visual emphasis 618 and / or the visual appearance of the icon 604. For example, the stylus 600 may be moved from Figure 6N The upper left corner of the search icon 604 is shown moved downward to Figure 6OThe lower left corner of the search icon 604 shown does not cause the device 500 to modify the visual emphasis 618 and / or the search icon 604 (e.g., the device 500 does not display the visual emphasis 618 and / or the icon 604 with a parallax effect and / or lighting effect that changes as the positioning of the stylus 600 on the icon 604 changes).

[0217] like Figure 6P As shown, in some embodiments, in response to selection of icon 604, such as contact of stylus 600 with touch screen 504 at a location on touch screen 504 corresponding to icon 604, one or more operations associated with search icon 604 are initiated, and visual emphasis 618 is modified. For example, the translucency of visual emphasis 618 is optionally reduced or increased, and / or the color of visual emphasis 618 is optionally modified. In some embodiments, the modified visual emphasis is maintained while contact of stylus 600 with touch screen 504 is maintained. In some embodiments, in response to ceasing selection of search icon 604, visual emphasis 618 is modified (e.g., to correspond to a location relative to the icon 604). Figure 6N (the visual appearance described in the hover event).

[0218] exist Figure 6Q , after selection of the search icon 604 is terminated (e.g., corresponding to the stylus 600 breaking contact / lifting from the touch screen 504), the stylus 600 remains within the hover threshold 601, but moves outside the threshold distance of the search icon 604 (e.g., the positioning of the stylus 600 does not correspond to the search icon 604). In some embodiments, in response, the device 500 stops the display of the visual emphasis of the icon 604. It should be understood that the user interface 609 optionally includes additional or alternative selectable options, and the interactions described with respect to the search icon 604 (e.g., hovering, not hovering, moving the stylus 600 to a position corresponding to the search icon 604 and / or optionally selecting the search icon) are optionally the same or similar for such additional or alternative selectable options.

[0219] Figures 6R to 6W The interaction between the electronic device 500 and a cursor input device (e.g., a mouse or trackpad) is illustrated. In some embodiments, based on determining that the position of a cursor controlled by such a cursor input device corresponds to a graphical object, the device 500 displays visual emphasis associated with the graphical object. In some embodiments, based on determining that the position of the cursor does not correspond to a graphical object, the visual emphasis of the graphical object is discarded.

[0220] For example, in Figure 6R, the touchpad 607 communicates with the electronic device 500 (e.g., wirelessly or via a wired connection). In some embodiments, in response to detecting contact 619 on the touchpad 607, the device 500 displays a cursor 613 in the user interface 609. In some embodiments, in response to the device 500 detecting communication between the touchpad 607 and the device 500, the cursor 613 is displayed in the user interface 609. In some embodiments, the positioning of the cursor 613 in the user interface 609 is modified in response to movement of the contact 619 on the touchpad 607 (e.g., movement to the right). For example, in Figure 6S In response to detecting that contact 619 moves to the right on trackpad 607, device 500 causes cursor 613 to move to the right in user interface 609. Figure 6T , further movement of contact 619 to the right is detected, and based on determining that the new position of cursor 613 corresponds to the position of search icon 604, a visual emphasis 618 of icon 604 is displayed (e.g., cursor 613 ceases to be displayed and is represented in user interface 609 by visual emphasis 618). In some embodiments, visual emphasis 618 includes one or more lighting effects 618A. In some embodiments, lighting effect 618A includes a specular highlight (e.g., a simulated lighting effect that simulates movement of an object relative to one or more simulated or real light sources, such as reflection and / or refraction) to provide the user with a sense of how further input (e.g., movement of contact 619) corresponds to the current state of interaction with search icon 604. For example, the display gloss of the specular highlight around a portion of visual emphasis 618 (e.g., a border of the visual emphasis) is optionally modified in response to user input (e.g., an indication of movement of contact 619 received from trackpad 607).

[0221] exist Figure 6U , user input is received to move the cursor (e.g., the device 500 detects an upward and leftward movement of contact 619 on the trackpad 607) while the cursor's position corresponds to the search icon 604 and the visual emphasis 618 is displayed. In some embodiments, based on determining that the modified position of the cursor continues to correspond to the search icon 604 (e.g., with a threshold distance from the search icon 604, such as 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm), the lighting effect 618A is optionally modified and / or a parallax effect is optionally applied to the visual emphasis 618 based on the movement of the contact 619. For example, as Figure 6UAs shown, contact 619 moves toward the upper left of trackpad 607. In response to the movement of contact 619, and based on determining that the modified position of the cursor corresponds to search icon 604, a portion of specular highlight 618A (e.g., the upper left portion) is modified, and / or the shape of the visual emphasis and / or icon 604 is modified, thereby creating a parallax effect between the search icon 604 and the visual emphasis 618 based on the movement of contact 619. In some embodiments, the modification of the specular highlight includes modification of a point of a simulated light source that points to the corresponding graphical representation.

[0222] exist Figure 6V , device 500 detects that contact 619 has moved downward, although optionally less than the amount required for the cursor to move more than a threshold amount from the area corresponding to search icon 604, and in response, modifies the parallax effect described above. Figure 6U As described above, the specular highlight 618A is optionally modified accordingly. Such modification optionally includes reducing the brightness or otherwise modifying the lighting effect at locations of the visual emphasis 618 that do not correspond to one or more portions of the motion, and / or increasing the brightness or otherwise modifying the lighting effect at one or more portions of the visual emphasis 618 that correspond to the motion. Figure 6W In the example, when the cursor position corresponds to the search icon 604, Figure 6W An input is shown receiving a selection of search icon 604 (e.g., a click or tap of contact 619 on trackpad 607). In response to receiving the selection, one or more operations as previously described with reference to icon 604 are optionally performed (e.g., performing a search operation).

[0223] Figures 6X to 6Y Contextual information displayed in response to determining that timing information associated with a hover event satisfies one or more criteria is illustrated.

[0224] For example, Figure 6X As shown, the position of the stylus 600 corresponds to the search icon 604, and the stylus 600 is within the threshold distance 601 of the touch screen 504. In some embodiments, when the position of the stylus 600 corresponds to the search icon 604, a timer 623 is started. Based on determining that the criterion requiring the stylus 600 to hover over the search icon 604 for a time exceeding the time threshold 625 is met, one or more operations are performed. For example, Figure 6YAs shown, in response to detecting that the stylus 600 has been hovering over the icon 604 for a time longer than the time threshold 612, the device 500 displays a tooltip 631. The tooltip 631 optionally includes contextual information associated with the corresponding visual element, such as the search icon 604. In some embodiments, the contextual information is the name or description of the related operation (e.g., "Search," indicating that selection of the icon 604 will result in initiating a search operation).

[0225] Figures 6Z to 6D D illustrates a hover event and interaction with media content, such as a media player. In some embodiments, user interface 609 includes media content, such as a media player. Although some embodiments are described with respect to video displayed in a media player, it should be understood that the media player may additionally or alternatively include audio content.

[0226] exist Figure 6Z , media player 608 is displayed in user interface 609, and stylus 600 is positioned corresponding to the media player. As shown in icon 603, stylus 600 is outside of hover threshold 601, so device 500 does not modify the display of user interface 609 in response to the presence of stylus 600. Figure 6AA , stylus 600 enters hover threshold 601, and in response to entering the hover threshold, device 500 displays one or more media controls 620 for media player 608. Media controls 620 optionally include one or more selectable options for modifying playback of corresponding media content (e.g., fast forward, rewind, pause, play, skip forward, skip backward, modify audio volume, and / or navigate a playback queue). Figure 6AB , the stylus 600 moves to a position corresponding to the fast forward icon 620A while remaining within the hover threshold 601 but not in contact with the touch screen 504. In some embodiments, based on determining that the stylus 600 corresponds to a media control such as the fast forward icon 620A, a visual emphasis associated with the corresponding media control is displayed, such as a modification to the scale, color, opacity, shadow, border, and / or lighting effects, such as Figure 6AB In some embodiments, selecting a corresponding selectable option will initiate the execution of one or more operations that modify the playback or characteristics of the media, such as a fast-forward operation with respect to the media. Figure 6AC , selection of the fast forward icon 620A is detected. In some embodiments, selection is determined in response to other gestures or indications (e.g., a double-click on the surface, a tap or gesture on the input device, and / or a hand gesture). Figure 6AD In response to detecting selection of the fast-forward icon 620A, the device 500 advances the playback position of the media (eg, navigates the media content according to the selection).

[0227] Figures 6AE to 6AF Illustrated is a hover interaction associated with linked content, such as a web page link. Figure 6AE , as illustrated relative to touch screen 504, stylus 600 is outside of hover threshold 601. Figure 6AE , the relative positioning of stylus 600 corresponds to content link 610, but device 500 does not modify the visual appearance of link 610 because stylus 600 is outside of hover threshold 601 of touch screen 504. In some embodiments, content link 610 can be selected to initiate one or more actions, such as display of a corresponding web page, launch of an application, and / or bookmarking based on the web page. In some embodiments, in response to a hover event of stylus 600 at a position corresponding to contextual link 610 over touch screen 504, device 500 visually emphasizes link 610, such as Figure 6AF For example, in response to the hover event, device 500 optionally modifies the visual appearance of background color, opacity, and / or font characteristics, including bolding and / or underlining link 610.

[0228] Figures 6AG to 6AI Examples of interactions with non-editable content (such as displayed text content or other graphic objects) according to examples of the present disclosure are illustrated. Figure 6AG In , non-editable text 612 is displayed in user interface 609. Such text optionally corresponds to text of a web page or text included in an image. Figure 6AG , the position of stylus 600 corresponds to non-editable text 612, but stylus 600 is outside threshold distance 601 of touch screen 504, so device 500 does not modify user interface 609. Figure 6AH , in response to a hover event between stylus 600 and touch screen 504, device 500 displays a text selection cursor preview 615 in a first visual appearance at a location in text 612 corresponding to stylus 600 (e.g., corresponding to the tip of stylus 600). The first visual appearance optionally indicates to the user the location at which subsequent selection of text 612 will begin (e.g., in response to bringing stylus 600 into contact with touch screen 504). In some embodiments, the first visual appearance includes a first opacity level and / or color. Figure 6AI In some embodiments, the device 500 detects a selection, such as contact between the stylus 600 and the touch screen 504. In response to the contact, the display of the text selection preview cursor 615 stops, and the text selection preview cursor 615 is displayed at the location of the contact on the touch screen 504 (e.g., at the Figure 6AI) initiates display of a text selection cursor 617. In some embodiments, the text selection cursor 617 is displayed with a second visual appearance to visually distinguish the selection cursor preview from the actual selection cursor. For example, the second visual appearance is optionally darker and / or more opaque than the first visual appearance. Figure 6AJ As shown, while maintaining contact between stylus 600 and touch screen 504, stylus 600 is moved in a rightward direction along touch screen 504. In response to the movement, the position of text selection cursor 617 is updated, and the content (e.g., text) between the position corresponding to the initial contact in the user interface and the ending position in the user interface is selected (e.g., highlighted with a degree of translucency), as shown by selection 644.

[0229] exist Figures 6AK to 6AO In , hover interaction with a selection of non-editable text is illustrated. Figure 6AK , as indicated by selection 644 including one or more selectable options (including grabber 646A), non-editable text 612 is selected (e.g., as indicated by reference to Figure 6AJ ). A selectable option including grabber 646A is optionally selected to modify the boundaries of selection 644 relative to text 612. In some embodiments, the boundaries of selection 644 are modified in response to selecting a corresponding grabber such as grabber 646A (e.g., bringing stylus 600 into contact with touch screen 504) and modifying the positioning of grabber 646A (e.g., dragging stylus 600 on touch screen 504). Figure 6AL In response to a hover event corresponding to gripper 646A, device 500 displays arrow 648A indicating the direction of manipulation of selectable option 646A. Figure 6AL , arrow 648A indicates that selectable option 646A can be manipulated in a left or right direction, which will cause selection 644 to contract or expand accordingly.

[0230] exist Figure 6AM In response to device 500 detecting selection of grabber 646A by stylus 600, device 500 initiates operations to manipulate grabber 646A and / or selection 644. For example, selection 644 is modified based on movement of the stylus along touch screen 504 while the stylus contacts grabber 646A. Figure 6AN , selection 644 expands to the right in response to a corresponding movement of stylus 600 to the right while maintaining selection of selectable option 646A.

[0231] In some embodiments, the device 500 displays different functional indications in response to detecting that the stylus is hovering over different selectable options. Figure 6AO, in response to a hover event corresponding to stylus 600 being over gripper 646B, device 500 displays arrow 648B, where arrow 648B indicates that gripper 646B can be manipulated vertically (rather than horizontally) to expand or contract selection 644 vertically. It should be understood that vertical manipulation of selection 644 is optionally coupled to manipulation of gripper 646B relative to selection 644. Figures 6AL to 6AM Similar or identical to the description.

[0232] exist Figure 6AP to Figure 6AT , a hover event directed to a selectable icon and / or a user interface object including a selectable icon, and the display of additional selectable options are illustrated.

[0233] exist Figure 6AP , an area 614 of the user interface 609 including a plurality of selectable options (e.g., “Breaking,” “Sports,” “USA,” and “World”) is displayed in the user interface 609, each of which can be selected to initiate one or more operations, such as display of a corresponding web page, launch of an application, and / or bookmarking based on the web page. Figure 6AQ , the stylus 600 moves within the hover threshold 601 of the touch screen 504, as shown by the icon 603, and is located at a position corresponding to the area 614 including the selectable options. In response, the device 500 displays a plurality of navigation arrows, including arrow 624. In some embodiments, arrow 624 can be selected to scroll (e.g., to the right) through the plurality of selectable options. For example, the device 500 optionally displays one or more selectable options that are not currently displayed, stops displaying one or more of the currently displayed selectable options, and / or modifies the positioning of the corresponding currently displayed selectable options in response to selecting arrow 624.

[0234] exist Figure 6AR , the device 500 detects a hover event that includes the stylus 600 moving while remaining within the hover threshold 601 (e.g., while not in contact with the touch screen 504) to correspond to the position of the navigation arrow 624. In response to the hover event, the navigation arrow 624 is displayed with visual emphasis (e.g., a modification to the scale, color opacity, shadow, border, and / or lighting effect) having a first appearance (e.g., a first scale, color opacity, shadow, border, and / or lighting effect), as shown. Figure 6AR As shown. Figure 6AS , device 500 detects selection of navigation arrow 624 (e.g., stylus 600 contacts touch screen 504 at a location corresponding to navigation arrow 624). It should be understood that in some embodiments, selection of navigation arrow 624 is optionally one or more alternative inputs (e.g., a hand gesture and / or a double-click between stylus 600 and the surface). Figure 6AT In response to Figure 6AS , device 500 scrolls right through the selectable options in region 614 to reveal the selectable option “Alliums” in region 614. Additionally, stylus 600 moves beyond hover threshold 601, and in response to such movement, device 500 stops displaying the navigation arrows, as shown in FIG. Figure 6AT In some embodiments, the display of the navigation arrow 624 remains despite the stylus 600 moving beyond the hover threshold 601. In some embodiments, after the stylus 600 is moved beyond the hover threshold 601, the display of the navigation arrow remains for a threshold amount of time (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds), and after the time threshold is exceeded, the device 500 stops displaying the navigation arrow.

[0235] exist Figure 6AU to Figure 6AY , user interface 609 corresponds to a web browser interface (eg, a Safari web browser), and illustrates hover events corresponding to multiple tabs of the web browser.

[0236] exist Figure 6AU In the example, the content corresponding to the first label 640A is not displayed in the user interface 609, and the content corresponding to the second label 640B is displayed (for example, the second label 640B is Figure 6AU In some embodiments, content corresponding to the corresponding tab is displayed (e.g., content from two or more tabs is displayed simultaneously), and in some embodiments, content corresponding to the corresponding tab is not displayed (e.g., a default landing page is displayed, such as a page including the user's bookmarked links). Figure 6AU As shown by the icon 603 in , the stylus 600 is outside the hover threshold 601. Figure 6AV , the electronic device 500 detects a hover event that includes the stylus 600 moving into the hover threshold 601 and moving to a location corresponding to the label 640A (e.g., an area of ​​the user interface 609 corresponding to the displayed portion of the label 640A). In response to the hover event, the device 500 displays the selectable option 682A for the label 640A. In some embodiments, in response to the hover event, the visual emphasis (e.g., scale, color, opacity, shadow, border, and / or lighting effects) of the area corresponding to the first label 640A is modified or displayed. In some embodiments, in response to the hover event corresponding to the selectable options 682A and / or 682B, the visual emphasis of the selectable options 682A and / or 682B is displayed, or the currently displayed visual emphasis is modified, as will be described later. In Figure 6AWIn FIG, the stylus 600 moves from a position corresponding to the first label 640A to a position corresponding to the area associated with the second label 640B. In response to determining that the position of the stylus 600 does not correspond to the first label 640A, the device 500 stops displaying the selectable option 682A, such as Figure 6AW In some embodiments, the visual emphasis corresponding to the first label 640A is additionally or alternatively stopped based on such a determination. Figure 6AW In response to determining that the stylus 600 is hovering over the touch screen 504 at a location corresponding to the second label 640B, the display of the second label 640B is modified. For example, visual emphasis of the second label 640B is displayed, and / or the device 500 initiates the display of the selectable option 682B; in some embodiments, the details of the displayed label 640B and / or the selectable option 682B are different from those of the display with respect to the second label 640B. Figure 6AU Same or similar as described above (e.g., visual emphasis and / or display of option 682A may be selected). Figure 6AX In some embodiments, the input that selects the selectable option 682B is a hand gesture and / or one or more gestures on the input device.

[0237] exist Figure 6AY In response to Figure 6AX In response to selecting the selectable option 682B in the display, the device 500 stops displaying the content corresponding to the second tab 640B. In addition, in response to stopping displaying the content corresponding to the second tab 640B, the content corresponding to the first tab 640A is started, such as Figure 6AY In some embodiments, based on determining that the user interface 609 includes a single tab, default content (eg, a login page) is displayed in response to a request to stop displaying content corresponding to the single tab.

[0238] exist Figures 6AZ to 6BF , a hover event corresponding to a request for interaction with a user interface object is shown.

[0239] exist Figure 6AZIn FIG, user interface 609 corresponds to a drawing user interface including a control palette 630 that includes selectable options for modifying the writing produced by stylus 600 in the drawing user interface. For example, selectable options 632A, 632B, and 632C can each be selected to modify the currently selected writing and / or drawing tool for stylus 600. Based on the positioning of stylus 600 relative to touch screen 504 and the currently selected writing and / or drawing tool, a virtual shadow 662 is displayed in user interface 609. With respect to method 900 and Figures 8A to 8C The behavior and appearance of virtual shadows are further described. The currently selected user interface object 642 is displayed in the content input area of ​​the user interface 609 including a grabber 650A, which is optionally selectable to modify the user interface object 642. In some embodiments, a corresponding "grabber" is a selectable option that can be selected to modify the corresponding virtual object. For example, detecting a selection of a corresponding grabber and detecting a modification of the corresponding grabber while maintaining the selection can optionally modify (e.g., scale, translate, and / or extend) a virtual object associated with the corresponding grabber in some manner (e.g., in a direction to scale, translate, and / or extend the virtual object) based on the modification.

[0240] exist Figure 6BA , stylus 600 moves to a position relative to touch screen 504 that corresponds to gripper 650A but is outside of hover threshold 601, as shown by icon 603. Because stylus 600 is outside of hover threshold 601, device 500 does not modify the display of object 642 and / or the various grippers of user interface object 642. Figure 6BB , stylus 600 moves within hover threshold 601 and to a position corresponding to gripper 650A, and in response, device 500 displays directional arrow 648A. In some embodiments, in response to hovering over gripper 650A, one or more visual indications associated with the direction of manipulating (e.g., scaling, deforming, and / or expanding) user interface object 642 are displayed. In some embodiments, the directional arrows indicate possible manipulation directions. For example, directional arrow 648B is oriented to extend horizontally toward the left and toward the right, indicating that manipulation toward the left and / or right of gripper 650 (e.g., expanding the visual object) is possible. In some embodiments, the directional arrows indicate one manipulation direction (e.g., only up, only down, only right, or only left).

[0241] exist Figure 6BC , when the positioning of stylus 600 corresponds to the positioning of gripper 650A, device 500 receives a selection indication including stylus 600 contacting touch screen 504. In response to the contact, device 500 initiates an operation associated with manipulating user interface object 642. For example, Figure 6BD In keeping relative to Figure 6BC When the contact described is made, a movement indication is received (e.g., stylus 600 slides to the left along touch screen 504). In response to the operation of manipulating user interface object 642, device 500 scales user interface object 642 according to the movement (e.g., scaling object 642 to the left). In some embodiments, such manipulation corresponds to scaling (e.g., stretching, enlarging, and / or shrinking) the user interface object by an amount that is correlated (e.g., positively correlated or negatively correlated) with the amount of movement of stylus 600.

[0242] exist Figure 6BE , the user interface object 642 is displayed, and directional arrows 648B corresponding to different manipulations of the user interface object 642 are also displayed. Specifically, in response to a hover event including the stylus 600 hovering over the touch screen 504 at a position corresponding to the position of the corresponding grabber 650B to which the navigation arrow 648B corresponds (e.g., the upper left grabber of the object 642), the device 500 optionally displays the navigation arrow 648B. In some embodiments, the navigation arrow 648B indicates that a selection and modification of the positioning of the grabber 650B is detected that can modify the user interface object 642 in two non-parallel directions (e.g., scaling, deforming, and / or expanding the object toward the upper left and / or lower right of the touch screen 504). Similarly, in Figure 6BF In some embodiments, in response to a hover event that includes movement of the stylus 600 to a positioning corresponding to the positioning of the corresponding gripper (e.g., the upper gripper of the object 642) above the touch screen 504, a direction arrow 648C corresponding to the manipulation of the user interface object in the vertical direction is displayed. In some embodiments, the navigation arrow 648C indicates that the selection and modification of the positioning of the gripper 650C is detected to modify the user interface object 642 vertically (e.g., to scale, deform, and / or expand the object toward the top and / or bottom of the touch screen 504). In some embodiments, in response to detecting a hover event of the stylus 600 over other corresponding grippers of the user interface object 642, the device 500 similarly displays visual indications corresponding to the manipulation directions of these grippers, such as relative to Figures 6AZ to 6BF As stated.

[0243] Figure 7A 7J is a flowchart illustrating a method 700 for displaying additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device. Method 700 is optionally performed on an electronic device, such as device 100, device 300, and device 500, as described above with reference to FIG. Figure 1A to Figure 1B 、 Figures 2 to 3 、 Figures 4A to 4B and 5A to 5ISome operations in method 700 are optionally combined, and / or the order of some operations is optionally changed.

[0244] As described below, method 700 provides a method for displaying additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device. This method reduces the cognitive burden on a user when interacting with the disclosed device user interface, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charges.

[0245] In some embodiments, method 700 is performed at an electronic device that communicates with a display generation component and one or more sensors (e.g., a touch-sensitive surface). For example, a mobile device (e.g., a tablet device, a smart phone, a media player, or a wearable device) or a computer that optionally communicates with one or more of a mouse (e.g., external), a trackpad (optionally integrated or external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), and / or a controller (e.g., external). In some embodiments, the display generation component is a display (optionally a touch-sensitive and / or touch screen display) integrated with the electronic device, an external display such as a monitor, a projector, a television, and / or a hardware component (optionally integrated or external) for projecting a user interface or making a user interface visible to one or more users.

[0246] In some embodiments, the electronic device displays (702a) via a display generation component a user interface including a first user interface object, such as Figure 6C The user interface 609 includes a search box. For example, the user interface is optionally a system user interface of the electronic device (eg, a home screen interface, such as Figure 4A exemplified), a user interface of a content creation application (e.g., a drawing user interface), a user interface of a note-taking application, a content browsing user interface, or a web browsing user interface. In some embodiments, the first user interface object is a selectable option in the user interface that can be selected to perform a corresponding function that is optionally associated with the user interface. For example, the first user interface object is optionally an application icon on a home screen user interface that can be selected to cause display of an application via a display generation component, a button in a drawing user interface that can be selected to display an option panel including options for changing characteristics of handwriting produced in the drawing user interface using a stylus, or a representation of media within a content browsing interface that can be selected to initiate playback of the corresponding media.

[0247] In some embodiments, while displaying a user interface including a first user interface object via a display generation component, the electronic device detects (702b) via one or more sensors a corresponding object that is close to but not touching a surface associated with the user interface, such as detecting Figure 6C Stylus 600 in, or Figure 6F and / or Figure 6G 605 in (e.g., a touch-sensitive surface, a physical surface onto which the user interface is projected, or a virtual surface corresponding to at least a portion of the user interface). For example, the corresponding object is optionally a finger of the hand of a user interacting with the surface. In some embodiments, the object is a stylus that communicates with the electronic device, or the object is optionally a stylus that does not communicate with the electronic device. In some embodiments, one or more signals sent between the corresponding object, the electronic device and / or the surface are used to determine the proximity between the corresponding object and the surface. For example, the corresponding object is optionally a stylus having one or more sensors that are configured to detect one or more signals sent from the surface. In some embodiments, the received signal strength of the one or more signals is used as a criterion for determining proximity. In some embodiments, the one or more signals include data encoding one or more relative distances between the stylus and the surface. In some embodiments, the corresponding object is determined to be near but not in contact with the surface when the corresponding object is greater than a first threshold distance (e.g., 0.0 cm, 0.01 cm, 0.03 cm, 0.05 cm, 0.1 cm, 0.3 cm, 0.5 cm, or 1 cm) from the surface and less than a second threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm) greater than the first threshold distance from the surface. Otherwise, the corresponding object is optionally determined to be not near the surface, or is determined to be in contact with the surface.

[0248] In some embodiments, in response to detecting a corresponding object that is proximate to but not contacting a surface (702c), the corresponding object is determined to be an input device (such as an input device) that communicates with the electronic device. Figure 6C stylus 600 in ) and the positioning of the input device corresponds to the first user interface object (such as Figure 6C The electronic device displays (702d) in the user interface a first selectable option that can be selected to perform a first operation associated with the first user interface object, such as displaying Figure 6C690 . For example, the input device is optionally a stylus device. In some embodiments, the input device is a wearable device (e.g., a glove or a thimble). Communication between the input device and the electronic device optionally includes one or more data streams sent and / or received by the input device and the electronic device. In some embodiments, determining that the positioning of the input device corresponds to a first user interface object (e.g., including a selection indicator and / or highlighting) optionally includes determining that the input device (and / or a vertical or other projection of the input device on the touch-sensitive surface) is located within a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm) of the corresponding positioning of the first user interface object displayed by the display generation component or the corresponding positioning corresponding to the first user interface object. As mentioned herein, determining that the positioning of the input device corresponds to the first user interface object and that the input device is located within a threshold distance of a corresponding portion of the first user interface object or a corresponding portion corresponding to the first user interface object is optionally a "hover event." Similarly, the state of the input device within a threshold distance is optionally referred to as "hovering" (e.g., above the surface). In some embodiments, the selectable option is not displayed until the input device is located within a threshold distance of the corresponding positioning of the first user interface object or the corresponding positioning corresponding to the first user interface object. For example, the input device, electronic device and / or touch-sensitive surface optionally determines that the magnitude of a vector extending from a part of the input device (e.g., the tip) toward the surface or from the surface toward the input device is less than a threshold magnitude. In some embodiments, the selectable option is a button that controls the media associated with the displayed media playback (e.g., a play, pause, rewind or fast forward button). In some embodiments, the selectable option can be selected to initiate a scrolling process of scrolling one or more displayed elements. For example, the selectable option optionally controls a scrolling operation operating on a list of text and / or icons. In some embodiments, the user interface includes a text input box, and the selectable option can be selected to delete or select one or more characters displayed in the text input box. In some embodiments, if the positioning of the input device does not correspond to the first user interface object, the display of the first selectable option is abandoned, even if the input device is close to but not in contact with the surface. In some embodiments, in response to detecting each hover event in a sequence of multiple hover events, the electronic device responds to the hovering input device in one or more of the various ways described herein. In some embodiments, while hovering, a subsequent input is detected (e.g., the input device contacts the surface at the location of the first selectable option) to perform a first operation.

[0249] In some embodiments, based on determining that the corresponding object proximate the surface is not an input device (such as an input device) that communicates with the electronic device, Figures 6F to 6GThe electronic device abandons (702e) displaying in the user interface a first selectable option that can be selected to perform a first operation associated with the first user interface object, such as a reference Figures 6F to 6G The text insertion cursor preview 690 is not displayed. For example, the object is optionally a user's finger approaching the surface. In some embodiments, the object is an input device (e.g., a stylus) that does not communicate with the electronic device. Displaying selectable options for performing additional operations when the input device is close to the user interface object reduces the number of inputs required to access these operations.

[0250] In some embodiments, in response to detecting a corresponding object approaching but not contacting a surface, the electronic device modifies (704) a visual characteristic of the first user interface object to indicate that the first user interface object is selectable, such as a 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm distance from the first user interface object based on determining that the corresponding object approaching the surface is an input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object (e.g., a perpendicular projection of the tip of the input device onto the surface is within a threshold distance of the first user interface object, such as 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm). Figure 6N , modifying the display of icon 604 in step 702. As previously described with respect to step 702, in response to determining that the input device initiates a hover event, one or more functions are optionally performed. One such embodiment includes modifying one or more visual characteristics of the first user interface object. For example, the modification includes displaying one or more visual indications to indicate the interactivity of the user interface object (e.g., a button). The visual indication optionally includes changing the color, shading, hue, saturation, lighting effects, displaying or modifying a border around the user interface object, initiating or modifying an animation of the first user interface object, changing the shadow associated with the first user interface object, scaling one or more portions of the user interface object, and / or modifying the perceived positioning (e.g., depth) of the first user interface object in the user interface. In some embodiments, the visual indication optionally includes highlighting the first user interface object. In some embodiments, different visual indications are displayed in response to hover events corresponding to corresponding portions of the first user interface object. For example, hovering over a portion of the first user interface object corresponding to the top border optionally results in the display of one or more arrows (e.g., extending upward and / or downward). Similarly, in response to hovering over a lateral border of the first user interface object, one or more arrows (e.g., extending left and / or right) are optionally displayed. Displaying one or more modifications to one or more visual characteristics of the first user interface object to indicate that the user interface object is interactive effectively communicates that interaction with the first user interface object is possible and reduces errors in interacting with the first user interface object.

[0251] In some embodiments, the first user interface object is associated with a selection of a first area of ​​the user interface and is not associated with a selection of a second area of ​​the user interface that is different from the first area of ​​the user interface, and the first user interface object is interactive to modify the area of ​​the user interface selected by the first user interface object (706a), such as Figure 6AK 644 in the selection. For example, the user interface is a font-based handwriting and / or drawing user interface, and a first user interface object (e.g., a selection indicator and / or highlight) indicates a selection of a first portion of content displayed in the user interface. It should be understood that the user interface optionally includes fonts, drawings, and / or handwriting content, either alone or in some combination, and optionally includes graphical content (e.g., images and / or hand-drawn shapes or curves). In some embodiments, the first user interface object corresponds to the highlighted portion of the user interface. For example, the first portion of the user corresponds to a selected portion of content including text and / or handwriting displayed in a visually distinguishable manner (e.g., a partially transparent background having a first color or fill pattern based on the positioning and / or size of the underlying selected content).

[0252] In some embodiments, in response to detecting a corresponding object approaching but not contacting a surface, based on determining that the corresponding object approaching the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to a first user interface object (e.g., as previously described with respect to step 704), the electronic device displays (706b) via a display generation component a visual indication associated with one or more directions of the modification to the area of ​​the user interface selected by the first user interface object, such as Figure 6AL and Figure 6AO648A or 648B in the display. One or more visual elements such as circles or orbs are optionally displayed at the location of the visually distinguished area (e.g., overlaid on a border) to convey the interactivity of the selected portion (e.g., before the input device hovers over the first user interface object), and in response to a hover event corresponding to the corresponding visual element, one or more arrows are optionally displayed indicating one or more directions of modification to the selection. For example, the selection is optionally a semi-rectangular highlight containing font-based text, and the hover event optionally causes left and / or right arrows to be displayed on the lateral edges to indicate that the highlight can be extended laterally. In some embodiments, interaction with the first user interface object modifies the selection to include a second portion of the content instead of the first portion. For example, contacting a surface with an input device and then moving the input device along the surface optionally causes the text selection to expand (e.g., the text selection expands vertically and / or laterally based on vertical and / or lateral movement of the input device when in contact with the surface). In some embodiments, the hover event depends on determining that the input device corresponds to the first user interface object (e.g., the vertical projection of the tip of the input device positioned on the surface is within a threshold distance of the first user interface object such as 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm). In some embodiments, one or more visual indications are displayed in response to the hover event associated with the corresponding portion of the first user interface object. For example, hovering over the portion of the first user interface object corresponding to the top border optionally results in the display of one or more vertical arrows (e.g., extending upward and / or downward). Similarly, in response to hovering over the lateral borders of the first user interface object, one or more horizontal arrows (e.g., extending left and / or right) are optionally displayed. Displaying visual indications in response to a hover event reduces erroneous input from the user and prevents the continuous display of such visual indications, thereby reducing the power consumption and computational load required for such operations.

[0253] In some embodiments, the first user interface object corresponds to a first portion of the user interface, such as Figure 6AW and the first selectable option is associated with stopping the first portion of the user interface (such as Figure 6AWIn some embodiments, the first selectable option is associated with the display of the first portion of the content (e.g., a web page, or other content "page") in the user interface of the application. For example, the first user interface object is a label corresponding to a first portion of content in the user interface of the application (e.g., a web page, or other content "page"). For example, in response to detecting a selection of the first user interface object, the electronic device displays the content corresponding to the first user interface object in the user interface of the application without displaying the content corresponding to the second user interface object, and in response to detecting a selection of the second user interface object, the electronic device displays the content corresponding to the second user interface object in the user interface of the application without displaying the content corresponding to the first user interface object. In some embodiments, the first selectable option is a text and / or graphic object that can be selected to stop display of the first portion of the content, and optionally is not displayed until a hover event directed to the first user interface object is detected.

[0254] In some embodiments, upon displaying a first selectable option based on determining that a corresponding object proximate to the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to a first user interface object (e.g., as previously described with respect to step 704), the electronic device receives (708b) via one or more sensors a signal related to a selection of the first selectable option (such as a Figure 6AX One or more inputs corresponding to the selection of selectable option 682B).

[0255] In some embodiments, in response to receiving one or more inputs corresponding to selection of the first selectable option, the electronic device stops (708c) display of the first portion of the user interface (and optionally the first user interface object), such as stopping display of the portion of the web browser user interface corresponding to tab 640B, as shown in FIG. Figure 6AYAs shown. For example, in response to detecting a hover event over any portion of a first user interface object, the first user interface object is visually emphasized (e.g., with a change in color, boldness, and / or scale), and / or a first selectable option is displayed within the first user interface object or in association with the first user interface object. While hovering over a surface, in response to receiving a selection of the first selectable option (e.g., contacting the surface with the tip of an input device at the location of the first selectable option), the electronic device optionally stops displaying the first portion of the content. For example, hovering over an "X" included in a label corresponding to a web page displayed in a user interface optionally causes the electronic device to visually distinguish the "X," and a subsequent selection of the "X" (e.g., contacting the surface with an input device) causes the electronic device to stop displaying the web page. In some embodiments, the electronic device also stops displaying the first selectable option (e.g., "X") (e.g., stops displaying the label including the "X"). Displaying the first selectable option in response to hovering prevents the continued display of the first selectable option and avoids cluttering the user interface, thereby reducing the power and computational load required for such display.

[0256] In some embodiments, the first user interface object includes a content input area that includes content, such as Figure 6J A search box including text 602 is included in the content input area, and a first selectable option is associated with stopping display of content in the content input area (710a), such as Figure 6J 621 in the selectable options. In some embodiments, the content input area is a text-based area (e.g., a text box) or other content area that includes handwriting and / or characters (e.g., handwritten and / or font-based text and / or graphical objects). The first selectable option is optionally selectable to stop display of all or a subset of the content included in the content input area. For example, the first selectable option is a button or icon for stopping display of text in the text input area (e.g., deleting or clearing the text).

[0257] In some embodiments, upon displaying a first selectable option based on determining that a corresponding object proximate to the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to a first user interface object (e.g., as previously described with respect to step 704), the electronic device receives (710b) via one or more sensors a signal related to a selection of the first selectable option (such as a Figure 6K One or more inputs corresponding to the selection of selectable option 621).

[0258] In some embodiments, in response to receiving one or more inputs corresponding to selection of the first selectable option, the electronic device stops (710c) display of the content in the content input area, such as by Figure 6K 602 is shown in the figure. For example, in response to a hover event corresponding to a portion of the text input area, a first selectable option is displayed. In response to receiving a selection of the first selectable option (e.g., contacting the surface with the tip of the input device at the location of the first selectable option), the display of the entire or a subset of the text input area is optionally stopped. It should be understood that although the embodiments described herein are directed to text input areas, such functionality is optionally applied to other content input areas (e.g., stopping the display of handwriting, including font-based and handwriting content, and / or graphical objects). Displaying the first selectable option in response to hovering prevents the continuous display of the first selectable option and avoids cluttering the user interface, thereby reducing the power and computing load required for such display.

[0259] In some embodiments, the first user interface object is associated with presenting media content, such as Figure 6Z , and the first selectable option is associated with modifying playback of the media content (712a), such as Figure 6AB In the selectable option 620A of embodiment.For example, the first user interface object comprises or is the media player for video and / or audio content.The first selectable option is optionally associated with the navigation of such media content, for example, searches forward or backward or refreshes to traverse media content.In some embodiments, select the first selectable option to start, stop or resume the playback of media content.In some embodiments, can improve or reduce playback speed, or carry out the navigation between another content item and the current content item (for example, last or next in the media content queue).All such operations are all to envision with respect to media playback and navigation.

[0260] In some embodiments, upon displaying a first selectable option based on determining that a corresponding object proximate to the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to a first user interface object (e.g., as previously described with respect to step 704), the electronic device receives (712b) via one or more sensors a signal related to a selection of the first selectable option (such as a Figure 6AC One or more inputs corresponding to the selection of selectable option 620B).

[0261] In some embodiments, in response to receiving one or more inputs corresponding to selection of the first selectable option, the electronic device modifies (712c) playback of the media content, such as Figure 6AD, as shown in media player 608 in . In some embodiments, upon hovering over a location corresponding to the first user interface object, one or more of the selectable options described relative to the first selectable option are displayed. In response to receiving a selection of the first selectable option (e.g., contacting a surface with a tip of an input device at the location of the first selectable option) while the corresponding selectable option is displayed (e.g., upon hovering), performance of the associated operation is optionally initiated. Displaying the first selectable option in response to hovering prevents continuous display of the first selectable option, thereby reducing the power and computational load required for such display.

[0262] In some embodiments, in response to detecting a corresponding object that is proximate to but not contacting a surface, based on determining that the corresponding object that is proximate to the surface is an input device that communicates with the electronic device and that the positioning of the input device corresponds to a first user interface object (714a) (e.g., as previously described with respect to step 704), based on determining that the positioning of the input device satisfies one or more criteria, including a criterion that is satisfied when the positioning of the input device corresponds to the first user interface object for longer than a threshold amount of time, such as relative to Figure 6Y As shown in FIG. 6 , the electronic device displays ( 714 b ) information associated with the first user interface object via the display generation component, such as Figure 6Y 6 is a tooltip 631 shown for icon 604. For example, in response to a hover event involving an input device and a surface, information describing a function corresponding to a first user interface object (e.g., information indicating what function will be performed if the first user interface object and / or a first selectable option is selected) is displayed in response to the input device being hovered over a location corresponding to the user interface object on the surface for longer than a threshold amount of time (e.g., 0.05s, 0.1s, 0.25s, 0.5s, 0.75s, 1s, 2.5s, or 5s). The information optionally includes the name of the function associated with the user interface object and / or the selectable option, and optionally describes one or more inputs required to initiate the function. In some embodiments, in response to detecting the hover event, the first user interface object is visually emphasized (e.g., with color, boldness, and / or scale).

[0263] Displaying information associated with the first user interface object in response to hovering prevents the user from mistakenly initiating functions or unnecessarily reviewing documents to understand functions bound to the first user interface object, thereby reducing the computational load and power consumption required for such operations.

[0264] In some embodiments, the first user interface object includes a content (eg, text) input area (716a), such as Figure 6CIn some embodiments, in response to detecting the corresponding object approaching but not contacting the surface, based on determining that the corresponding object approaching the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to the first user interface object (e.g., as previously described with respect to step 704), the electronic device displays (716b) via the display generation component a visual indication of a content (e.g., text) insertion cursor at a position corresponding to the input device in the content input area, such as Figure 6C , an insertion cursor preview 690 is displayed. For example, hovering an input device over a portion of the user interface corresponding to a user interface object causes a shadow corresponding to a text insertion cursor to be displayed. The shadow of the text insertion cursor is optionally displayed with a first visual appearance (e.g., color, saturation, hue, opacity, and / or with a first animation) to convey a suggested positioning of the text insertion cursor to the user. In some examples, the suggested positioning of the text insertion cursor corresponds to a portion of the input device (e.g., the tip). For example, the suggested positioning corresponds to a position where the tip of the input device is projected (e.g., vertically) onto a surface, and the projected surface position optionally corresponds to the suggested positioning in the content input area. In some embodiments, a new content input (e.g., text input) will not cause the corresponding content to be displayed at the location indicated by the text insertion cursor until the text insertion cursor is placed at the location indicated by the text insertion cursor. In some embodiments, when the indication of the text insertion cursor is displayed at the suggested positioning, the text insertion cursor is displayed at a different location in the content input area (e.g., a new content input (e.g., text input) will cause the corresponding content to be displayed at the location of the text insertion cursor) and / or is not displayed in the content input area.

[0265] In some embodiments, while displaying the visual indication, the electronic device receives (716c) via one or more sensors one or more inputs corresponding to a selection of the visual indication (and / or a selection of an area within a threshold distance of the visual indication, such as 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm). Figure 6D In some embodiments, in response to receiving one or more inputs corresponding to selections of visual indications, the electronic device displays (716d) a text insertion cursor in the text input area at a position corresponding to the input device via the display generation component, such as Figure 6D, display of a text insertion cursor 692 in . While hovering over the surface, in response to receiving a selection of a first selectable option (e.g., contacting the surface with the tip of the input device at the location of the first selectable option), the text insertion cursor is optionally inserted at the suggested location and / or moved to the suggested location. The insertion and / or movement optionally includes displaying a text insertion cursor having a second visual appearance (e.g., corresponding to the first visual appearance having different corresponding visual characteristics, such as a lower opacity and / or a darker color) at the suggested location. In some embodiments, in response to selection of the visual indication, the shadow of the text insertion cursor is no longer displayed. In some embodiments, new content input (e.g., text input) will cause the corresponding content to be displayed at the location of the text insertion cursor. Displaying a visual indication corresponding to the text insertion cursor prevents text edits from being entered at undesirable locations within the content (e.g., text) input area, thereby reducing the computational load and power consumption required to process and display erroneous operations.

[0266] In some embodiments, the first user interface object includes content (718a), such as Figure 6AH 612 in. For example, the content is text displayed in a content (e.g., text) display area. In some embodiments, in response to detecting a corresponding object approaching but not contacting a surface, based on determining that the corresponding object approaching the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to the content (718b) (e.g., as previously described with respect to step 704), based on determining that the content is non-editable content (e.g., the text is part of an image and is not text that is editable (e.g., cannot be deleted or changed) in response to input such as from a virtual keyboard), the electronic device displays (718c) a visual indication of a content (e.g., text) selection cursor at a position in the content corresponding to the input device, e.g., via a display generation component. Figure 6AH 6. Display of a selection cursor preview 615 in the content at the location of the tip of the stylus 600. In some examples, the positioning of the content selection cursor corresponds to a portion of the input device (e.g., the tip). For example, the positioning corresponds to the positioning of the tip of the input device projected (e.g., vertically) onto a surface, and the projected surface positioning optionally corresponds to the positioning in the content display area. Even if the text in the content is optionally not editable, the text in the content is optionally selectable (e.g., via highlighting) for subsequent operations (e.g., copy, paste, and / or cut). Displaying the content selection cursor in response to hovering prevents the content selection cursor from being continuously displayed, thereby reducing the power and computational load required for such display.

[0267] In some embodiments, in response to detecting a corresponding object approaching but not contacting a surface, based on determining that the corresponding object approaching the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to the content (720a) (e.g., as previously described with respect to step 704), based on determining that the content is editable content (e.g., text that can be edited in response to input such as input from a virtual keyboard), such as text 602, the electronic device abandons (720b) ​​displaying, via a display generating component, a visual indication of a content (e.g., text) selection cursor at a location in the content corresponding to the input device, such as Figure 6AH In some embodiments, the stylus 600 is hovering over text 602 rather than text 612. For example, the editable content is font-based or handwriting-based text and is displayed in an area that includes a content input area. In some embodiments, the content selection cursor is displayed when the positioning of the content selection cursor corresponds to non-editable text in the content, and the display of the content selection cursor stops according to the input device moving to a position corresponding to the editable text (e.g., hovering over the surface). In some embodiments, a second cursor different from the content selection cursor, such as a text insertion cursor and / or a shadow of the text insertion cursor (e.g., as described with respect to step 716), is displayed in response to the input device hovering over the editable content. Abandoning the display of the content selection cursor in response to hovering over editable text indicates that the text is editable, thereby reducing errors in the interaction between the input device and the text.

[0268] In some embodiments, before a corresponding object is detected approaching but not contacting the surface, the first user interface object is displayed (722a) with a first amount of separation from the backplane (e.g., the backplane and / or background of the user interface), such as Figure 6M 604 is separated by a first amount from a backplane (e.g., a background of user interface 609) above which the rest of the content of user interface 609 is being displayed. In some embodiments, in response to detecting a corresponding object approaching but not contacting the surface, the electronic device displays (722b) the first user interface object at a second amount of separation from the backplane that is greater than the first amount of separation, such as based on determining that the corresponding object approaching the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to the first user interface object (e.g., as previously described with respect to step 704). Figure 6NThe middle icon 604 is separated from the backplane by a second amount. For example, the backplane of the user interface shares the same plane as the plane of the user interface displayed in a two-dimensional (or near-two-dimensional) environment (e.g., on a display device such as a computer monitor or smart phone). In some embodiments, displaying the first user interface object with a first separation amount from the backplane includes a non-zero or zero separation amount from the plane of the user interface. Displaying non-zero separation optionally includes displaying a shadow, a border (e.g., the width in different parts of the border), a scale, an applied lighting effect, and / or other visual characteristics of the first user interface object to convey a sense of separation and / or depth difference of the first user interface object relative to the plane of the user interface. In some examples, the user interface is a three-dimensional environment, and the first user interface object is separated from a flat or curved plane by a first separation amount (e.g., 0 cm, 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm). For example, the first user interface object is displayed in a mixed reality environment and occupies a space or position having a spacing that is perceived as corresponding to the physical distance between a real-world object and a real-world plane (e.g., a flat or curved plane). In some embodiments, in response to detecting that an input device is hovering over a first user interface object and / or over a location of a surface determined to correspond to the user interface object, the first user interface object is displayed at a greater or lesser separation (e.g., 0 cm, 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm) from a plane (e.g., a flat or curved plane). If the corresponding object is not an input device and / or is not at a location corresponding to the first user interface object, displaying the first user interface object at the second separation from the backplate is optionally abandoned (e.g., and optionally maintaining display of the first user interface object at the first separation from the backplate). Displaying the first user interface object at a variable separation indicates to the user the interactivity of the first user interface object, thereby preventing erroneous input directed to user interface areas other than the user interface object, thereby reducing the computational load and power consumption required to process the erroneous input.

[0269] In some embodiments, when a user interface including a first user interface object is displayed via a display generation component, where the first user interface object has a first visual appearance in which a first visual characteristic (e.g., color, shading, fill, border, animation, shadow, and / or lighting effect) has a first value, such as Figure 6S The electronic device detects (724a) via a cursor control input device a first input corresponding to a cursor moving from a position away from the first user interface object to the first user interface object, such as from Figures 6S to 6Tfor moving the cursor 613 to the icon 604. For example, the first user interface object is a selectable representation, such as a button or icon, that can be selected to initiate a function of the device. Such a function is optionally launching an application, displaying a menu for entering content into the content input area, adding an image or font-based text to the content input area, or displaying a menu for changing a mark made to the simulated handwriting in the content input area. In some embodiments, the first visual appearance includes displaying the first user interface object with a default color, shading, fill, border, animation, shadow, and / or lighting effect. In some examples, the cursor control device is a computer mouse, a touchpad, a stylus, a hand, or a finger wearing a peripheral device, and / or a gaze / gesture detection unit configured such that movement or interaction with the peripheral device changes the positioning of a cursor optionally displayed in the user interface. The displayed cursor is optionally positioned at a location in two-dimensional or three-dimensional space in the user interface, and the cursor is optionally moved via subsequent movement or indication of a cursor control device to correspond to the location at which the first user interface object is displayed in the user interface (e.g., overlying the location, within a threshold distance of the location (e.g., 0 cm, 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm), and / or occupying the location).

[0270] In some embodiments, in response to detecting the first input, the electronic device moves (724b) the cursor to the first user interface object, such as from Figures 6S to 6T As shown, and the first user interface object is displayed in a second visual appearance in which the first visual characteristic has a second value different from the first value, such as Figure 6T . In some embodiments, in response to moving the cursor to correspond to the first user interface object, the first user interface object is optionally displayed with a second visual appearance (e.g., color, shading, fill, border, animation, shadow and / or lighting effect). For example, before moving the cursor to a button displayed in the user interface, the button is initially displayed as including a first set of graphics and / or fonts having various colors, with an initial amount of shadow on an initial background (e.g., a transparent, translucent or solid background). In response to moving the cursor to correspond to the button, the background of the button is optionally displayed with a second set of visual values, including at least one or more different visual values ​​for the same first visual characteristic (e.g., shadow, lighting effect, background and / or line width). For example, borders and fills having different opacities but the same color are optionally displayed in the gap space around and / or within the first user interface object.

[0271] In some embodiments, when the positioning of the input device corresponds to the first user interface object (e.g., as previously described with respect to step 704), such as Figures 6N to 6O In some embodiments, if the input device (e.g., optionally a device not contextually or generally associated with the cursor) is positioned corresponding to the first user interface object, the first user interface object is optionally displayed with a third visual appearance that includes a third set of visual values ​​(e.g., including the same set or subset of visual values ​​relative to the first and second visual values, but optionally including at least one or more different visual values). For example, the visual value includes an opacity that is different from the second opacity value and the first opacity value, but has the same background color. In some examples, the third visual value is a subset or superset of the second visual value, and vice versa. Displaying user interface objects with consistent visual changes across interactions with different input devices reduces erroneous interactions with electronic devices, thereby reducing the computational load and power consumption required to process these operations.

[0272] In some embodiments, in response to detecting the first input, the electronic device displays (726) the first user interface object with a parallax effect based on movement of the cursor while the cursor is located over the first user interface object, such as Figures 6U to 6V wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the parallax effect based on movement of the input device while the positioning of the input device corresponds to the first user interface object (e.g., as previously described with respect to step 704), such as from Figures 6N to 6OA parallax effect is lacking for the displayed icon 604. For example, as described above, the third visual appearance including the third visual value includes a superset or subset of the second visual value. Possible visual appearances and characteristics are previously described herein by non-limiting embodiments and are omitted for brevity. For example, interaction with a stylus (e.g., hovering an input device over the button) is the same or nearly the same as interaction with a comparable cursor (e.g., hovering a cursor based on a cursor input device over the button). In some embodiments, hovering over a button with a cursor includes displaying a first user interface object and / or its background area with a parallax effect, such that movement of the cursor causes movement of one or more portions of the button to be different than movement of another area of ​​the button (e.g., the background and / or border surrounding the button) in response to movement of the cursor (e.g., to indicate that movement of the cursor is being detected, and further movement of the cursor will optionally cause the cursor to move away from the first user interface object). In some embodiments, when hovering over the button with an input device, the visual appearance of the button is the same as when hovering with a cursor, however, the parallax effect is not displayed in response to detecting movement of the input device. In some embodiments, the opposite is true: the parallax effect is not displayed when the cursor device is moved over the button, and the parallax effect is displayed when the input device is hovered and moved at a position relative to the button on the surface. In some embodiments, hovering over the button with the input device includes displaying a different level of parallax effect than hovering over the button with the cursor device. Although some embodiments are described with respect to buttons, it should be understood that any suitable visual object (e.g., graphical objects, text objects, and animated objects) optionally includes such behavior. Displaying parallax effects based on the type of input device reduces the computing power required to display such effects for input device types that are not suitable for parallax effects, and avoids the situation where the parallax effect increases the difficulty of selecting a button with a stylus (e.g., due to movement of the button).

[0273] In some embodiments, in response to detecting the first input, the electronic device displays the first user interface (728) with a lighting effect based on movement of the cursor while the cursor is located at the first user interface object, such as relative to Figures 6U to 6V 604. For example, the lighting effect optionally includes a specular highlight applied to a portion of the first user interface object. The portion optionally is a border surrounding an area associated with the first user interface object.

[0274] In some embodiments, displaying the first user interface object with a third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with a lighting effect based on movement of the input device when the positioning of the input device corresponds to the first user interface object (728) (e.g., as previously described with respect to step 704), such as Figures 6N to 6O , icon 604 displayed in lacks a lighting effect. For example, as described with respect to the parallax effect, the lighting effect associated with a first user interface object (e.g., a button) optionally differs based on whether a cursor or input device is used to interact with the first user interface object. In some embodiments, the first user interface object is displayed with a lighting effect when the cursor corresponds to the first user interface object, but such lighting effect is optionally not displayed when the input device corresponds to the first user interface object. In some embodiments, the lighting effect (e.g., the brightness value of a portion of a simulated light source positioning or a specular highlight) is modified in response to movement of the cursor. In some embodiments, the opposite is true: the lighting effect is displayed in response to interaction with the input device rather than interaction with the cursor. Optionally, the lighting effect is displayed in response to determining that the cursor device and the input device respectively correspond to the first user interface object. In some embodiments, the lighting effect is a specular highlight applied to a portion (e.g., a border or portion) of an area in the user interface that includes the first user interface object. Displaying lighting effects based on the type of input device can reduce the computing power required to display lighting effects for input device types that are not suitable for such effects.

[0275] In some embodiments, the first user interface object corresponds to a link to content (730a) (e.g., a web link to web-based content), such as Figure 6AF In some embodiments, in response to detecting the corresponding object approaching but not contacting the surface, the electronic device modifies (730b) the visual appearance of the first user interface object, such as based on determining that the corresponding object approaching the surface is an input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object (e.g., as previously described with respect to step 704). Figure 6AF610 in the content. For example, the content is a website, application, media, or other user interface environment associated with the link. In some embodiments, the first user interface object is a graphic or text object and is displayed with a modified visual appearance in response to the hover event, such as highlighting, bolding, underlining, and / or other appropriate visual emphasis. For example, the first user interface object is text associated with a hyperlink (e.g., pointing to a web page), and the modification includes highlighting a portion of the user interface that includes the text (e.g., based on an outline of the text or an area of ​​surrounding text), bolding, and / or underlining the text. Additionally, the text is optionally enlarged (e.g., scaled up or increased in font size) to further convey potential interactions with the text. In some embodiments, while displaying the modified visual appearance, in response to receiving a selection of a first selectable option (e.g., contacting a surface with the tip of an input device at the location of the first selectable option), a function corresponding to the first user interface object is initiated. For example, launching an application from the system user interface (e.g., if the first selectable option is an application icon displayed on the home screen user interface of the electronic device, such as a reference to the application icon displayed on the home screen user interface of the electronic device). Figure 4A The present invention also provides a method for displaying a first user interface object with a modified visual appearance, which can convey the interactivity of the underlying object and thus reduce the input required to initiate operations associated with the user interface object and avoid errors in interacting with the user interface object, thereby reducing the computational load and power consumption that would otherwise be required to initiate such operations.

[0276] It should be understood that Figures 7A to 7G The particular order in which the operations are described in the description is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 900, 1100, and 1300) also apply in a similar manner to the details of the processes described above with respect to Figures 7A to 7G Method 700 is described. For example, the interaction between the input device and the surface, the response of the electronic device, the virtual shadow of the input device, and / or the input detected by the electronic device and / or the input device optionally have one or more of the characteristics of the interaction between the input device and the surface, the response of the electronic device, the virtual shadow of the input device, and / or the input detected by the electronic device and / or the input device described herein with reference to other methods described herein (e.g., methods 900, 1100, and 1300). For the sake of brevity, these details are not repeated here.

[0277] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., a processor in combination with a processor). Figure 1A to Figure 1B 、 Figure 3 、 5A to 5I In addition, the above reference Figures 7A to 7G The operation is optionally performed by Figure 1A to Figure 1B . For example, display operations 702a and 702d and detection operation 702b are optionally implemented by event classifier 170, event identifier 180, and event handler 190. When a corresponding predefined event or sub-event is detected, event identifier 180 activates event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally utilizes or calls data updater 176 or object updater 177 to update the application internal state 192. In some embodiments, event handler 190 accesses corresponding GUI updater 178 to update the content displayed by the application. Similarly, a person of ordinary skill in the art will clearly know how to update the content displayed by the application based on Figure 1A to Figure 1B The components depicted in the implementation of other processes.

[0278] Provides feedback about the gestures of an input device

[0279] There are many ways for users to interact with electronic devices, including using input devices such as styluses to interact with such devices. In some embodiments, the electronic device receives input from such input devices, which are based on the relative posture (e.g., orientation and / or positioning) of the input device relative to the surface with which the input device interacts (e.g., contacts and / or hovers). The embodiments described below provide methods for an electronic device to provide feedback about the posture of the input device relative to the surface, thereby enhancing the interaction with the device. Enhancing the interaction with the device reduces the amount of time required for the user to perform an operation, thereby reducing the power consumption of the device and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, this person is optionally referred to as the user of the device.

[0280] Figures 8A to 8AF The present invention illustrates an exemplary method for an electronic device to display an indication of a posture of an input device relative to a surface according to some embodiments of the present disclosure. The embodiments in these figures are used to illustrate the process described below, including reference to Figures 9A to 9K Describe the process.

[0281] Figure 8AA first set of exemplary simulated shadows 832 displayed by an electronic device are illustrated, corresponding to different orientations of the input device 800 relative to the surface (e.g., assuming a virtual light source is positioned above a surface 852). Surface 852 optionally corresponds to a touch screen of the electronic device, but could also be another surface, such as described with reference to method 800. In portions 858a, 860a, and 862a, the z-axis 850 points outward from the surface 852 (e.g., in a direction perpendicular to the plane of the surface 852), the x-axis is parallel to the surface 852, and the y-axis is perpendicular to the x-axis and also parallel to the surface 852. Portions 858a, 860a, and 862a correspond to different orientations of the input device 800 relative to the surface 852, and corresponding portions 858b, 860b, and 862b, respectively, illustrate exemplary simulated shadows 832 displayed by the electronic device in response to detecting that the input device 800 is in these orientations. In some embodiments, the simulated shadow 832 displayed by the electronic device is characterized relative to one or more thresholds, such as thresholds 802 and 854. For example, if the input device 800 (or the tip of the input device 800) is more than the threshold distance 802 from the surface 852, the electronic device optionally does not display the simulated shadow. If the input device 800 (or the tip of the input device 800) is less than the threshold distance 802 from the surface 852, the electronic device optionally displays a simulated shadow of the input device 800 based on the position and / or orientation of the input device 800 relative to the surface 852.

[0282] For example, in Figure 8A854 ). When the input device 800 is perpendicular to the surface and / or within the threshold angle 854 of the normal, the electronic device optionally does not display a simulated shadow of the input device, such as shown in portion 858b . In contrast, in portion 860a , the tilt of the input device 800 is a first amount that is greater than the threshold angle 854 relative to the normal of the surface 852, such as 15 degrees, 20 degrees, 25 degrees, 30 degrees, or 35 degrees. In response, the electronic device displays a simulated shadow 832 of the input device 800 having a first intensity level (e.g., a first blur level, a first shadow spread level, and / or a first opacity level), such as shown in portion 860b . In portion 862a , the tilt of the input device 800 relative to the normal 850 is greater than the tilt of the input device 800 in portion 860a . Compared to the simulated shadow 832 in portion 860b corresponding to the inclination of the input device 800 in portion 860a, the intensity of the simulated shadow 832 in the corresponding portion 862 is less blurred, the shadow spread is less, and the opacity is increased. In some embodiments, the intensity of the visual representation of the simulated shadow 832 changes in response to a change in the inclination of the input device 800. For example, the inclination of the input device 800 in portion 862a is greater than the inclination of the input device 800 in portion 860a, and its corresponding simulated shadow 832 in 862b is stronger (e.g., darker and / or more defined) than the simulated shadow 832 in 860b, while the simulated shadow 832 included in 860b is lighter and more blurred. In addition, in some embodiments, as the inclination of the input device 800 relative to the normal 850 decreases, the length of the simulated shadow 832 becomes shorter, such as Figure 8A As shown in part 860b and part 862b.

[0283] Figure 8B A second set of exemplary simulated shadows 832 are illustrated that change in visual appearance in response to changes in the distance between the input device 800 and the surface. Figure 8B , the tilt of the input device 800 relative to the normal 850 remains constant, while the distance of the input device 800 relative to the surface 852 changes (e.g., from no distance (or contact with the surface) to distance 870 to distance 872) as the input device 800 moves away from the surface. In some embodiments, if the input device 800 (e.g., the tip of the stylus) moves above (or beyond) the predefined threshold distance 802, the simulated shadow 832 is not displayed (e.g., not included in the user interface), similar to Figure 8A As shown in part 858b of Figure 8BAs shown, the visual appearance of the simulated shadow 832 changes in response to changes in the distance of the input device 800. For example, in portion 864a, the input device 800 is in contact with the surface 852 (e.g., there is little or no distance between the tip of the input device 800 and the surface 852). When the input device 800 is in contact with the surface 852, the electronic device displays a simulated shadow of the input device having a first intensity level (e.g., a first blur level, a first shadow spread level, and / or a first opacity level), such as shown in portion 864b. In contrast, in portion 866a, the tip of the input device 800 is at a first distance from the surface 852, such as 0.1 cm, 0.2 cm, 0.5 cm, 0.8 cm, 1 cm, 3 cm, or 5 cm. In response, the electronic device displays a simulated shadow 832 of the input device 800 having a second blur level greater than the first blur level, a second shadow spread level greater than the first shadow spread level, and / or a second opacity level greater than the first opacity level, such as shown in portion 866b. In portion 868a, the distance 872 between the tip of the input device 800 and the surface 852 is greater than the distance 870 in portion 866a. The intensity levels of the simulated shadow 832 corresponding to portion 868a include a third blurriness greater than the second blurriness, a third shadow spread greater than the second shadow spread, and / or a third opacity greater than the second opacity, as shown in portion 868b. For example, in portion 868b, the opacity of the simulated shadow 832 is lower when the tip of the input device 800 is at distance 872 from the surface 852, while in portion 866b, the opacity of the simulated shadow 832 is higher when the tip of the input device 800 is at distance 870 from the surface 852. Compared to portions 868b and 866b, the opacity of the simulated shadow 832 is higher when the input device 800 is in contact with the surface 852 (e.g., there is no distance between the tip of the input device and the surface), as shown in portion 864b.

[0284] like Figure 8A 、 Figure 8B and / or Figure 8C As shown, the direction and / or orientation of the simulated shadow 832 changes as the orientation of the input device 800 relative to the surface 852 changes. Figure 8C A third set of exemplary simulated shadows 832 is illustrated wherein the input device 800 is in contact with the surface 852 and positioned in a downward and rightward orientation, which is the same orientation as Figure 8B The downward and leftward orientations shown in are different. Figure 8B and Figure 8CAs shown, the orientation of the simulated shadow 832 changes accordingly as the orientation of the input device 800 relative to the surface 852 changes (for example, when the tip of the input device 800 is facing downward and right, the tip of the simulated shadow 832 is also facing downward and right, and when the tip of the input device 800 is facing downward and left, the tip of the simulated shadow 832 is also facing downward and left).

[0285] Figure 8C Also illustrated is the change in simulated shadow 832 as the tilt of input device 800 relative to normal 850 changes. For example, in portion 874a, input device 800 is within a threshold angle 854 of normal, despite being at a substantially normal angle. Figure 8A The inclination of the input device 800 in portion 858a is different. When the input device 800 is within a threshold angle 854 of the normal 850, the electronic device optionally does not display the simulated shadow 832 of the input device, such as shown in portion 874b. In contrast, in portion 876a, the inclination of the input device 800 is a first amount that is greater than the threshold angle 854 relative to the normal 850 of the surface 852, such as 15 degrees, 20 degrees, 25 degrees, 30 degrees, or 35 degrees. In response, the electronic device displays a simulated shadow 832 of the input device 800 with a first intensity level (e.g., a first blur level, a first shadow spread level, and / or a first opacity level), such as shown in portion 876b. In portion 878a, the inclination of the input device 800 is greater than the inclination of the input device 800 in portion 876a. Compared to the simulated shadow 832 in portion 876b corresponding to the tilt of the input device 800 shown in portion 876a, the simulated shadow 832 in portion 878b corresponding to portion 878a has a lower intensity, a less diffuse shadow, and / or an increased degree of opacity. Figures 8A to 8C As shown in the example of , in some embodiments, the simulated shadow changes in blurriness, length, intensity, opacity, size, and / or color in response to one or more of the tilt, orientation, and / or distance of the input device 800 relative to the surface 852.

[0286] In some embodiments, the electronic device displays details of the simulated shadow 832 for the input device 800, such as Figures 8A to 8C As described and illustrated, optionally applied to one or more or all simulated shadows illustrated and described with reference to methods 700 , 900 , 1100 , and / or 1300 .

[0287] Figure 8D An exemplary device 500 including a touch screen 504 is illustrated (object 500 corresponding to glyph 804). Device 500 is optionally Figures 8A to 8C The electronic equipment referred to in the description. Figure 8DAs shown, the electronic device 500 is displaying a home screen user interface 890. The home screen user interface 890 includes one or more virtual objects (e.g., virtual objects 826 and 828). Figures 8D to 8K As shown, virtual objects 826 and 828 are representations of application icons, such as application icon 826 and application icon 828, which can be selected to cause device 500 to display and / or launch corresponding applications. Figure 8D Also illustrated is an icon 804 including a side view of the device 500. Icon 804 indicates a relative gesture, including the distance of the input device 800 relative to the surface of the device 500 (e.g., touch screen 504). The icon also includes two thresholds. Threshold 802 is a first distance threshold from the surface of the device 500 (e.g., 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, 50 cm, or 100 cm). Threshold 830 is a second, smaller threshold from the surface of the device 500 (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, or 50 cm). As shown and described later, the device 500 optionally displays a virtual shadow and / or other indication in response to the positioning of the input device relative to thresholds 802 and 830.

[0288] It should be understood that although the virtual objects 826, 828 and user interface 890 are illustrated as being displayed on the touch screen 504, the virtual objects 826, 828 and user interface 890 are optionally displayed on a head-mounted display that includes a display generation component that displays these items to the user in a computer-generated environment (e.g., an augmented reality environment or a three-dimensional environment). In some embodiments, the virtual objects 826, 828 and user interface 890 are displayed on a physical surface on which these items are projected, or on a virtual surface corresponding to at least a portion of these items.

[0289] exist Figure 8D , the input device 800 is more than the threshold 802 away from the device 500 , so the device 500 does not display a virtual shadow corresponding to the input device 800 . Figure 8E The input device 800 is illustrated as being moved so that the input device 800 is within a threshold distance 802 of the surface of the device 500. In response to the device 500 detecting that the input device 800 has moved below the threshold 802 in the glyph 804, the device 500 displays a virtual shadow 832 having a visual appearance corresponding to the input device, as shown in FIG. Figures 8A to 8C In some embodiments, when the input device 800 (e.g., the tip of the stylus) is above the threshold 802, the device 500 stops displaying the virtual shadow 832 in the user interface 890, as shown in FIG. Figure 8DThus, in some embodiments, even if the input device 800 is positioned above the virtual object 826, if the input device is not within the threshold distance 802 of the surface, the user interface displayed by the device 500 does not include the virtual shadow 832. Figure 8E , Figure 8E It is also illustrated that the device 500 presents the virtual object 826 as visually distinguishable from other virtual objects (e.g., virtual object 828) (e.g., the application icon 826 is enlarged) to indicate that the current focus of the input device 800 is on the virtual object 826 (e.g., if the device 500 detects that the input device is in contact with the surface of the touch screen 504, the virtual object 826 will be selected).

[0290] Figure 8F The example shows that the input device 800 is moved so that the input device 800 exceeds a lateral threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) of the virtual object 826, and in response to the device 500 detecting that the input device 800 is away from the virtual object 826, the device 500 displays the virtual object 826 as restored to its original size, as shown in FIG. Figure 8F Thus, in some embodiments, even if the input device 800 is within the threshold distance 802 of the surface shown by the icon 804, if the input device is not within the lateral threshold distance of the virtual object 826, the user interface 890 displayed by the device 500 does not include the virtual object 826 having focus (indicated at least in part by its enlarged visual appearance) (e.g., capable of being selected to initiate a process associated with the virtual object 826).

[0291] Figure 8F Also illustrated is an example of device 500 displaying an indication of a particular portion of virtual shadow 832 in user interface 890 when input device 800 is within (or below) threshold distance 830 as illustrated by icon 804. Figure 8G As shown, the virtual shadow 832 includes a first portion (e.g., virtual shadow portion 832a) corresponding to the barrel of the input device 800 and a second portion (e.g., virtual shadow portion 832b) corresponding to the tip of the input device 800. The virtual shadow portion 832b optionally indicates where the tip of the input device 800 would make contact in the user interface 890 if the input device 800 were closer to the surface of the touch screen 504, thereby providing feedback to the user about how the input device 800 is interacting with the device 500. Furthermore, the virtual shadow portion 832b is optionally visually distinguishable (e.g., darker, more intense, and / or clearer) from the virtual shadow portion 832a.

[0292] Figure 8GThe example shows that when the input device 800 is within the threshold distance 830, the input device 800 moves so that the input device 800 is again within the lateral threshold distance of the virtual object 826. In response to the device 500 detecting that the input device 800 has moved within the lateral threshold distance of the virtual object 826, the electronic device 500 changes the visual appearance of the virtual shadow portion 832b to the shape of a cursor, such as a circular cursor (represented by a small circle on the virtual object 826), indicating that the current focus of the input device is on the virtual object 826. In some embodiments, Figure 8G The size or shape of the circular cursor in is not based on the size or shape of the virtual object 826. Additionally or alternatively, in some embodiments, the electronic device 500 changes the visual appearance of the virtual shadow portion 832b according to the size and / or shape of the virtual object having the focus of the input device 800. For example, Figure 8H As shown, the virtual object 828 has the focus of the input device 800 and is an enlarged square, and therefore, the device 500 changes the visual appearance of the tip of the virtual shadow portion to an enlarged square, providing a visual effect that the virtual object 828 includes a shadow and / or highlight.

[0293] Figures 8I to 8K An example is illustrated of a change in the tilt 836 of the input device 800 relative to the surface of the touch screen 504, and in response to the change in tilt, the device 500 updates the visual appearance of the virtual shadow 832. Figures 8I to 8K In FIG. 5 , the orientation, position, and distance of the input device 800 relative to the surface remain constant, while the tilt of the input device 800 changes (e.g., from Figure 8I The slope is 836 to Figure 8J The inclination is 836 and then Figure 8K For example, Figures 8I to 8K As shown, as the tilt of the input device 800 gets closer to the normal (e.g., closer to the threshold angle 854), the device 500 changes the visual appearance of the virtual shadow 832 (e.g., from Figure 8I Virtual shadow 832 to Figure 8J The virtual shadow 832 in Figure 8KIn some embodiments, in response to detecting that the tilt of the input device decreases to zero (e.g., when the input device is greater than a threshold angle 854 of the normal), the device 500 gradually weakens and / or reduces the visual appearance of the virtual shadow 832 until the virtual shadow 832 is no longer displayed in the user interface. Conversely, in some embodiments, in response to detecting that the tilt of the input device increases to 90 degrees relative to the normal as the tilt of the input device becomes closer to parallel to the surface, the device 500 gradually increases the intensity and / or visual appearance of the virtual shadow 832. Figures 8I to 8K As shown, in some embodiments, as the tilt of the input device 800 is closer to the normal to the surface of the touch screen 504 (but greater than the threshold angle 854 of the normal), the virtual object 832 changes opacity (e.g., becomes less opaque), changes size (e.g., decreases in size), and / or changes color (e.g., becomes lighter in color). For more information on changes in the visual appearance of the virtual shadow 832 based on changes in the posture of the input device 800 relative to the surface, see Figures 8A to 8C .

[0294] In some embodiments, the user interface 890 is a user interface of a drawing application, or a user interface for drafting content using the input device 800. In some embodiments, the drawing application is an application installed on the device 500. Figures 8L to 8AF As shown, user interface 890 includes one or more virtual objects (eg, virtual object 844 ). Figure 8L The virtual object 844 in is a content input palette that includes one or more selectable options associated with content. For example, the content input palette 844 includes an option for selecting a drawing tool (e.g., a content input tool) simulated by the input device 800, an option for undoing the most recent content input-related operation or redoing (e.g., performing again) the most recent content input-related operation, an option for changing the color of the content, and / or an option for selecting a virtual keyboard for entering text. In some embodiments, possible drawing tools of the input device 800 include a text input tool, a pen input tool, a highlighter (or marker) input tool 810, a pencil input tool, an eraser tool, and / or a content selection tool.

[0295] In some embodiments, when the input device 800 (e.g., the tip or other representative portion of the input device) is more than a threshold value 830 away from the surface, the device 500 displays a virtual shadow 832 having a first visual appearance; when the input device 800 is less than the threshold value 830 away from the surface, the device 500 displays a virtual shadow 832 having a second visual appearance different from the first visual appearance. For example, in Figure 8L, when the device 500 detects that the input device 800 is located below the threshold distance 802 but above the threshold distance 830 , the device 500 displays a virtual shadow 832 including a first virtual shadow portion 832 a corresponding to the barrel of the input device 800 . Figure 8M An example is illustrated in which the device 500 displays a second virtual shaded portion 832b corresponding to the tip of the input device 800 when the device 500 detects that the input device 800 is located below the threshold distance 830. In some embodiments, the second virtual shaded portion 832b indicates the location where the input device will touch (or mark) the user interface 890 before the input device 800 touches (or contacts) the surface of the touch screen 504. In some embodiments, the device 500 presents the second virtual shaded portion 832b having a shape and / or color corresponding to the tip of the currently selected drawing tool, which is simulated by the input device 800. For example, in Figure 8M , the currently selected drawing tool is the marker input tool 810 and the selected color is black, as indicated via the color picker tool 814. In response to the device 500 detecting that the input device 800 has moved below the threshold 830, because the device 500 has determined that the currently selected drawing tool is the marker input tool 810, the second virtual shadow portion 832b of the virtual shadow 832 presented by the device 500 has a visual appearance corresponding to the marker input tool. For example, the second virtual shadow portion 832b is a black rectangle corresponding to the active color (e.g., selected from the color picker tool 814) and the flat chisel tip of the marker input tool 810, as shown in FIG. Figure 8M In some embodiments, when the tip of the input device 800 contacts and moves on the surface, the device 500 updates the user interface to display handwriting and / or lines that are black in color and rectangular in shape (e.g., corresponding to the color and / or shape of the second virtual shadow portion 832b).

[0296] In some embodiments, the device 500 detects a gesture indication (e.g., one or more taps) on the input device 800 and interprets the gesture indication as a request to initiate an operation. Figure 8N , the device 500 detects a gesture indication (e.g., indicated by 816) detected by the input device 800, the gesture indication corresponding to a request to change the currently selected drawing tool from the marker tool 810 to the pen input tool 818, and in response to detecting the request to change the currently selected drawing tool, the device 500 selects the pen tool 818 as the currently selected drawing tool, as shown in FIG. Figure 8OIn some embodiments, the input for changing the currently selected drawing tool is any appropriate input for implementing such a change, such as voice input, touch input on the touch screen 504, etc. In response to selecting the pen input tool 818, the device displays a virtual object 844 in the user interface 890, wherein the pen input tool 818 is the currently selected (e.g., active) tool, and the second virtual shadow portion 832b has a visual appearance corresponding to the pen input tool 818 (e.g., a round bullet head), as shown. Figure 8O In addition to changing the shape of the second virtual shadow portion 832b, the device 500 also changes the color of the second virtual shadow portion 832b to correspond to the active color (e.g., selected from the color picker 814), which is gray. In some embodiments, when the tip of the input device 800 contacts and moves on the surface, the device 500 updates the user interface to display the handwriting and / or line in the active color (e.g., the color corresponding to the second virtual shadow portion 832b).

[0297] Go to Figure 8P In some embodiments, upon detecting that the input device 800 is within the threshold distance 830, the device 500 detects a gesture indication on the input device 800 (or other appropriate input, such as voice input, touch input on the touch screen 504, etc.) that corresponds to a request to change one or more drawing settings of the input device 800. In response to detecting the request to change one or more drawing settings of the input device 800, the device 500 displays a content input user interface element 840 (which is optionally positioned based on the positioning of the input device 800 relative to the surface) at or near the second virtual shadow portion 832b, as shown. Figure 8P The content input user interface element 840 includes one or more selectable options for changing one or more drawing settings (e.g., opacity and / or thickness level) associated with the currently selected drawing tool. The device 500 detects an input to change the line thickness level from level 846 (e.g., thinnest) to level 848 (thickest), as shown. Figure 8Q In response to the determined change, the device 500 changes the visual appearance of the second virtual shadow portion 832b to correspond to the change in line thickness level (e.g., from Figure 8P The tiny tip to Figure 8Q In some embodiments, when the tip of the input device 800 contacts and moves on the surface, the device 500 updates the user interface to display the handwriting and / or line with the active line thickness (e.g., corresponding to the size and / or thickness of the second virtual shadow portion 832b), and / or propagates the drawing settings to the handwriting and / or line that has already been drawn.

[0298] exist Figure 8R , device 500 detects a gesture indication 816 (or other appropriate input, such as voice input, touch input on touch screen 504, etc.) that corresponds to a request to change the color of the currently selected drawing tool from gray to black, and in response to detecting the request to change the color of the currently selected drawing tool, device 500 changes the active color from gray to black. Figure 8R The gray shown in the color picker 814 is changed to Figure 8S The color picker 814 shown in FIG. 8 will be displayed in black even if the input device 800 is more than the threshold distance 802 from the surface of the touch screen 504. Figure 8Q The second virtual shadow portion 832b is displayed when the tip of the input device 800 is below the threshold 830 and when an input for changing the line thickness of the currently selected drawing tool is detected, Figure 8R and Figure 8S , the input device 800 is more than a threshold distance 802 from the surface of the touch screen 504, so the device 500 does not display a virtual shadow 832 of the input device 800 when detecting input to change the color of the currently selected drawing tool.

[0299] exist Figure 8T In FIG. 5 , the device 500 detects that the input device 800 is below a threshold value 830 and within a lateral threshold distance of a selectable virtual object 822 that can be selected to create a new drawing in a drawing application. In response, the device 500 displays a virtual shadow 832 of the input device 800 as previously described, and the visual appearance of the virtual shadow portion 832b is based on the shape and / or size of the selectable virtual object 822. Figure 8T As shown, because the selectable virtual object 822 has focus (e.g., the input device 800 is within a lateral threshold distance of the selectable virtual object 822), the visual appearance of the virtual shadow portion 832b changes to a shape similar to or based on the shape of the selectable virtual object 822 (e.g., a square), thereby presenting the selectable virtual object 822 with a visual appearance that includes a shadow and / or highlight. In some embodiments, the visual appearance of the first portion of the virtual shadow corresponding to the barrel of the input device 800 (e.g., the virtual shadow portion 832a) is not based on the currently selected drawing tool (e.g., having the same visual appearance between different selected drawing tools).

[0300] Figure 8UThe example shows that when the input device 800 is within a lateral threshold distance of the selectable virtual object 822, the input device 800 is moved so that the tip of the input device 800 contacts the surface at a location corresponding to the location of the selectable virtual object 822. In response to the device 500 detecting that the input device 800 contacts the surface at a location corresponding to the location of the selectable virtual object 822, the electronic device 500 changes the visual appearance of the selectable virtual object 822 and / or the virtual shadow portion 832b (by changing the size of the virtual shadow portion 832b). Figure 8T 822 is indicated by a darker shading and / or highlighting of the visual appearance of the selectable virtual object 822 in the drawing to indicate selection of the virtual object 822. In response to such selection, the device 500 optionally displays a blank drawing canvas in the user interface 890.

[0301] exist Figures 8V to 8AF , user interface 890 includes a content input area 812. In some embodiments, content input area 812 is configured to receive handwriting input (e.g., drawing input via input device 800) and display a representation of the handwriting input (e.g., if drawing input is provided) and / or display font-based text (e.g., if font-based text input is provided and / or if the handwriting input is converted to font-based text based on the currently selected drawing tool of input device 800). In some embodiments, such as Figure 8V As shown, selecting the text input tool 820 as the currently selected drawing tool causes the device 500 to enter a text input mode in which handwriting input drawn in the content input area 812 is analyzed as text characters, recognized, and converted into font-based text in the content input area 812. Figure 8V , input device 800 is within a threshold distance 802 of the surface of touch screen 504 , but more than a threshold distance 830 from the surface of the touch screen, and therefore, device 500 displays a virtual shadow 832 of input device 800 .

[0302] Figure 8WThe example illustrates that when the input device is below the threshold 830, the device continues to display the virtual shadow 832 corresponding to the currently selected drawing tool (i.e., the text input tool 820), but does not display the tip portion of the virtual shadow as described above (e.g., virtual shadow portion 832b, which has a shape and / or color corresponding to the tip of the selected drawing tool simulated by the input device 800). In some embodiments, when the currently selected drawing tool is the text input tool 820, the handwriting input and the corresponding (e.g., converted) font-based text are displayed with a default color and / or line thickness. Therefore, when the text input tool 820 is the currently selected drawing tool, the color and / or line thickness of the provided handwriting input are optionally independent of the text input tool 820, and the device 500 does not display the virtual shadow portion 832b having a color and / or shape corresponding to the text input tool 820.

[0303] Figure 8X It is illustrated that when the input device 800 is below the threshold 830, the input device 800 is moved so that the input device 800 is within a lateral threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) of the content (e.g., text) within the content input area 812. In response to the device 500 detecting that the input device 800 is within the threshold distance of the content, the electronic device 500 changes the visual appearance of the second virtual shadow portion 832b of the virtual shadow 832 to a text insertion cursor. For example, when the tip of the input device 800 is located within a lateral threshold distance of the content within the content input area 812, the second virtual shadow portion 832b is optionally replaced with an indication of the text insertion cursor (e.g., a simulated shadow on the cursor). In some embodiments, the vertical positioning of the virtual shadow portion 832b in the content is aligned to the content and / or text line closest to the tip of the input device 800, such as Figure 8X Additionally, the horizontal positioning of the virtual shadow portion 832b within the content optionally corresponds to the horizontal positioning of the tip of the input device 800.

[0304] For example, in Figure 8Y In the embodiment, the device 500 detects that the input device 800 moves upward in the user interface 890, and correspondingly displays the virtual shadow portion 832b including the text insertion cursor from the lower row of text in the content input area 812 to the middle row of text. Figure 8X and Figure 8YAs shown, the movement of the text insertion cursor in the content corresponds to the movement of the input device 800. For example, when the tip of the input device 800 is located above a specific character of the font-based text in the content input area 812, the text insertion cursor is displayed accordingly (e.g., the text insertion cursor is positioned to the nearest character of the font-based text without requiring the user to more precisely move the tip of the input device 800 to the specific character). Therefore, in some embodiments, the vertical and / or horizontal positioning of the virtual shaded portion 832b and / or the text insertion cursor are separated from the actual positioning in the user interface 890 corresponding to the positioning of the tip of the input device 800, because the device 500 automatically aligns the virtual shaded portion 832b and / or the text insertion cursor to the nearest line and / or character of the content in the content input area 812.

[0305] exist Figures 8Z to 8AF In some embodiments, the user interface 890 includes one or more virtual objects (e.g., virtual objects 842 and 842a). Virtual object 842 is a content alignment user interface element (e.g., a virtual ruler), which includes one or more virtual objects 842a (e.g., guide points). In some embodiments, the device 500 aligns the second virtual shadow portion 832b to the nearest virtual object 842a to facilitate automatically drawing the aligned line based on the content alignment user interface element. Figure 8Z In the illustrated example, the device 500 detects that the input device 800 is below a threshold 802 (eg, the tip of the input device is within the threshold distance 802 from the surface), and in response, the device 500 displays a virtual shadow 832 as previously described. Figure 8AA The input device 800 is illustrated as being moved to a position below a threshold 830 (e.g., the tip of the input device is within the threshold distance 830 from the surface). In response to the device 500 detecting that the input device 800 is below the threshold 830, the electronic device 500 changes the visual appearance of the virtual shadow 832 to include a second virtual shadow portion 832b corresponding to the tip of the currently selected drawing tool (e.g., the pen tool 818), as previously described with reference to FIG. Figures 8N to 8Q As stated.

[0306] Figures 8AB to 8AD The example shows the contact and movement of the tip of the input device 800 along the guide line of the virtual object 842. In response to the contact and movement of the input device 800, the device 500 generates a line at the position of the guide line of the virtual object 842 according to the movement of the input device 800, such as Figures 8AB to 8ADIn some embodiments, during movement of the input device 800, the virtual shaded portion 832b is vertically aligned to the horizontal guide line (e.g., when the tip of the input device 800 is within a lateral threshold distance of the horizontal guide line, such as 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, or 5 cm) to facilitate drawing a straight line according to the movement of the input device 800, even if the tip of the input device 800 is not located at each position of the guide line.

[0307] from Figures 8AD to 8AE , the input device 800 moves further to the right to the object 842a. In response, as Figure 8AE As shown, the second virtual object portion 832b corresponding to the tip of the currently selected drawing tool is aligned to the nearest virtual object 842a because the virtual object is the closest guide point to the position of the input device 800 and / or because the tip of the input device 800 has moved to within a lateral threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, or 5 cm) of the virtual object 842a. In some embodiments, in conjunction with aligning the virtual shadow portion 832b to the object 842a, the device 500 also completes a line drawn by the input device 800 from the left square to the object 842a, and the line is optionally aligned based on the alignment provided by the alignment object 842 (e.g., relative to the actual path traversed by the input device 800). In some embodiments, as long as the device 500 detects movement of the input device 800 within the lateral threshold distance of the virtual object 842a, the device 500 will continue to display the second virtual object portion 832b at the virtual object 842a, as shown. Figures 8AE to 8AF In some embodiments, when the device 500 detects the input device 800 within a lateral threshold distance of one or more guide points, the device 500 traverses the one or more guide lines to automatically generate lines aligned with the guide lines along the one or more guide points (e.g., aligned relative to an actual path traversed by the input device 800). Thus, in some embodiments, even if the second virtual object portion 832b is part of the virtual object 832, if the input device is within a lateral threshold distance of a content alignment guide (e.g., virtual object 842a), the device 500 displays the second virtual object portion 832b as being offset (or separated) from the virtual object 832.

[0308] Figures 9A to 9K is a flow chart illustrating a method 900 for providing feedback about a posture of an input device relative to a surface. Method 900 is optionally performed on an electronic device (such as device 100, device 300, and device 500), as described above with reference to Figure 1A to Figure 1B 、 Figures 2 to 3 、 Figures 4A to 4B and 5A to 5ISome operations in method 900 may be optionally combined, and / or the order of some operations may be optionally changed.

[0309] As described below, method 900 provides a method for providing feedback regarding the posture of an input device relative to a surface. This method reduces the cognitive burden on a user when interacting with the user interface of a device of the present disclosure, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charges.

[0310] In some embodiments, method 900 is performed at an electronic device that communicates with a display generating component, one or more sensors (e.g., a touch-sensitive surface), and an input device. For example, the electronic device is a mobile device (e.g., a tablet device, a smart phone, a media player, or a wearable device) that includes a touch screen and wireless communication circuitry, or a computer that includes one or more of a keyboard, a mouse, a trackpad, and a touch screen and wireless communication circuitry, and optionally has one or more of the characteristics of the electronic device of method 700. In some embodiments, the display generating component has one or more characteristics of the display generating component in method 700. In some embodiments, the input device has one or more characteristics of one or more input devices in method 700. In a device that includes a touch-sensitive surface (e.g., Figure 1A touch-sensitive display system 112 or Figure 4A In some embodiments of the present invention, an input device (e.g., a stylus in communication with the electronic device, such as the stylus described with reference to methods 700, 900, 1100, and / or 1300) is detected hovering over the touch-sensitive surface so that when the input device is detected hovering over the touch-sensitive surface, movement of the input device relative to the touch-sensitive surface changes the representation of the virtual shadow corresponding to the input device, as described in method 900 herein. In some embodiments, the one or more sensors optionally include Figure 1A One or more sensors in the

[0311] In some embodiments, the electronic device displays (902a) a user interface via a display generation component, such as Figures 8D to 8AF890 in the user interface. For example, the user interface of an application installed and / or running on the electronic device, or the user interface of the operating system of the electronic device. In some embodiments, the user interface is the home screen user interface of the electronic device, or the user interface of an application accessible by the operating system of the electronic device, such as a word processing application, a note application, an image management application, a digital content management application, a drawing application, a presentation application, a word processing application, a spreadsheet application, a messaging application, a web browsing application, and / or an email application. In some embodiments, the user interface simultaneously includes multiple user interfaces of one or more applications and / or the operating system of the electronic device. In some embodiments, the user interface has one or more characteristics of the user interface of method 700.

[0312] In some embodiments, while displaying a user interface via a display generation component, the electronic device detects (902b) a first posture (e.g., position and / or orientation) of an input device (e.g., a stylus) relative to a surface (e.g., a touch-sensitive surface, a physical surface on which the user interface is projected, or a virtual surface corresponding to at least a portion of the user interface), such as Figure 8A For example, the electronic device and / or touch-sensitive surface obtains information including position / attitude (pitch, yaw, and / or roll), orientation, tilt, path, force, distance, and / or location of the input device relative to the surface through one or more sensors of the input device, one or more electrodes in the surface, one or more planar surfaces of physical objects (or physical areas) in the physical environment, other defined coordinate systems, other sensors, and / or other input devices (e.g., an input trackpad that includes a specialized surface configured to convert motion and posture information of the input device).

[0313] In some embodiments, in response to detecting a first posture of the input device relative to a surface, and based on determining that the first posture of the input device relative to the surface includes the input device being within a threshold distance of the surface (e.g., 0 cm, 0.01 cm, 0.05 cm, 0.1 cm, 0.2 cm, 0.5 cm, 0.8 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm), the electronic device displays (902c) the user interface via the display generation component as a representation including a virtual shadow corresponding to the input device, such as Figure 8A 860b of the virtual shadow 832, wherein the representation of the virtual shadow has a first visual appearance and a first positioning in the user interface based on a first posture of the input device relative to the surface, such as Figure 8AThe appearance and / or positioning of the virtual shadow 832 in portion 860b of . For example, the electronic device optionally determines that the input device is within a threshold distance of a surface (in some embodiments, not touching the surface, in some embodiments, touching the surface), and based on this determination, displays a virtual shadow having a first visual appearance corresponding to a first intensity (e.g., a first degree of tint, a first shape, a first size, a first degree of transparency, a first angle, a first distance, a first degree of blur, and / or a first other characteristic of the virtual shadow), which is virtually projected onto the user interface by the input device. For example, the first positioning of the representation of the virtual shadow virtually projected onto the user interface is based on posture information of the physical position and / or posture of the input device relative to the surface. In some embodiments, the positioning and / or visual appearance of the virtual shadow corresponding to the input device is like one or more external light sources at one or more locations relative to the electronic device shining onto the surface and / or surface of the input device and / or display generating component, thereby displaying the virtual shadow.

[0314] In some embodiments, while displaying a user interface via a display generation component as a representation that includes a virtual shadow corresponding to an input device, wherein the virtual shadow representation has a first visual appearance and a first positioning in the user interface, the electronic device detects (902d) movement of the input device relative to the surface from a first posture to a second posture (e.g., positioning and / or orientation) that is different from the first posture, such as from Figure 8A The gesture of the input device 800 in portion 860a is Figure 8A Movement of the posture of the input device 800 in portion 862a of . For example, the electronic device detects input from a user (e.g., finger manipulation on the input device, gestures on the input device, and / or rotational or translational movement of the input device) to move the positioning and / or orientation of the input device relative to the surface from a first posture to a second posture. For example, the second posture of the input device is optionally within a threshold distance of the surface and is optionally oriented vertically relative to a reference axis, whereas the input device in the first posture is optionally oriented horizontally relative to the reference axis. In some embodiments, transitioning from the first posture to the second posture includes changing the distance of the input device from the surface without changing the orientation of the input device relative to the surface; in some embodiments, transitioning from the first posture to the second posture includes changing the orientation of the input device relative to the surface without changing the distance from the surface; in some embodiments, transitioning from the first posture to the second posture includes changing the orientation of the input device relative to the surface and changing the distance from the surface.

[0315] In some embodiments, in response to detecting movement of the input device from a first posture to a second posture relative to a surface and based on determining that the second posture relative to the surface includes the input device being within a threshold distance of the surface, the electronic device displays (902e) the user interface via the display generation component as a representation including a virtual shadow corresponding to the input device, the representation having a second visual appearance different from the first visual appearance and a second position different from the first position in the user interface based on the second posture of the input device relative to the surface, such as Figure 8A The appearance and / or positioning of the virtual shadow 832 in portion 862b of . For example, the virtual shadow visually changes the appearance of the simulated shadow with a second degree of tint, a second shape, a second size, a second degree of transparency, a second angle, a second distance, a second degree of blur, and / or otherwise. In some embodiments, once the input device is within a threshold distance of the surface, the electronic device displays (or presents) a representation of the virtual shadow that changes based on the positioning and / or orientation information of the physical position and / or posture of the input device relative to the surface; for example, the virtual shadow is more intense (clearer and / or darker) at the second position (closer to the surface) than at the first position (further away from the surface but within the threshold distance). In some embodiments, the positioning and / or visual appearance of the virtual shadow corresponding to the input device in the second posture is like the one or more external light sources mentioned above at the (same) one or more positions relative to the electronic device that illuminate the surface and / or surface of the input device and / or display generating component, thereby displaying the virtual shadow. Displaying a virtual shadow of an input device that changes based on the posture of the input device provides an indication of the posture of the input device, the distance to the surface, and / or the distance to the target user interface element, and enables the user to precisely place the input device, thereby reducing errors in interaction between the input device and / or the surface (e.g., avoiding accidental handwriting by the input device in the user interface) and reducing the input required to correct such errors.

[0316] In some embodiments, the representation of the virtual shadow corresponding to the input device includes a first portion (such as a portion of a drawing tool) corresponding to the barrel of the currently selected drawing tool of the input device. Figure 8M and a second portion (904) corresponding to the tip of the currently selected drawing tool (such as Figure 8M832b in ). For example, the input device simulates one or more virtual drawing tools (e.g., a pen, pencil, paintbrush, and / or highlighter), and when it is determined that the input device is within a threshold distance of the surface, the user interface optionally includes an indication of a first portion corresponding to the barrel of the currently selected drawing tool and an indication of a second portion corresponding to the tip of the currently selected drawing tool. In some embodiments, the indication of the first portion corresponding to the barrel of the currently selected drawing tool indicates one or more of the distance of the input device relative to the surface, the orientation of the input device relative to the surface, and / or the tilt of the input device relative to the surface. In some embodiments, the indication of the second portion corresponding to the tip of the currently selected drawing tool indicates one or more of the following: the proximity of the input device to the surface, and / or the location where the user interface will draw (or render a display of) the handwriting if and / or when the input device provides handwriting input to the user interface (e.g., movement of the input device while the tip of the input device is in contact with the surface). In some embodiments, a representation of the virtual shadow comprising the first portion and the second portion is not displayed (e.g., because the user interface of an application installed on the electronic device already presents a visual indication of the input device, because the user interface of the application installed on the electronic device does not support (is not configured to) present a representation of the virtual shadow, and / or because the input device is beyond a threshold distance from the surface). In some embodiments, the user interface includes only a portion of the first portion corresponding to the barrel of the currently selected drawing tool of the input device, rather than the entire first portion corresponding to the barrel of the currently selected drawing tool of the input device (e.g., because the remainder of the first portion would extend beyond the display boundaries of the display generation component). In some embodiments, the user interface includes the first portion corresponding to the barrel of the currently selected drawing tool of the input device, but does not include the second portion, as described in more detail later. In some embodiments, the user interface includes the second portion corresponding to the tip of the currently selected drawing tool of the input device, but does not include the first portion, as described in more detail later with reference to step 950. In some embodiments, the user interface first includes the first portion corresponding to the barrel of the currently selected drawing tool of the input device, and then includes the second portion corresponding to the tip of the currently selected drawing tool of the input device, as described in more detail later with reference to step 948. By presenting a virtual shadow with two distinct portions, the electronic device enables a user to accurately place an input device relative to a surface and understand the type and / or characteristics of a virtual drawing tool before providing handwriting input on a user interface, thereby reducing errors in interactions between the input device and / or surface (e.g., avoiding accidental handwriting in the user interface caused by the input device) and reducing the input required to correct such errors.

[0317] In some embodiments, while displaying a representation of a virtual shadow corresponding to the input device, the electronic device detects (906a) movement of the input device relative to the surface from a second posture to a third posture different from the second posture, such as movement of the input device 800 from a second posture to a third posture different from the second posture. Figures 8Q to 8R For example, the third posture is outside the above threshold distance of the surface.

[0318] In some embodiments, in response to detecting movement of the input device from the second posture to the third posture relative to the surface and based on determining that the third posture relative to the surface includes the input device being outside a threshold distance of the surface (e.g., 0 cm, 0.01 cm, 0.05 cm, 0.1 cm, 0.2 cm, 0.5 cm, 0.8 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm), the electronic device stops (906b) displaying a representation of the virtual shadow corresponding to the input device, such as not displaying Figure 8R 832 in the virtual shadow. For example, when the input device is detected to be outside a threshold distance from the surface, the representation of the virtual shadow disappears from the user interface (e.g., is not included). In some embodiments, the threshold distance at which the electronic device stops displaying the representation of the virtual shadow corresponding to the input device (the "hidden distance") is different from the threshold distance at which the electronic device begins displaying the representation of the virtual shadow corresponding to the input device (the "displayed distance") (e.g., the threshold hysteresis avoids jittering when displaying the representation of the virtual shadow when the input device is at or very close to the threshold distance). In some embodiments, the "hidden distance" is greater than the "displayed distance"; in some embodiments, the "hidden distance" is less than the "displayed distance". In some embodiments, the representation of the virtual shadow disappears only after the entire input device is detected to be outside the threshold distance from the surface. In some embodiments, the representation of the virtual shadow disappears only after a majority (e.g., greater than 70%, 75%, 80%, 85%, 90%, or 95%) of the input device is detected to be outside the threshold distance from the surface. In some embodiments, the representation of the virtual shadow disappears only after the tip of the input device (or other specific portion of the input device) is detected to be outside the threshold distance from the surface. Disabling display of the virtual shadow indicates that input from the input device will not be detected by the electronic device, thereby reducing errors in interactions between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in the user interface) and reducing the input required to correct such errors.

[0319] In some embodiments, the first posture relative to the surface includes the input device being a first distance from the surface, such as Figure 8L , and the first visual appearance includes an intensity of the representation of the virtual shadow is a first intensity (908), such as Figure 8LFor example, a representation of a virtual shadow having a first intensity is displayed with a first degree of tint, a first shape, a first size, a first degree of transparency, a first angle, a first deviation (e.g., a gap between the virtual shadow and the input device), a first degree of blur, and / or other first characteristics of the virtual shadow. In some embodiments, a second posture relative to a surface includes the input device being at a second distance from the surface that is different from the first distance, such as Figure 8M , and the second visual appearance includes an intensity of the representation of the virtual shadow that is a second intensity different from the first intensity (908), such as Figure 8M . The electronic device detects a change in the distance of the input device from the surface (or relative to the surface) and, in response to the change in distance, optionally updates the representation of the virtual shadow. For example, the second distance is less than the first distance (e.g., closer to the surface), and the representation of the virtual shadow with the second intensity is displayed with a second tint that is darker than the first tint (e.g., darker), a second shape that is shorter than the first shape, a second size that is smaller than the first size, a second transparency that is lower than the first transparency, a second angle that is greater than the first angle relative to the input device (e.g., closer to the input device after rotation), a second deviation that is smaller than the first deviation (e.g., a smaller gap between the virtual shadow and the input device), and / or a second blur that is smaller than the first blur (e.g., sharper). In some embodiments, if the second distance is greater than the first distance (e.g., further from the surface), the representation of the virtual shadow includes being displayed at a third intensity that is lighter than the first tint (e.g., lighter), a second shape that is longer than the first shape, a third size that is larger than the first size, a third transparency that is higher than the first transparency, a third angle that is less than the first angle relative to the input device (e.g., farther from the input device after rotation), a third deviation that is less than the first deviation (e.g., a larger gap between the virtual shadow and the input device), and / or a third blur that is greater than the first blur (e.g., blurrier). Displaying a virtual shadow of the input device that changes intensity based on a change in the posture of the input device provides an indication of the relative positioning of the input device relative to the surface, the distance from the surface, and / or the distance from a target user interface element, and enables the user to accurately place the input device, thereby reducing errors in interaction between the input device and / or the surface (e.g., avoiding accidental handwriting by the input device in the user interface) and reducing the input required to correct such errors.

[0320] In some embodiments, the first posture relative to the surface includes the input device having a first orientation relative to the surface, such as Figure 8CThe orientation of the input device 800 in portion 876a of the image processing unit 800 is determined, and the first visual appearance includes an intensity of the representation of the virtual shadow being a first intensity (e.g., as described in more detail with reference to step 908), wherein the intensity of the representation of the virtual shadow is based on the orientation of the input device relative to the surface (910), such as Figure 8C The intensity of the virtual shadow 832 in portion 876b of the input device is determined. The input device has a first orientation relative to the surface, optionally including a first tilt (or angle) relative to a normal (perpendicular) to the surface, as described in more detail later with reference to step 914. In some embodiments, the electronic device detects a change in tilt relative to the normal to the surface, and in response to the change in tilt relative to the normal to the surface, the electronic device updates a representation of the virtual shadow, as described in more detail later with reference to step 914. Displaying a virtual shadow of the input device that changes intensity based on the change in orientation of the input device provides an indication of the tilt of the input device relative to the surface and enables the user to accurately place the input device, thereby reducing errors in interaction between the input device and / or the surface (e.g., avoiding accidental handwriting by the input device in the user interface) and reducing the input required to correct such errors.

[0321] In some embodiments, while displaying a representation of a virtual shadow corresponding to an input device with the input device having a first orientation relative to a surface, the electronic device detects (912a) movement of the input device relative to the surface from a first posture to a third posture different from the first posture, such as movement of the input device from a first posture to a third posture different from the first posture. Figure 8C Section 876a to Figure 8C For example, the third posture is within a threshold distance of the surface (e.g., 0 cm, 0.01 cm, 0.05 cm, 0.1 cm, 0.2 cm, 0.5 cm, 0.8 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm).

[0322] In some embodiments, in response to detecting movement of the input device relative to the surface from a first posture to a third posture and based on determining that the third posture includes the input device having a second orientation within the first orientation range (e.g., within 1 degree, 2 degrees, 3 degrees, 5 degrees, 10 degrees, 15 degrees, 20 degrees, 30 degrees, 35 degrees, 40 degrees, or 45 degrees, and / or between 0 degrees and 45 degrees, between 3 degrees and 30 degrees, or between 5 degrees and 20 degrees) relative to the surface, (e.g., the third posture includes a third inclination (e.g., 15 degrees from normal) that is less than the third inclination wherein the first tilt of the posture is in a first degree (e.g., 45 degrees from the normal), and the representation of the virtual shadow has a third visual appearance that is different from the first visual appearance (e.g., a lower degree of intensity than the intensity of the first visual appearance), the electronic device display (912b) displays a representation of the virtual shadow corresponding to the input device in a third visual appearance that is different from the first visual appearance, wherein the third visual appearance includes the representation of the virtual shadow having a third intensity that varies based on the orientation of the input device relative to the surface within the first orientation range (e.g., as described in more detail with reference to step 908), such as Figures 8A to 8C , the intensity of the virtual shadow 832 changes based on the orientation of the input device 800 changing to within a threshold angle 854 of the normal 850. For example, when the orientation of the input device relative to the surface is within a first orientation range, the intensity of the representation of the virtual shadow optionally changes gradually. For example, as the inclination of the input device decreases, the intensity of the virtual shadow also decreases. In some embodiments, when the inclination of the input device decreases from 20 degrees to 5 degrees from the normal, the intensity of the representation of the virtual shadow gradually decreases. For example, when the orientation of the input device includes a 20-degree inclination from the normal, the electronic device optionally displays a representation of the virtual shadow whose intensity decreases (e.g., the shape becomes shorter, smaller, and / or blurrier). As the inclination continues to decrease from 20 degrees to, for example, 10 degrees from the normal, the electronic device optionally displays a representation of the virtual shadow whose intensity is weaker than when the input device includes a 20-degree inclination from the normal (e.g., the shape is shorter, smaller, and / or blurrier). In some embodiments, the intensity of the representation of the virtual shadow decreases until a second orientation range is reached, as described in more detail with reference to step 912. Displaying a virtual shadow of an input device that gradually changes intensity based on changes in the orientation of the input device provides an indication of the tilt of the input device relative to a surface and enables a user to precisely place the input device, thereby reducing errors in interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in a user interface) and reducing the input required to correct such errors.

[0323] In some embodiments, in response to detecting movement of the input device relative to the surface from a first posture to a third posture and based on determining that the third posture includes the input device having a third orientation relative to the surface within a second orientation range that is different from the first orientation range (e.g., within 0 degrees, 1 degree, 2 degrees, 3 degrees, 5 degrees, 10 degrees, 15 degrees, 20 degrees, 30 degrees, 35 degrees, 40 degrees, or 45 degrees from normal, and / or between 30 degrees and 0 degrees from normal, between 20 degrees and 0 degrees from normal, or between 5 degrees and 0 degrees from normal), Figures 8A to 8C The electronic device stops (914) displaying a representation of a virtual shadow corresponding to the input device, such as Figure 8A Section 858b and Figure 8C 874b in . For example, the intensity of the representation of the virtual shadow decreases until a second orientation range is reached, within which the representation of the virtual shadow optionally gradually weakens. In some embodiments, the second orientation range is closer to the vertical direction than the first orientation range. For example, when the third orientation of the input device includes zero tilt, the zero tilt is within the second orientation range, and the representation of the virtual shadow disappears from the user interface (e.g., is not included). In some embodiments, when the orientation of the input device is within the second orientation range, the orientation change of the input device remains within the second orientation range without causing a change in the display in the user interface (e.g., aspects of the representation of the virtual shadow will not be displayed). Stopping the display of the virtual shadow when the input device is substantially perpendicular to the surface provides an indication that the input device is substantially perpendicular to the surface, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding the input device from causing accidental handwriting in the user interface) and reducing the input required to correct such errors.

[0324] In some embodiments, the first posture relative to the surface includes the input device being in a first orientation relative to the surface, and the first visual appearance includes the shape of the representation of the virtual shadow being a first shape (916), such as Figure 8A860b of the virtual shadow 832. For example, the first orientation is perpendicular or approximately perpendicular to the surface (e.g., perpendicularity is within 1 degree, 3 degrees, 5 degrees, 10 degrees, 15 degrees, or 20 degrees). When the input device is perpendicular or approximately perpendicular to the surface, the representation of the virtual shadow optionally includes the first shape. In some embodiments, when the input device is parallel or more or approximately parallel to the surface, the electronic device optionally displays a representation of the virtual shadow having the first shape, which is less visible (e.g., shorter and / or smaller) than the second shape of the representation of the virtual shadow, as described in more detail later with reference to step 914. In some embodiments, the representation of the virtual shadow includes a second portion corresponding to the tip of the drawing tool currently selected by the input device, but does not include a first portion corresponding to the barrel of the currently selected drawing tool of the input device. In some embodiments, when the first orientation of the input device is perpendicular or approximately perpendicular to the surface, the representation of the virtual shadow includes a first corresponding portion of the first portion corresponding to the barrel of the currently selected drawing tool of the input device, but does not include a second corresponding portion of the first portion of the virtual shadow.

[0325] In some embodiments, the second posture relative to the surface includes the input device being in a second orientation relative to the surface that is different from the first orientation, and the second visual appearance includes a shape of the representation of the virtual shadow being a second shape that is different from the first shape (916), such as Figure 8A The shape of the virtual shadow 832 in the portion 862b of the input device. For example, the first orientation is parallel or approximately parallel to the surface (e.g., parallelism is within 1 degree, 3 degrees, 5 degrees, 10 degrees, 15 degrees, or 20 degrees). When the input device is parallel or approximately parallel to the surface, the representation of the virtual shadow optionally includes a second shape. In some embodiments, the electronic device displays a representation of the virtual shadow with the second shape that is more visible than the representation of the virtual shadow with the first shape. For example, the second shape is optionally longer than the first shape, and / or the second shape is optionally larger than the first shape. In some embodiments, when the first orientation of the input device is parallel or approximately parallel to the surface, the representation of the virtual shadow includes most (or all) of the first portion corresponding to the barrel of the currently selected drawing tool of the input device, compared to when the input device is perpendicular or approximately perpendicular to the surface, only a portion of the first portion corresponding to the barrel of the currently selected drawing tool of the input device is included. Displaying a virtual shadow of an input device that changes based on the orientation of the input device provides an indication of the orientation of the input device relative to a surface and enables a user to precisely place the input device, thereby reducing errors in interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in a user interface) and reducing the input required to correct such errors.

[0326] In some embodiments, the first posture relative to the surface includes the input device being in a first orientation relative to the surface, and the first visual appearance includes the orientation of the representation of the virtual shadow being in a first corresponding orientation relative to the user interface (918), such as Figure 8A The orientation of the virtual shadow 832 in portion 860b of the input device is determined. For example, in a first orientation, the end of the barrel of the input device (e.g., farthest from the tip of the input device) points in a direction toward an edge of the user interface (e.g., the top edge, the left edge, the right edge, or the left edge). In some embodiments, the electronic device displays a representation of the virtual shadow in a corresponding orientation relative to the user interface based on the detected orientation of the input device (e.g., the end of the first portion corresponding to the barrel of the currently selected drawing tool of the input device points to the top edge of the user interface, rather than the bottom edge, the right edge, or the left edge of the user interface). In some embodiments, the tip of the input device points in a direction toward an edge of the user interface (e.g., the top edge, the left edge, the right edge, or the left edge), and the electronic device displays a representation of the virtual shadow in a corresponding orientation relative to the user interface based on the detected orientation of the input device (e.g., the end of the second portion corresponding to the tip of the currently selected drawing tool of the input device points to the top edge of the user interface, rather than the bottom edge, the right edge, or the left edge).

[0327] In some embodiments, the second posture relative to the surface includes the input device being in a second orientation relative to the surface that is different from the first orientation (e.g., in the second orientation, the barrel end of the input device is pointing in a direction toward the bottom edge of the user interface that is opposite (or different) from the first orientation that is directed toward the top edge of the user interface), and the second visual appearance includes an orientation of the representation of the virtual shadow that is a second corresponding orientation relative to the user interface that is different from the first corresponding orientation (918), such as Figure 8CThe orientation of the virtual shadow 832 in portion 876b of the input device. For example, the user interface includes a representation of the virtual shadow in a second corresponding orientation relative to the user interface (e.g., the end of the first portion corresponding to the barrel of the currently selected drawing tool of the input device points to the bottom edge of the user interface, rather than the top edge, right edge, or left edge of the user interface). The representation of the virtual shadow in the second corresponding orientation is optionally in an opposite direction (or a different direction) to the representation of the virtual shadow in the first corresponding orientation because the representation of the virtual shadow in the first corresponding orientation optionally includes the end of the first portion corresponding to the barrel of the currently selected drawing tool of the input device pointing to the top edge of the user interface. In some embodiments, the tip of the input device points in a direction toward the bottom edge of the user interface, and the electronic device displays the representation of the virtual shadow in a corresponding orientation relative to the user interface (e.g., the end of the second portion corresponding to the tip of the currently selected drawing tool of the input device points to the bottom edge of the user interface, rather than the top edge, right edge, or left edge of the user interface). Displaying a virtual shadow of an input device that changes orientation based on a change in the orientation of the input device provides an indication of whether the input device is pointing at a particular edge or boundary of a surface and enables a user to precisely place the input device, thereby reducing errors in interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in a user interface) and reducing the input required to correct such errors.

[0328] In some embodiments, the user interface is a user interface of a drawing application (920), such as Figure 8L890 in the user interface. For example, the representation of the virtual shadow is included in the user interface of a drawing application. A drawing application is optionally an application in which handwriting input received from an input device is displayed in the user interface in the form of handwriting on a drawing canvas. In some embodiments, the representation of the virtual shadow is included in the user interface of various applications accessible to the electronic device, such as a word processing application, a photo management application, a spreadsheet application, a presentation application, a website creation application, an email application, or other content creation application. In some embodiments, the electronic device displays the virtual shadow in the user interface of a drawing application (e.g., an application configured to receive drawing input from an input device), but does not display the virtual shadow in other types of applications (e.g., applications that are not configured to receive drawing input from an input device, such as a calendar application, a TV / movie browsing application, a digital wallet application, and / or a map / navigation application). In some embodiments, the electronic device does not display the virtual shadow in the system user interface (e.g., as described with reference to step 922). Displaying virtual shadows in drawing applications enables users to precisely place input devices when providing drawing input, thereby reducing errors in interactions between input devices and / or surfaces (e.g., avoiding accidental handwriting from an input device in a user interface) and reducing the input required to correct such errors.

[0329] In some embodiments, displaying the representation of the virtual shadow with a first visual appearance includes (922a), based on determining that the currently selected drawing tool for the input device is the first drawing tool, displaying the representation of the virtual shadow with a first shape corresponding to the first drawing tool (922b), such as Figure 8N . For example, if the first drawing tool is a virtual pen (e.g., the input device is used as a virtual pen), the first shape of the virtual shadow corresponds to the round bullet tip of the virtual pen. In another example, the second portion corresponding to the tip of the currently selected drawing tool of the input device (represented by the virtual shadow having the first shape) is optionally a round bullet tip (corresponding to the round bullet tip of the virtual pen).

[0330] In some embodiments, displaying the representation of the virtual shadow having the first visual appearance includes, based on determining that the currently selected drawing tool for the input device is a second drawing tool that is different from the first drawing tool, displaying the representation of the virtual shadow in a second shape corresponding to the second drawing tool, wherein the second shape is different from the first shape (922b), such as Figure 8O934 ). Before input providing handwriting on the user interface is detected, presenting a virtual shadow change based on the currently selected drawing tool indicates to the user the type and / or characteristics of the currently selected drawing tool, thereby reducing errors in interactions between input devices and / or surfaces (e.g., avoiding accidental handwriting in the user interface by an input device) and reducing the input required to correct such errors.

[0331] In some embodiments, the electronic device displays (924a) a second user interface different from the user interface via the display generation component, wherein the second user interface is a system user interface of the electronic device, such as Figure 8E For example, the second user interface is an interface accessible to the electronic device, such as an application launch user interface having multiple application icons (eg, selectable user interface objects), such as a reference to the user interface 890. Figure 4A In some embodiments, the second user interface is a system settings user interface, from which one or more system settings of the electronic device (e.g., Wi-Fi settings, display settings, cellular settings, and / or sound settings) can be changed. In some embodiment...

Claims

1. A method comprising: At an electronic device in communication with the display generating component and the one or more sensors: displaying, via the display generation component, a user interface including a first user interface object; while displaying, via the display generation component, the user interface including the first user interface object, detecting, via the one or more sensors, a corresponding object proximate to but not contacting a surface associated with the user interface; as well as In response to detecting the corresponding object proximate to but not contacting the surface: based on determining that the corresponding object proximate to the surface is an input device in communication with the electronic device and that the positioning of the input device corresponds to the first user interface object, displaying in the user interface a first selectable option that is selectable to perform a first operation associated with the first user interface object; as well as Based on determining that the corresponding object proximate to the surface is not an input device in communication with the electronic device, abandoning the display in the user interface of the first selectable option that can be selected to perform the first operation associated with the first user interface object.

2. The method according to claim 1, further comprising: In response to detecting the corresponding object near but not contacting the surface, based on determining that the corresponding object near the surface is an input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object, modifying the visual characteristics of the first user interface object to indicate that the first user interface object is selectable.

3. The method of claim 1 , wherein the first user interface object is associated with a selection of a first area of ​​the user interface and is not associated with a selection of a second area of ​​the user interface that is different from the first area of ​​the user interface, and the first user interface object is interactable to modify the area of ​​the user interface selected by the first user interface object, the method further comprising: In response to detecting the corresponding object approaching but not contacting the surface, based on determining that the corresponding object approaching the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object, a visual indication associated with one or more directions of the modification of the area of ​​the user interface selected by the first user interface object is displayed via the display generating component.

4. The method of claim 1 , wherein the first user interface object corresponds to a first portion of the user interface, and the first selectable option is associated with ceasing display of the first portion of the user interface, the method further comprising: receiving, via the one or more sensors, one or more inputs corresponding to a selection of the first selectable option while displaying the first selectable option based on the determination that the corresponding object proximate the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object; as well as In response to receiving the one or more inputs corresponding to the selection of the first selectable option, ceasing to display the first portion of the user interface.

5. The method of claim 1 , wherein the first user interface object comprises a content input area, the content input area comprising content, and the first selectable option is associated with stopping display of the content in the content input area, the method further comprising: receiving, via the one or more sensors, one or more inputs corresponding to a selection of the first selectable option while displaying the first selectable option based on the determination that the corresponding object proximate the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object; as well as In response to receiving the one or more inputs corresponding to the selection of the first selectable option, display of the content within the content input area is ceased.

6. The method of claim 1 , wherein the first user interface object is associated with presenting media content and the first selectable option is associated with modifying playback of the media content, the method further comprising: receiving, via the one or more sensors, one or more inputs corresponding to a selection of the first selectable option while displaying the first selectable option based on the determination that the corresponding object proximate the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object; as well as In response to receiving the one or more inputs corresponding to the selection of the first selectable option, playback of the media content is modified.

7. The method according to claim 1, further comprising: In response to detecting the corresponding object proximate to but not contacting the surface, based on the determination that the corresponding object proximate to the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object: Based on determining that the positioning of the input device satisfies one or more criteria, Information associated with the first user interface object is displayed via the display generation component, the one or more criteria including a criterion satisfied when the positioning of the input device corresponds to the first user interface object for longer than a threshold amount of time.

8. The method of claim 1 , wherein the first user interface object comprises a content input area, the method further comprising: in response to detecting the corresponding object proximate to but not contacting the surface, displaying, via the display generation component, a visual indication of a content insertion cursor at a location in the content input area corresponding to the input device based on the determination that the corresponding object proximate to the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the first user interface object; while displaying the visual indication, receiving, via the one or more sensors, one or more inputs corresponding to a selection of the visual indication; as well as In response to receiving the one or more inputs corresponding to the selection of the visual indication, the content insertion cursor is displayed via the display generation component at the position corresponding to the input device in the content input area.

9. The method of claim 1 , wherein the first user interface object comprises content, the method further comprising: In response to detecting the corresponding object proximate to but not contacting the surface, according to the determination that the corresponding object proximate to the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the content: Based on determining that the content is non-editable content, a visual indication of a content selection cursor is displayed via the display generation component at a location in the content corresponding to the input device.

10. The method according to claim 9, further comprising: In response to detecting the corresponding object proximate to but not contacting the surface, according to the determination that the corresponding object proximate to the surface is the input device in communication with the electronic device and the positioning of the input device corresponds to the content: Based on determining that the content is editable content, the visual indication of displaying the content selection cursor at the position in the content corresponding to the input device via the display generation component is foregone.

11. The method of claim 1 , wherein before detecting the corresponding object proximate to but not contacting the surface, displaying the first user interface object at a first amount of separation from a backplate, the method further comprising: In response to detecting the corresponding object approaching but not contacting the surface, based on determining that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed with a second separation amount from the back panel that is greater than the first separation amount.

12. The method according to claim 1, further comprising: while displaying, via the display generation component, the user interface including the first user interface object, wherein the first user interface object has a first visual appearance in which a first visual characteristic has a first value, detecting, via a cursor control input device, a first input corresponding to a cursor moving from a location remote from the first user interface object to the first user interface object; as well as In response to detecting the first input, moving the cursor to the first user interface object and displaying the first user interface object in a second visual appearance in which the first visual characteristic has a second value different from the first value, Wherein when the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed in a third visual appearance, in which the first visual characteristic has the second value.

13. The method according to claim 12, further comprising: In response to detecting the first input, displaying the first user interface object with a parallax effect based on movement of the cursor when the cursor is located over the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the parallax effect based on movement of the input device when the positioning of the input device corresponds to the first user interface object.

14. The method according to claim 12, further comprising: In response to detecting the first input, displaying the first user interface object with a lighting effect based on movement of the cursor when the cursor is located at the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the lighting effect based on movement of the input device when the positioning of the input device corresponds to the first user interface object.

15. The method of claim 1 , wherein the first user interface object corresponds to a link to content, the method further comprising: In response to detecting the corresponding object approaching but not contacting the surface, modifying the visual appearance of the first user interface object based on determining that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object.

16. An electronic device, comprising: one or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 15.

17. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 15.

Images