Interaction between input devices and electronic devices

Abstract

The present disclosure relates to interactions between input devices and electronic devices. Some embodiments described in the present disclosure relate to displaying additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device. Some embodiments described in the present disclosure relate to providing feedback about the pose of an input device relative to a surface. Some embodiments of the present disclosure relate to performing contextual actions in response to input provided from an input device. Some embodiments of the present disclosure relate to providing handwriting input using an input device for conversion to font-based text.

Description

[0001] This application is a divisional application of the invention patent application filed on May 10, 2023, with application number 202380051163.X and title "Interaction between an input device and an electronic device".

[0002] Cross-reference to related applications

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 364,488, filed May 10, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] This invention relates generally to electronic devices that interact with input devices, and to user interaction with such devices. Background Technology

[0005] In recent years, user interaction with electronic devices has increased significantly. These devices can include computers, tablets, televisions, multimedia devices, mobile devices, and so on.

[0006] In some situations, users prefer to interact with electronic devices that have input devices such as styluses. Enhancing these interactions can improve the user experience and reduce user interaction time, which is especially important when the input device is battery powered.

[0007] As is widely recognized, the use of personally identifiable information should comply with privacy policies and practices that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users. Summary of the Invention

[0008] Some embodiments described in this disclosure involve displaying additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device. Some embodiments described in this disclosure involve providing feedback on the gesture of the input device relative to a surface. Some embodiments of this disclosure involve performing contextual actions in response to input provided from the input device. Some embodiments of this disclosure involve using an input device to provide handwriting input for conversion into font-based text. Attached Figure Description

[0009] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, in which similar reference numerals indicate corresponding parts throughout the drawings.

[0010] Figure 1AThis is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some implementation schemes.

[0011] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some implementation schemes.

[0012] Figure 2 Examples of portable multi-functional devices with touchscreens according to some implementation schemes are shown.

[0013] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some implementation schemes.

[0014] Figure 4A An exemplary user interface for applying menus on a portable multifunction device according to some implementations is shown.

[0015] Figure 4B An exemplary user interface of a multifunctional device having a touch-sensitive surface separate from the display is illustrated according to some embodiments.

[0016] Figure 5A Examples of personal electronic devices according to some implementation schemes are shown.

[0017] Figure 5B This is a block diagram illustrating a personal electronic device according to some implementation schemes.

[0018] Figures 5C to 5D Exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor according to some embodiments are illustrated.

[0019] Figures 5E to 5H Exemplary components and user interfaces of a personal electronic device according to some implementation schemes are illustrated.

[0020] Figure 5I A block diagram illustrating an exemplary architecture for a device according to some embodiments of the present disclosure is shown.

[0021] Figures 6A to 6BF Examples of ways in which an electronic device displays additional controls and / or information when an input device, such as a stylus, is hovered over a user interface displayed by the electronic device are illustrated, according to some implementations.

[0022] Figures 7A to 7G This is a flowchart illustrating, according to some implementations, a method for displaying additional controls and / or information when an input device, such as a stylus, is hovered over a user interface displayed by an electronic device.

[0023] Figures 8A to 8AFExemplary ways in which an electronic device, according to some embodiments, provides feedback on the posture of an input device relative to a surface are illustrated.

[0024] Figures 9A to 9K This is a flowchart illustrating a method for providing feedback on the posture of an input device relative to a surface, according to some implementation schemes.

[0025] Figures 10A to 10AP Exemplary ways in which an electronic device, according to some implementations, performs contextual actions in response to input provided from an input device are illustrated.

[0026] Figures 11A to 11H This is a flowchart illustrating a method for performing a contextual action in response to input provided from an input device, according to some implementation schemes.

[0027] Figures 12A to 12AT An exemplary manner in which an electronic device, according to some implementations, uses an input device to provide handwriting input for conversion into font-based text is illustrated.

[0028] Figures 13A to 13K This is a flowchart illustrating a method for providing handwritten input using an input device for conversion into font-based text, according to some implementation schemes. Detailed Implementation

[0029] The following description illustrates exemplary methods, parameters, etc. However, it should be understood that such description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.

[0030] There is a need for electronic devices that provide efficient methods for interaction between electronic devices and input devices (e.g., from a stylus or other input device). Such technologies can reduce the cognitive burden on users of such devices. Furthermore, such technologies can reduce processor and battery power that would otherwise be wasted on redundant user input.

[0031] Although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch may be named a second touch and similarly, a second touch may be named a first touch, without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0032] The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “comprising” and / or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] Depending on the context, the term "if" may optionally be interpreted as meaning "when," "in response to," or "in response to detection." Similarly, depending on the context, the phrases "if it is determined..." or "if [the stated condition or event] is detected" may optionally be interpreted as meaning "in response to determining..." or "in response to detecting [the stated condition or event]."

[0034] This document describes implementations of electronic devices, user interfaces for such devices, and associated processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functionalities such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Equipment and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0035] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0036] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, gaming applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0037] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0038] Now let’s turn our attention to implementation schemes for portable devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and is sometimes referred to as or called a “touch-sensitive display system.” Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (e.g., generating haptic output on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0039] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute (alternative) for the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface near the contact, and / or the resistance and / or variation of the touch-sensitive surface near the contact may optionally be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement) has been exceeded. In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure) has been exceeded. Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited physical space, which is used (e.g., on a touch-sensitive display) to display an indication and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0040] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the tactile surface remains unchanged, movement of the tactile surface can optionally be interpreted or perceived by the user as the “roughness” of the tactile surface. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of the user (e.g., “press click,” “release click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or common) user.

[0041] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits.

[0042] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0043] Peripheral interface 118 can be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0044] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via short-range communication radio components. Wireless communication may optionally employ any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution, Pure Data (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE...). 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)) and / or Short Message Service (SMS), or any other suitable communication protocol that has not been developed as of the date of this document submission.

[0045] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and sends the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and sends the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or sent to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a single-ear or dual-ear headphone) and input (e.g., a microphone).

[0046] I / O subsystem 106 couples input / output peripherals on device 100, such as touchscreen 112 and other input control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / transmit electrical signals to the other input control device 116. The other input control devices 116 optionally include physical buttons (e.g., push-buttons and / or rocker buttons), dial pads, slide switches, joysticks, click wheels, etc. In some alternative embodiments, input controller 160 is optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing device such as mouse. One or more buttons (e.g., Figure 2 Optionally, 208) includes an increase / decrease button for volume control of speaker 111 and / or microphone 113. The one or more buttons optionally include a push-button (e.g., Figure 2 (206 in the middle).

[0047] A quick press of the push button optionally disengages the touchscreen 112 from its lock or optionally initiates a process of unlocking the device using gestures on the touchscreen, as described in U.S. Patent Application 11 / 322,549 (i.e., U.S. Patent No. 7,657,849), filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," the entire contents of which are incorporated herein by reference. A long press of the push button (e.g., 206) optionally powers the device 100 on or off. The function of one or more buttons is optionally user-customizable. The touchscreen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0048] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0049] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In an exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.

[0050] Touchscreen 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. Touchscreen 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that from Apple Inc. (Cupertino, California). and iPod The technology used.

[0051] In some embodiments of the touchscreen 112, the touch-sensitive display optionally resembles a multi-touch touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touchscreen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.

[0052] The touch-sensitive display in some embodiments of the touchscreen 112 is described in the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, “Multipoint Touchscreen”; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. Patent Application No. 11 / 48,264, filed January 31, 2005, “Gestures For Touch Sensitive Input Devices”; and (5) U.S. Patent Application No. 11 / 38,590, filed January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input”. (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, “Virtual Input Device Placement On ATouch Screen User Interface”; (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, “Operation Of A Computer With ATouch Screen Interface”; (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, “Multi-Functional Hand-Held Device”. The full text of all these applications is incorporated herein by reference.

[0053] Touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. Users optionally use any suitable object or accessory such as a stylus, finger, etc., to interact with touchscreen 112. In some embodiments, the user interface is designed to operate primarily through finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positioning or commands to perform the user-desired actions.

[0054] In some embodiments, device 100 is a portable computing system that communicates with a display generating component (e.g., via wireless or wired communication). The display generating component is configured to provide visual output, such as a display via a CRT monitor, a display via an LED monitor, or a display via image projection. In some embodiments, the display generating component is integrated with the computer system (e.g., an integrated display and / or touchscreen 112). In some embodiments, the display generating component is separate from the computer system (e.g., an external monitor and / or projection system). As used herein, “display” content includes displaying content (e.g., video data rendered or decoded by display controller 156) by sending data (e.g., image data or video data) to the integrated or external display generating component via a wired or wireless connection to visually generate content.

[0055] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0056] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0057] The device 100 may optionally also include one or more optical sensors 164. Figure 1AAn optical sensor 164 is shown coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, facing away from a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing images of the user to be optionally acquired for video conferencing while the user views other video conferencing participants on the touchscreen display. In some embodiments, the positioning of the optical sensor 164 can be changed by the user (e.g., by rotating the lenses and sensors within the device housing), allowing a single optical sensor 164 to be used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.

[0058] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor is shown coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100, opposite to the touchscreen display 112 located on the front of device 100.

[0059] The device 100 optionally also includes one or more proximity sensors 166. Figure 1AA proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 may optionally be coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 may optionally perform as described in the following U.S. patent applications: 11 / 241,839, entitled "Proximity Detector In Handheld Device"; 11 / 240,788, entitled "Proximity Detector In Handheld Device"; 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, the proximity sensor is turned off and the touchscreen 112 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0060] The device 100 may optionally also include one or more tactile output generators 167. Figure 1A A haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.

[0061] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may be coupled to an input controller 160 in an I / O subsystem 106. The accelerometer 168 may optionally perform as described in the following U.S. Patent Publications: 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and 20060017692, entitled "Methods And Apparatuses For Operating A Portable DeviceBased On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 100 may optionally include, in addition to the accelerometer 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information about the position and orientation (e.g., portrait or landscape) of device 100.

[0062] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, memory 102 ( Figure 1A ) or 370 ( Figure 3 Storage device / global internal state 157, such as Figure 1A and Figure 3 As shown in the figure. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, indicating what applications, views or other information occupy various areas of the touch screen display 112; sensor state, including information obtained from various sensors and input control devices 116 of the device; and position information relating to the device's position and / or orientation.

[0063] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, and / or power management) and facilitates communication between various hardware and software components.

[0064] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), and / or FireWire) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, and / or a wireless LAN). In some embodiments, the external port is connected to… (Trademark of Apple Inc.) The same or similar and / or compatible multi-pin (e.g., 30-pin) connectors used in Apple Inc. devices.

[0065] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., touchpads or physical click-based rotary dials). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disengagement). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to single-point contact (e.g., single-finger contact) or multi-point simultaneous contact (e.g., "multi-touch" / multiple-finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0066] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen can be set to any of a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0067] The touch / motion module 130 optionally detects gesture input performed by the user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0068] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0069] In some implementations, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives from the application one or more codes specifying the graphics to be displayed, and, if necessary, also receives coordinate data and other graphic attribute data, and then generates screen image data for output to the display controller 156.

[0070] The haptic feedback module 133 includes various software components for generating instructions which are used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0071] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0072] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to telephone 138 for use in location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services, such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0073] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0074] ●Contacts module 137 (sometimes called address book or contact list);

[0075] ● Telephone module 138;

[0076] ●Video conferencing module 139;

[0077] ●Email client module 140;

[0078] ●Instant Messaging (IM) module 141;

[0079] ● Fitness support module 142;

[0080] ● Camera module 143 for still images and / or video images;

[0081] ●Image Management Module 144;

[0082] ●Video player module;

[0083] ●Music player module;

[0084] ● Browser module 147;

[0085] ● Calendar module 148;

[0086] ● Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5 and other widgets acquired by the user, as well as user-created widgets 149-6;

[0087] ● Wrapper module 150 for creating user-created widgets 149-6;

[0088] ●Search module 151;

[0089] ● Video and music player module 152, which combines a video player module and a music player module;

[0090] ●Notes module 153;

[0091] ●Map module 154; and / or

[0092] ● Online video module 155.

[0093] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0094] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., in application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0095] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 137, modify entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0096] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0097] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0098] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving an instant message, and viewing received instant messages. In some embodiments, the sent and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages delivered using SMS or MMS) and internet-based messages (e.g., messages delivered using XMPP, SIMPLE, or IMPS).

[0099] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes executable instructions for creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (executive devices); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing music for fitness; and displaying, storing, and transmitting fitness data.

[0100] In conjunction with the touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0101] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0102] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet according to user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as links to attachments and other files on web pages.

[0103] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and associated data (e.g., calendar entries and / or to-dos) according to user instructions.

[0104] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is optionally a micro-application downloaded and used by a user (e.g., weather widget 149-1, stock market widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or a user-created micro-application (e.g., user-created widget 149-6). In some embodiments, the widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! widgets).

[0105] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, widget creator module 150 can optionally be used by the user to create widgets (e.g., to convert user-specified portions of a webpage into widgets).

[0106] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0107] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0108] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the note-taking module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0109] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data related to shops and other points of interest at or near a specific location, and other location-based data) according to user instructions.

[0110] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions for: allowing users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Further descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,67, filed December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.

[0111] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more functions described above and the methods described in this patent application (e.g., computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. For example, a video player module may optionally be combined with a music player module into a single module (e.g., Figure 1A (e.g., video and music player module 152). In some embodiments, memory 102 optionally stores subgroups of the aforementioned modules and data structures. Additionally, memory 102 optionally stores other modules and data structures not described above.

[0112] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on device 100 can be optionally reduced.

[0113] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main menu, home menu, or root menu. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0114] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0115] Event classifier 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0116] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0117] Event monitor 171 receives event information from peripheral interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.

[0118] In some implementations, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 sends event information. In other implementations, peripheral device interface 118 sends event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0119] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0120] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0121] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates the touch-based gesture.

[0122] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.

[0123] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.

[0124] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0125] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.

[0126] In some implementations, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other implementations, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some implementations, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0127] The corresponding event identifier 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event based on the event information. The event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event identifier 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0128] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a longitudinal orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).

[0129] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 112, and lifting off the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0130] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0131] In some implementations, the definition of the corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0132] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0133] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0134] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from delivering (and deferred delivering) the sub-events to the corresponding hit view. In some implementations, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag retrieves the flag and executes a predefined procedure.

[0135] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined procedure.

[0136] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positioning of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and transmits the display information to graphics module 132 for display on touch-sensitive display.

[0137] In some implementations, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.

[0138] It should be understood that the above discussion regarding event handling of user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, and / or scrolling; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0139] Figure 2A portable multifunction device 100 with a touchscreen 112 is illustrated according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some specific embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0140] In some embodiments, the stylus 203 is an active device and includes one or more electronic circuits. For example, the stylus 203 includes one or more sensors and one or more communication circuits (such as communication module 128 and / or RF circuitry 108). In some embodiments, the stylus 203 includes one or more processors and a power system (e.g., similar to power system 162). In some embodiments, the stylus 203 includes an accelerometer (such as accelerometer 168), a magnetometer, and / or a gyroscope capable of determining the positioning, angle, location, and / or other physical characteristics of the stylus 203 (e.g., whether the stylus is placed down, tilted toward or away from the device, and / or close to or away from the device). In some embodiments, the stylus 203 communicates with an electronic device (e.g., via communication circuitry, through a wireless communication protocol such as Bluetooth) and transmits sensor data to the electronic device. In some embodiments, the stylus 203 is capable of determining (e.g., via an accelerometer or other sensors) whether a user is holding the device. In some implementations, the stylus 203 may accept tapping input (e.g., single or double tap) from a user (e.g., received by an accelerometer or other sensor) on the stylus 203 and interpret the input as a command or request to perform a function or change to a different input mode.

[0141] Device 100 optionally also includes one or more physical buttons, such as a "main desktop" or menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.

[0142] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic outputs for a user of device 100.

[0143] Figure 3 This is a block diagram of an exemplary multi-functional device with a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the reference above). Figure 1A The described tactile output generator 167) and sensor 359 (e.g., optical sensor, accelerometer, proximity sensor, touch sensor and / or contact intensity sensor (similar to the one described above)) Figure 1AThe described contact strength sensor 165). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores information related to portable multifunction device 100. Figure 1A The memory 370 stores programs, modules, and data structures similar to those in the memory 102 of the portable multifunction device 100, or subsets thereof. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 (… Figure 1A The memory 102 may optionally not store these modules.

[0144] Figure 3 Each of the elements described above is optionally stored in one or more memory devices of the previously mentioned memory devices. Each of the modules described above corresponds to a set of instructions for performing the functions described above. The modules or programs (e.g., instruction sets) described above need not be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subgroup of the modules and data structures described above. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0145] Now let’s turn our attention to the implementation of the user interface, which is optionally implemented on, for example, a portable multifunction device 100.

[0146] Figure 4A An exemplary user interface for an application menu on a portable multifunction device 100 according to some embodiments is illustrated. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes elements or a subset or superset thereof:

[0147] ●Signal strength indicator 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0148] ●Time 404;

[0149] ●Bluetooth indicator 405;

[0150] ● Battery status indicator 406;

[0151] ● Tray tray 408 features icons for frequently used applications, such as:

[0152] ○ The telephone module 138 has an icon 416 labeled "telephone", which optionally includes an indicator 414 indicating the number of missed calls or voicemail messages;

[0153] ○ An icon 418 labeled "Mail" in the email client module 140, which optionally includes an indicator 410 for the number of unread emails;

[0154] ○ The icon 420 labeled "Browser" in browser module 147; and

[0155] ○ The icon 422 labeled "iPod" for the video and music player module 152 (also known as the iPod (a trademark of Apple Inc.) module 152); and

[0156] ● Icons of other applications, such as:

[0157] ○The icon 424 of the IM module 141 marked as "Message";

[0158] ○The calendar module 148 has an icon 426 labeled "Calendar";

[0159] ○ The icon 428 of the image management module 144 is labeled "Photo".

[0160] ○ The icon 430 of camera module 143, which is labeled "camera";

[0161] ○ The icon 432 of the online video module 155, which is labeled "Online Video";

[0162] ○ The icon 434 labeled "Stock Market" in the Stock Market widget 149-2;

[0163] ○The icon 436 of the map module 154 that is labeled "map";

[0164] ○The weather widget 149-1 has icon 438 labeled "weather";

[0165] ○ The alarm clock widget 149-4 has an icon 440 labeled "clock";

[0166] ○ The icon 442 of the fitness support module 142 is labeled "fitness support";

[0167] ○ The icon 444 labeled "Notes" in Notes module 153; and

[0168] ○ The icon 446, labeled "Settings," is used to set up an application or module that provides access to settings for the device 100 and its various applications 136.

[0169] It should be pointed out that, Figure 4A The illustrated icon labels are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels may be optionally used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0170] Figure 4B An example is illustrated having a touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)). Figure 3 A tablet device or touchpad 355) device (e.g., Figure 3 An exemplary user interface on the device 300. The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 359) for detecting the intensity of contact on the tactile surface 451 and / or one or more tactile output generators 357 for generating tactile outputs for the user of the device 300.

[0171] While some examples of input on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given below, in some implementations the device detects input on a touch-sensitive surface separate from the display, such as... Figure 4B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a spindle (e.g., on the display (e.g., 450)). Figure 4B The spindle corresponding to 453 in the middle (e.g., Figure 4B (452 in the example). According to these embodiments, the device detects the position corresponding to the corresponding position on the display (e.g., in the example). Figure 4B In the middle, 460 corresponds to 468 and 462 corresponds to 470) is in contact with the touch-sensitive surface 451 (e.g., Figure 4B (460 and 462 in the text). Thus, when the touch-sensitive surface (e.g., ...) Figure 4B 451) and the display of a multi-functional device (e.g., Figure 4B When 450 is separated from the touch-sensitive surface, user input detected by the device on that touch-sensitive surface (e.g., touches 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may optionally be used for other user interfaces described herein.

[0172] Additionally, while the examples below are given primarily with reference to finger input (e.g., finger touch, single-finger tap gesture, finger swipe gesture), it should be understood that in some implementations, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). As another example, when the cursor is at the location of a tap gesture, the tap gesture is optionally replaced by a mouse click (e.g., instead of detecting touch and then stopping contact detection). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mouse and finger touch are optionally used simultaneously.

[0173] Figure 5A An exemplary personal electronic device 500 is illustrated. Device 500 includes a body 502. In some embodiments, device 500 may include components relative to devices 100 and 300 (e.g., Figures 1A to 4B The device 500 may include some or all of the features described herein. In some embodiments, the device 500 has a touch-sensitive display 504, referred to below as a touchscreen 504. Alternatively, or in addition to the touchscreen 504, the device 500 may also have a display and a touch-sensitive surface. Similar to the cases of devices 100 and 300, in some embodiments, the touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., a touch). The one or more intensity sensors of the touchscreen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touch. The user interface of the device 500 may respond to touches based on the intensity of the touch, meaning that touches of different intensities may invoke different user interface operations on the device 500.

[0174] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, filed May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” published under WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” published under WIPO Patent Publication No. WO / 2014 / 105276, each of which is incorporated herein by reference in its entirety.

[0175] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) may be physical. Examples of physical input mechanisms include push-buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watch straps, bangles, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow a user to wear device 500.

[0176] Figure 5B An exemplary personal electronic device 500 is depicted. In some embodiments, device 500 may include, relative to... Figure 1A , Figure 1B and Figure 3Some or all of the components described. Device 500 has a bus 512 that operatively couples I / O portion 514 to one or more computer processors 516 and memory 518. I / O portion 514 may be connected to display 504, which may have touch-sensitive component 522 and optionally have intensity sensor 524 (e.g., contact intensity sensor). Furthermore, I / O portion 514 may be connected to communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 may optionally be a rotatable input device or a pressable input device and a rotatable input device. In some examples, input mechanism 508 may optionally be a button.

[0177] In some examples, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to the I / O section 514.

[0178] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, may cause the computer processors to perform, for example, the techniques described below, including processes 700, 900, 1100, and 1300 (Figures 7, 9, 11, and 13). The computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, and device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. The personal electronic device 500 is not limited to... Figure 5B It can be the components and configurations, or it can include other components or additional components in a variety of configurations.

[0179] Furthermore, in methods described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the method may be repeated in multiple repetitions such that, during the repetitions, all conditions determining the steps in the method are satisfied in different repetitions of the method. For example, if a method requires performing a first step (if a condition is satisfied) and a second step (if a condition is not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions determining the steps in the method are satisfied. Those skilled in the art will also understand that, similar to methods having discretionary steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.

[0180] As used herein, the term "power indication" refers optionally to the power indication in devices 100, 300, and / or 500 ( Figure 1A , Figure 3 and Figures 5A to 5B A user-interactive graphical user interface object displayed on a screen. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) optionally each constitute a functional representation.

[0181] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other positional marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3 The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of enabling direct interaction with user interface elements on the touchscreen display) Figure 1A The touch-sensitive display system 112 or Figure 4AIn some embodiments of the touchscreen 112, a touch detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that delivers the user-expected interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0182] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted away, before or after contact begins to move, before contact ends, before or after contact intensity is detected to increase and / or before or after contact intensity decreases). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some implementations, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the feature intensity and one or more thresholds is used to determine whether to perform one or more actions (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.

[0183] Figure 5C An example is shown using multiple intensity sensors 524A-524D to detect multiple contacts 552A-552E on a touch-sensitive display 504. Figure 5C It also includes an intensity graph, which shows the current intensity measurements of intensity sensors 524A-524D relative to intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are both 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are both 7 intensity units. In some embodiments, the cumulative intensity is the sum of the intensity measurements of the multiple intensity sensors 524A-524D, which is 32 intensity units in this example. In some implementations, each contact is assigned a corresponding intensity, i.e., a portion of the cumulative intensity. Figure 5DAn example is shown of assigning cumulative intensity to contacts 552A-552E based on their distance from the center of force 554. In this example, each of contacts 552A, 552B, and 552E is assigned a contact intensity of 8 intensity units of the cumulative intensity, and each of contacts 552C and 552D is assigned a contact intensity of 4 intensity units of the cumulative intensity. More generally, in some specific implementations, each contact j is assigned a corresponding intensity Ij according to a predefined mathematical function Ij = A·(Dj / ΣDi), which is a portion of the cumulative intensity A, where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i = 1 to the last) from the center of force. The reference can be performed using electronic equipment similar to or equivalent to devices 100, 300, or 500. Figures 5C to 5D The operation described above. In some embodiments, the characteristic intensity of the contact is based on one or more intensities of the contact. In some embodiments, an intensity sensor is used to determine the intensity of a single characteristic (e.g., a single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included within... Figures 5C to 5D This is intended to assist readers.

[0184] In some implementations, a portion of the gesture is identified for determining the characteristic intensity. For example, the touch-sensitive surface optionally receives a series of swipe contacts that transition from a starting position to an ending position, where the contact intensity increases. In this example, the characteristic intensity of the contact at the ending position is optionally based only on a portion of the series of swipe contacts, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the ending position). In some implementations, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: unweighted moving average smoothing algorithm, triangular smoothing algorithm, median filter smoothing algorithm, and / or exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic intensity.

[0185] Optionally, the contact intensity on a touch-sensitive surface is characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device performs an operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device performs an operation different from the operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity lower than the light press intensity threshold (e.g., and higher than the nominal contact detection intensity threshold, where contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector based on the movement of the contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.

[0186] An increase in contact intensity from below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact intensity from below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact intensity from below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact intensity from above a contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact being lifted off the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0187] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., a "downward stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing to above a press input intensity threshold and the intensity of the contact subsequently decreasing to below a press input intensity threshold, and the corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below a press input threshold (e.g., an "upward stroke" of the corresponding press input).

[0188] Figures 5E to 5HAn example of gesture detection is given, where the gesture includes the intensity of contact 562 ranging from less than... Figure 5E The light press intensity threshold (e.g., "IT") L The intensity of ”) increases to higher than Figure 5H The deep press intensity threshold (e.g., "IT") D The intensity of the press input corresponds to the strength of the press. On the user interface 570, which includes application icons 572A-572D displayed in the predefined area 574, when the cursor 576 is displayed above the application icon 572B corresponding to application 2, the gesture performed by contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch-sensitive display 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines the intensity of the contact 562 within a deep press intensity threshold (e.g., "IT"). D The intensity reaches its peak value above 562. Contact is maintained on the touch-sensitive surface 560. In response to a detected gesture, and based on the intensity increasing to a deep press intensity threshold (e.g., "IT") during the gesture... D The above contact 562 displays the scale representation 578A-578C (e.g., thumbnails) of the document recently opened for application 2, such as... Figures 5F to 5I As shown. In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in... Figures 5E to 5H This is intended to assist readers.

[0189] In some implementations, the display of icons 578A-578C includes animation. For example, icon 578A might initially appear near application icon 572B, such as... Figure 5F As shown, as the animation progresses, icon 578A moves upwards, and icon 578B appears near application icon 572B, as shown. Figure 5G As shown in the diagram. Then, representation 578A moves upward, 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown. Figure 5H As shown in the diagram. This indicates that 578A-578C form an array above icon 572B. In some implementations, the animation progresses according to the intensity of contact 562, such as... Figures 5F to 5G As shown, where 578A-578C appear and increase with the intensity of contact 562 towards the pressure threshold (e.g., "IT"). D The animation progresses upwards as it increases in intensity. In some implementations, the intensity upon which the animation progresses is based is the characteristic intensity of the contact. Reference can be performed using electronic devices similar to or equivalent to devices 100, 300, or 500. Figures 5E to 5H The aforementioned operation.

[0190] Figure 5I A block diagram illustrating an exemplary architecture for device 580 according to some embodiments of this disclosure is shown. Figure 5I In some implementations, media content or other content is optionally received by device 580 via network interface 582, which is optionally a wireless or wired connection. The one or more processors 584 optionally execute any number of programs stored in memory 586 or a storage device, the programs optionally including instructions for performing one or more of the methods and / or processes described herein (e.g., methods 700, 900, 1100, 1300, 1500, and 1700).

[0191] In some embodiments, the display controller 588 causes various user interfaces of this disclosure to be displayed on the display 594. Additionally, input to the device 580 is optionally provided by a remote controller 590 via a remote controller interface 592, which is optionally a wireless or wired connection. In some embodiments, input to the device 580 is provided by a multifunction device 591 (e.g., a smartphone) running a remote control application that configures the multifunction device to simulate remote control functionality, as will be described in more detail below. In some embodiments, the multifunction device 591 corresponds to... Figure 1A and Figure 2 Equipment 100 Figure 3 Equipment 300 and Figure 5A One or more of the 500 devices mentioned. It should be understood that... Figure 5I The implementation scheme does not imply limitation on the features of the device described herein, and other components that are advantageous to other features described herein may also be optionally included. Figure 5I In the architecture. In some implementations, device 580 optionally corresponds to Figure 1A and Figure 2 Multifunctional device 100 Figure 3 Equipment 300 and Figure 5A One or more of the devices 500; network interface 582 optionally corresponds to Figure 1A and Figure 2 The RF circuit 108, external port 124, and peripheral device interface 118 are included. Figure 3 One or more of the network communication interfaces 360; the processor 584 optionally corresponds to Figure 1A The processor 120 and Figure 3 One or more of the CPUs 310; the display controller 588 optionally corresponds to Figure 1A The display controller 156 and Figure 3 One or more of the I / O interfaces 330; the memory 586 optionally corresponds to Figure 1AThe memory 102 and Figure 3 One or more of the memory 370 in the memory; the remote control interface 592 optionally corresponds to Figure 1A The peripheral device interface 118 and I / O subsystem 106 (and / or its components) and Figure 3 One or more of the I / O interfaces 330; the remote control 590 optionally corresponds to and / or includes Figure 1A The speaker 111, touch-sensitive display system 112, microphone 113, optical sensor 164, contact intensity sensor 165, tactile output generator 167, other input control devices 116, accelerometer 168, proximity sensor 166, and I / O subsystem 106 are included. Figure 3 The keyboard / mouse 350, touchpad 355, haptic output generator 357, and contact intensity sensor 359 are one or more of the touch-sensitive surface 451 in FIG. 4; and the display 594 optionally corresponds to Figure 1A and Figure 2 The touch-sensitive display system 112 and Figure 3 One or more of the 340 displays.

[0192] In some implementations, the device employs intensity hysteresis to avoid unintended inputs sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Therefore, in some implementations, a press input includes an increase in the intensity of the corresponding contact above the press input intensity threshold and a subsequent decrease in the intensity of that contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some implementations, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and optionally the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity threshold, and a corresponding operation is performed in response to detecting a press input (e.g., an increase or decrease in contact intensity depending on the environment).

[0193] For ease of explanation, optionally, the description of an operation triggered in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is provided in response to detecting any of the following conditions: the contact strength increases to above the press input strength threshold, the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold, the contact strength decreases to below the press input strength threshold, and / or the contact strength decreases to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example where the operation is described as being performed in response to detecting a decrease in contact strength below the press input strength threshold, the operation is optionally performed in response to detecting a decrease in contact strength below a hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0194] As used herein, "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched on the device (e.g., become open). In some implementations, the downloaded application becomes an installed application using an installer that extracts program portions from the downloaded software package and integrates the extracted portions with the computer system's operating system.

[0195] As used herein, the terms "open application" or "running application" refer to a software application that maintains state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or running application is optionally any of the following types of applications:

[0196] ●The active application currently displayed on the screen of the device that is using the application;

[0197] ● Background applications (or background processes) that are not currently displayed but whose one or more processes are being handled by one or more processors; and

[0198] ●Not running but stored in memory (either volatile or non-volatile).

[0199] It can also be used to resume the execution of paused or hibernating applications by providing state information.

[0200] As used herein, the term "closed application" refers to a software application that does not retain state information (e.g., the state information of a closed application is not stored in the device's memory). Therefore, closing an application includes stopping and / or removing the application's process and removing the application's state information from the device's memory. Generally, while the first application is in use, opening the second application does not close the first application. When the second application is displayed and the first application stops displaying, the first application becomes a background application.

[0201] Now let’s turn our attention to the implementation of user interfaces (“UIs”) and associated processes on electronic devices (such as portable multifunction devices 100, 300 or 500).

[0202] User interface and related processes

[0203] Hover events and controls

[0204] Users interact with electronic devices in many different ways, including using peripheral devices that communicate with these devices. In some implementations, the electronic device receives indications that a peripheral device (e.g., a stylus) approaches but does not touch a surface (such as a touch-sensitive surface that communicates with the electronic device). The implementations described herein provide ways for the electronic device to respond to such indications and enhance interaction with the device, for example, by providing a visual preview or other indication of interaction with the electronic device based on the current positioning of the input device relative to the surface. Enhanced interaction with the device reduces the amount of time required for the user to perform actions, thereby reducing the device's power consumption and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, that person is optionally referred to as the user of the device.

[0205] Figures 6A to 6BF Exemplary methods are illustrated for an electronic device according to some embodiments to display selectable options and / or information in response to detecting that an input device is hovering over a surface associated with the electronic device. The embodiments in these figures are used to illustrate the processes described below, including references to... Figures 7A to 7G The process described.

[0206] Figure 6A An example is illustrated where electronic device 500 is displaying user interface 609 (e.g., via a display device and / or via a display generating component). In some embodiments, user interface 609 is displayed via a display generating component. In some embodiments, the display generating component is a hardware component (e.g., including electronic components) capable of receiving display data and displaying the user interface. In some embodiments, examples of the display generating component include a touchscreen display (e.g., touchscreen 504), a monitor, a television, a projector, an integrated, discrete, or external display device, or any other suitable display device communicating with device 500. In some examples, a surface (e.g., a touch-sensitive surface) communicates with device 500. For example, in Figure 6A In this device 500, a touchscreen 504 is included, which displays a user interface and detects touch or hover interactions with the device 500.

[0207] In some implementations, user interface 609 is the user interface of an application or a user interface in which media browsing, input, and interaction (e.g., for creating drawings, viewing drawings, modifying font-based text and / or handwritten text and / or interacting with it, navigating content such as web-based content, and / or interacting with media content) are performed. In some implementations, the application is an application installed on device 500.

[0208] exist Figure 6A In this embodiment, user interface 609 includes elements for media browsing and interaction. In some embodiments, device 500 communicates with an input device such as stylus 600. In some embodiments, device 500 is configured to receive indications of contact between stylus 600 and a surface such as touchscreen 504. In some embodiments, device 500 and / or stylus 600 are further configured to send and / or receive indications of proximity between a surface (e.g., touchscreen 504) and stylus 600. For example, icon 603 includes a hover distance threshold 601. Although threshold 601 is illustrated as a line extending parallel to touchscreen 504, it should be understood that such illustration is merely exemplary and not intended to be limiting in any way. In some embodiments, a “hover event” as referred to herein includes a situation where a corresponding portion of the input device (e.g., the tip of a stylus 600) moves to a position less than a threshold distance (e.g., threshold 601, such as 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) from a surface (e.g., touchscreen 504) without touching that surface. In some embodiments, determining the positioning of a corresponding portion of the input device relative to the projection of the input device onto the surface (e.g., the vertical projection of the tip of the stylus 600) corresponding to the positioning of a user interface element (e.g., selectable options, text, and / or graphical objects) is referred to herein as the position of the input device corresponding to the user interface element (e.g., the stylus tip corresponding to the object). Furthermore, as referred to herein, displaying or modifying one or more portions of the user interface corresponding to the user interface object in response to a hover event optionally describes a hover event between the input device and the surface at a position corresponding to the user interface object in the user interface.

[0209] like Figure 6AAs shown, user interface 609 includes multiple interactive and non-interactive visual elements. For example, text 602 includes editable font-based text in a text input area (e.g., a search box). A text input cursor preview is displayed in response to a hover event, including a stylus 600 corresponding to the text input area, as described later. In some embodiments, icon 604 can be selected to initiate one or more actions, such as a search query based on text 602, and is visually emphasized in response to a hover event, as described later. In some embodiments, media player 608 is interactive to control the playback of corresponding media and is visually modified in response to a hover event, as described later. In some embodiments, link 610 can be selected to initiate the execution of one or more actions, such as the display of content on the linked webpage, and is visually emphasized in response to a hover event, as described later. In some embodiments, text 612 is non-editable text (e.g., text that is part of an image including an image of a rugby ball and an image of text 612), and a selection cursor is displayed in response to a hover event, as described later. In some implementations, the corresponding element within selectable option 614 can be selected to view the corresponding linked content, and is displayed along with additional selectable options to navigate the corresponding element in response to a hover event, as described later. User interface 609 represents a view of electronic device 500 positioned from the top (e.g., perpendicular to a plane coplanar with touchscreen 504), while icon 603 represents a view of electronic device 500 from the corresponding side of electronic device 500 (e.g., parallel or nearly parallel to a plane coplanar with touchscreen 504). It should be understood that such representations are merely exemplary for illustrative purposes to indicate hover events and interactions as described herein, and are not intended to be limiting in any way.

[0210] exist Figure 6B In this context, the stylus 600 moves to a position above the touchscreen 504, but exceeds a threshold 601, as seen in icon 603. As mentioned herein, the corresponding position of the stylus 600 exceeding the threshold 601 is described as outside a hover threshold, as described herein. Because the stylus 600 exceeds the threshold 601, the device 500 does not respond to this placement of the stylus on the touchscreen 504 to modify the user interface 609.

[0211] exist Figure 6CIn this context, the stylus 600 moves to a position within a hover threshold 601 but without touching the touchscreen 504, corresponding to the location of the search box including text 602. In response to this hover event, a text insertion preview cursor 690 is displayed. In some embodiments, while the stylus 600 remains within the hover threshold 601 and corresponds to a position within a text input area including text 602, the text insertion preview cursor 690 moves within the user interface based on the movement of the stylus 600. The text insertion preview cursor 690 optionally indicates the position in the user interface 609 where the text insertion cursor will be placed and / or positioned in response to the device 500 detecting a downward touch of the stylus pen touching the touchscreen 504. Figure 6C In the text insertion preview, cursor 690 is displayed at the end of text 602 in the search box.

[0212] exist Figure 6D As shown in the diagram 603, the stylus 600 is in... Figure 6C The text insertion preview cursor 690 is positioned to contact the touchscreen 504. In response to this contact, the display of the text insertion preview cursor 690 stops, and the text insertion cursor 692 is displayed at the end of the text 602, as shown. Figure 6D As shown. In some embodiments, the text insertion preview cursor 690 and the text insertion cursor 692 are displayed with different visual appearances (e.g., different scales, colors, opacities, shadows, borders, and / or lighting effects) to distinguish the preview from the insertion of the text insertion cursor. In some embodiments, after the text insertion cursor 692 is inserted, the stylus 600 is removed from a hover threshold (e.g., to a position beyond threshold 601). In response to the stylus 600 moving outside threshold 601, the display of the text insertion cursor 692 is maintained, as shown. Figure 6E As shown. Furthermore, in Figure 6E In this embodiment, device 500 has detected text input (e.g., from an external keyboard, a soft keyboard, and / or from voice input) and, in response, displays new text (e.g., ABC) corresponding to the text input at the location of text insertion cursor 692. In some embodiments, while displaying text insertion cursor 692, in response to stylus 600 moving to hover threshold 601, text insertion preview cursor 690 is simultaneously displayed at the location of the stylus in user interface 609.

[0213] exist Figures 6F to 6H In the process, the corresponding object moves within a threshold distance 601 of the touchscreen 504 and then contacts the touchscreen 504. Figure 6FAn example is illustrated by a hand and / or finger 605 positioned outside a hover threshold 601. The hand and / or finger 605 is positioned at the location of a search box including text 602, but more generally, device 500 does not modify the display of text 602 or user interface 609 because the hand and / or finger 605 is outside the threshold distance 601 and / or because the hand and / or finger 605 is not a stylus 600. If device 500 detects movement of the hand and / or finger 605 toward touchscreen 504 within the hover threshold 601, device 500 optionally determines that the hand 605 is not an input device (e.g., not a stylus 600) and abandons displaying the text insertion preview cursor 690 (and / or other modifications to user interface 609). However, in Figure 6H In the process, device 500 detects a contact after a corresponding part of hand 605 (e.g., a finger) touches the text 602 within the search box, and in response, device 500 inserts a text insertion cursor 692 at the end of the text 602, such as... Figure 6H As shown. In some embodiments, the text insertion cursor 692 is displayed at a first location in the search box (e.g., at the end of the text 602), and in response to detecting contact of a hand and / or finger 605 at a second location of the text 602 (e.g., in the middle of the text 602), the display of the text insertion cursor 692 at the first location ceases, and the text insertion cursor 692 is displayed at the location of the contact corresponding to the second location of the text 602.

[0214] exist Figure 6I In this configuration, the stylus 600 is positioned outside the hover threshold 601, as shown by icon 603, corresponding to the positioning of the content (e.g., text) input area including text 602, and the device 500 does not modify the display of the user interface 609. In response to the stylus 600 being held at a position corresponding to the content input area, as... Figure 6J As shown, upon entering hover threshold 601, option 621 allows for display with visual emphasis 618. In some implementations, visual emphasis 618 is displayed with a primary visual appearance. Figure 6J In this configuration, the stylus 600 is positioned corresponding to the content input area including text 602, but optionally not corresponding to the selectable option 621. In some embodiments, the display of the selectable option 621 stops in response to the stylus 600 moving out of the hover threshold 601 (e.g., away from the touchscreen 504) and / or in response to the stylus moving to a position outside the content input area of ​​the user interface 609 (even if remaining within the hover threshold 601). When the selectable option 621 is displayed, selection of the selectable option 621 optionally initiates the execution of one or more operations associated with the content input area. For example, as... Figure 6KAs shown, in response to the selection of selectable option 621 (e.g., contact between stylus 600 and touchscreen 504 at a location corresponding to selectable option 621), the display of text 602 ceases. Furthermore, in response to the selection of selectable option, visual emphasis 618 is displayed with a second visual appearance different from the first visual appearance (e.g., different scale, color, opacity, shadow, border, and / or lighting effects). It should be understood that... Figures 6I to 6K The embodiments illustrated are merely exemplary, and in some embodiments, other alternative options are displayed in response to hover events between the input device and the surface corresponding to the positioning of user interface elements.

[0215] exist Figures 6L to 6P The text describes the positioning of the stylus 600 within or outside a hover threshold 601 and / or optionally corresponding to selectable options, and various responses of the device 500. For example, in... Figure 6L In the icon 603, the stylus 600 is positioned outside the hover threshold represented by threshold 601, at a location corresponding to the positioning of the search icon 604. In some embodiments, although the positioning of the stylus 600 corresponds to the search icon 604, the device 500 does not display any additional visual emphasis or elements when the stylus 600 is outside the hover threshold.

[0216] exist Figure 6M In this scenario, the stylus 600 moves to a position within a hover threshold 601, but does not correspond to the position of the search icon 604. For example, when the tip of the stylus 600 is outside a threshold distance (e.g., 0.5cm, 1cm, 3cm, 5cm, or 10cm) from the search icon 604 and within a threshold distance 601 from the touchscreen 504, the device 500 abandons displaying additional visual emphasis or visual elements associated with the search icon 604. In some embodiments, such as Figure 6N As shown, in response to movement of the stylus 600 within a threshold distance of the search icon 604, the device 500 displays a visual emphasis or change to the icon 604 (e.g., modified scale, color, opacity, shadow, border, and / or lighting effects). For example, visual emphasis 618 includes an area surrounding the search icon 604, optionally visually emphasized with a solid color and / or semi-transparent color surrounding the search icon 604. In some embodiments, movement of the stylus 600 while displaying visual emphasis 618 and hovering over the search icon 604 may or may not result in a modification to the visual appearance of visual emphasis 618 and / or icon 604. For example, movement of the stylus 600 from... Figure 6N The search icon 604 shown moves down from its top left corner. Figure 6OThe lower left corner of the search icon 604 shown will not cause the device 500 to modify the visual emphasis 618 and / or the search icon 604 (e.g., the device 500 will not display the visual emphasis 618 and / or the icon 604 with parallax effects and / or lighting effects that change as the stylus 600 is positioned on the icon 604).

[0217] like Figure 6P As shown, in some embodiments, in response to the selection of icon 604, such as when stylus 600 contacts touchscreen 504 at a location corresponding to icon 604, one or more operations associated with search icon 604 are initiated, and visual emphasis 618 is modified. For example, the semi-transparency of visual emphasis 618 may be optionally decreased or increased, and / or the color of visual emphasis 618 may be optionally modified. In some embodiments, the modified visual emphasis is maintained while stylus 600 remains in contact with touchscreen 504. In some embodiments, in response to stopping the selection of search icon 604, visual emphasis 618 is modified (e.g., to correspond to the position relative to the icon 604). Figure 6N (Visual appearance described in hover events).

[0218] exist Figure 6Q In this context, after selection of search icon 604 is terminated (e.g., corresponding to stylus 600 disengaging / lifting from touchscreen 504), stylus 600 remains within hover threshold 601 but moves beyond a threshold distance from search icon 604 (e.g., the stylus 600's position does not correspond to search icon 604). In some embodiments, in response, device 500 stops the visual emphasis on icon 604. It should be understood that user interface 609 optionally includes additional or alternative selectable options, and the interactions described relative to search icon 604 (e.g., hovering, not hovering, moving stylus 600 to a position corresponding to search icon 604 and / or optionally selecting the search icon) are optionally the same or similar to such additional or alternative selectable options.

[0219] Figures 6R to 6W An example of an interaction between an electronic device 500 and a cursor input device (e.g., a mouse or touchpad) is illustrated. In some embodiments, based on the determination that the positioning of a cursor controlled by such a cursor input device corresponds to a graphic object, device 500 displays visual emphasis associated with the graphic object. In some embodiments, based on the determination that the cursor positioning does not correspond to a graphic object, the visual emphasis on the graphic object is abandoned.

[0220] For example, in Figure 6RIn this embodiment, touchpad 607 communicates with electronic device 500 (e.g., wirelessly or via a wired connection). In some embodiments, in response to detecting contact 619 on touchpad 607, device 500 displays cursor 613 in user interface 609. In some embodiments, cursor 613 is displayed in user interface 609 in response to device 500 detecting communication between touchpad 607 and device 500. In some embodiments, the positioning of cursor 613 in user interface 609 is modified in response to movement of contact 619 on touchpad 607 (e.g., moving to the right). For example, in Figure 6S In response to detecting that touch 619 has moved to the right on touchpad 607, device 500 moves cursor 613 to the right in user interface 609. Figure 6T In the process, if contact 619 is detected to have moved further to the right, and based on the determination that the new position of cursor 613 corresponds to the position of search icon 604, visual emphasis 618 is displayed on icon 604 (e.g., cursor 613 stops displaying and is represented by visual emphasis 618 in user interface 609). In some embodiments, visual emphasis 618 includes one or more lighting effects 618A. In some embodiments, lighting effect 618A includes specular highlights (e.g., simulated lighting effects of simulated object movement relative to one or more simulated or real light sources, such as reflection and / or refraction) to provide the user with a sense of how further input (e.g., movement of contact 619) corresponds to the current state of interaction with search icon 604. For example, optionally in response to user input (e.g., indication of movement of contact 619 received from touchpad 607), the display gloss of specular highlights around portions of visual emphasis 618 (e.g., the border of the visual emphasis) is modified.

[0221] exist Figure 6U In the process, when the cursor is positioned corresponding to the search icon 604 and visual emphasis 618 is displayed, user input for moving the cursor is received (e.g., device 500 detects upward and leftward movement of touch 619 on touchpad 607). In some embodiments, based on determining that the modified cursor position continues to correspond to the search icon 604 (e.g., having a threshold distance from the search icon 604, such as 0.5cm, 1cm, 3cm, 5cm, or 10cm), lighting effect 618A is optionally modified and / or parallax effect is optionally applied to visual emphasis 618 based on the movement of touch 619. For example, as Figure 6UAs shown, contact 619 moves towards the upper left of touchpad 607. In response to the movement of contact 619, and based on determining that the modified positioning of the cursor corresponds to search icon 604, a portion of specular highlight 618A (e.g., the upper left portion) is modified, and / or the visual emphasis and / or the shape of icon 604 is modified, thereby creating a parallax effect between search icon 604 and visual emphasis 618 based on the movement of contact 619. In some embodiments, the modification of specular highlight includes modification of the point pointing to the simulated light source of the corresponding graphic representation.

[0222] exist Figure 6V In this process, device 500 detects that contact 619 has moved downwards, although optionally less than the amount required for the cursor to move beyond a threshold amount from the area corresponding to search icon 604, and in response, modifies the aforementioned parallax effect. Similarly, as relative to... Figure 6U The specular highlight 618A is optionally modified accordingly. Such modification optionally includes reducing the brightness or otherwise modifying the lighting effect of the visual emphasis 618 at locations not corresponding to one or more parts of the movement, and / or increasing the brightness or otherwise modifying the lighting effect of the visual emphasis 618 at locations corresponding to one or more parts of the movement. Figure 6W In the context of the search, when the cursor position corresponds to the search icon 604, such as... Figure 6W The input received is a selection of the search icon 604 (e.g., a tap or click on the touchpad 607). In response to receiving this selection, one or more operations as previously described with reference to icon 604 are optionally performed (e.g., a search operation is performed).

[0223] Figures 6X to 6Y Examples of contextual information displayed in response to determining that timing information associated with a hover event meets one or more criteria are provided.

[0224] For example, such as Figure 6X As shown, the stylus 600 is positioned corresponding to the search icon 604, and the stylus 600 is within a threshold distance 601 of the touchscreen 504. In some embodiments, a timer 623 is started when the stylus 600 is positioned corresponding to the search icon 604. One or more operations are performed based on a criterion that determines the stylus 600 hovers over the search icon 604 for a time threshold 625 exceeding a certain threshold. For example, as... Figure 6YAs shown, in response to detecting that the stylus 600 has been hovering over icon 604 for a longer period than time threshold 612, device 500 displays tooltip 631. Tooltip 631 optionally includes contextual information associated with the corresponding visual element, such as search icon 604. In some embodiments, this contextual information is the name or description of the relevant operation (e.g., "search," indicating that selecting icon 604 will result in initiating a search operation).

[0225] Figures 6Z to 6D D illustrates hover events and interactions with media content such as a media player. In some implementations, the user interface 609 includes media content, such as a media player. Although some implementations are described with respect to video displayed in the media player, it should be understood that the media player additionally or alternatively includes audio content.

[0226] exist Figure 6Z In this configuration, media player 608 is displayed in user interface 609, and stylus 600 is positioned corresponding to the media player. As shown in figure 603, stylus 600 is outside the hover threshold 601, therefore device 500 does not modify the display of user interface 609 in response to the presence of stylus 600. Figure 6AA In this context, the stylus 600 enters a hover threshold 601, and in response to entering the hover threshold, the device 500 displays one or more media controls 620 for the media player 608. The media controls 620 optionally include one or more selectable options for modifying the playback of the corresponding media content (e.g., fast forward, rewind, pause, play, skip forward, skip backward, modify audio volume, and / or navigate the playback queue). Figure 6AB In this process, the stylus 600 moves to a position corresponding to the fast-forward icon 620A, while remaining within a hover threshold 601 without contacting the touchscreen 504. In some implementations, depending on whether the stylus 600 corresponds to a media control such as the fast-forward icon 620A, visual emphasis associated with the corresponding media control is displayed, such as modifications to scale, color, opacity, shadow, border, and / or lighting effects. Figure 6AB As shown. In some implementations, selecting the appropriate optional option will initiate the execution of one or more operations that modify the playback or characteristics of the media, such as fast-forwarding relative to the media. Figure 6AC In the process, a selection of the fast-forward icon 620A is detected. In some implementations, the selection is determined in response to other gestures or indications (e.g., double-tapping on a surface, tapping or gesturing on an input device, and / or hand gestures). Figure 6AD In response to detecting a selection of the fast-forward icon 620A, device 500 advances the playback positioning of the media (e.g., navigating the media content based on the selection).

[0227] Figures 6AE to 6AF Examples of hover interactions associated with linked content, such as webpage links, are shown. Figure 6AE In this example, as illustrated relative to touchscreen 504, stylus 600 is outside the hover threshold 601. Figure 6AE In this context, the relative positioning of the stylus 600 corresponds to the content link 610, but the device 500 does not modify the visual appearance of the link 610 because the stylus 600 is outside the hover threshold 601 of the touchscreen 504. In some embodiments, the content link 610 can be selected to initiate one or more actions, such as displaying the corresponding webpage, launching an application, and / or adding a webpage-based bookmark. In some embodiments, in response to a hover event of the stylus 600 at the location corresponding to the context link 610 above the touchscreen 504, the device 500 visually emphasizes the link 610, such as... Figure 6AF As shown. For example, in response to the hover event, device 500 may optionally modify the visual appearance of the background color, opacity, and / or font characteristics, including bolding and / or underlining link 610.

[0228] Figures 6AG to 6AI Examples of interaction with non-editable content (such as displayed text content or other graphical objects) according to this disclosure are illustrated. For example, in Figure 6AG In this context, non-editable text 612 is displayed in the user interface 609. This text optionally corresponds to text on a webpage or text included in an image. Figure 6AG In this context, the position of the stylus 600 corresponds to the uneditable text 612, but the stylus 600 is outside the threshold distance 601 of the touchscreen 504, therefore the device 500 does not modify the user interface 609. Figure 6AH In response to a hover event between the stylus 600 and the touchscreen 504, the device 500 displays a text selection cursor preview 615 in a first visual appearance at a position in the text 612 corresponding to the stylus 600 (e.g., corresponding to the tip of the stylus 600). The first visual appearance optionally indicates to the user the positioning in which a subsequent selection of the text 612 will begin (e.g., in response to contacting the stylus 600 with the touchscreen 504). In some embodiments, the first visual appearance includes a first opacity level and / or color. Figure 6AI In this process, device 500 detects a selection, such as contact between stylus 600 and touchscreen 504. In some embodiments, in response to this contact, the display of text selection preview cursor 615 stops, and the location of the contact on touchscreen 504 (e.g., at...) Figure 6AIThe text selection cursor 617 is displayed at the beginning of the text 612. In some embodiments, the text selection cursor 617 is displayed with a second visual appearance to visually distinguish the selection cursor preview from the actual selection cursor. For example, the second visual appearance may optionally be darker and / or less transparent than the first visual appearance. Figure 6AJ As shown, while maintaining contact between the stylus 600 and the touchscreen 504, the stylus 600 moves to the right along the touchscreen 504. In response to this movement, the positioning of the text selection cursor 617 is updated, and the content (e.g., text) between the positioning corresponding to the initial contact and the ending positioning in the user interface is selected (e.g., highlighted with a degree of transparency), as shown in selection 644.

[0229] exist Figure 6AK to Figure 6AO The example demonstrates hover interaction with the selection of non-editable text. Figure 6AK In the case of selection 644, which includes one or more selectable options (including grabber 646A), the non-editable text 612 is selected (e.g., as indicated by reference). Figure 6AJ (As described). Selectable options, including gripper 646A, are optionally selected to modify the boundary of selection 644 relative to text 612. In some embodiments, the boundary of selection 644 is modified in response to selecting a corresponding gripper such as gripper 646A (e.g., bringing stylus 600 into contact with touchscreen 504) and modifying the positioning of gripper 646A (e.g., dragging stylus 600 on touchscreen 504). Figure 6AL In response to a hover event corresponding to gripper 646A, device 500 displays arrow 648A, which indicates the direction of manipulation for selectable option 646A. For example, in Figure 6AL In the middle, arrow 648A indicates that the selectable option 646A can be manipulated in the left and right directions, which will cause the selection 644 to shrink or expand accordingly.

[0230] exist Figure 6AM In response to device 500 detecting selection of gripper 646A by stylus 600, device 500 initiates an operation to manipulate gripper 646A and / or selection 644. For example, when stylus touches gripper 646A, selection 644 is modified based on movement of stylus along touchscreen 504. Figure 6AN In response to the stylus 600 moving to the right while maintaining the selection of selectable option 646A, selection 644 expands to the right.

[0231] In some implementations, device 500 displays different functional indications in response to detecting a stylus hovering over different selectable options. For example, in Figure 6AOIn response to a hover event corresponding to the stylus 600 above the gripper 646B, the device 500 displays an arrow 648B, indicating that the gripper 646B can be manipulated vertically (instead of horizontally) to vertically expand or contract the selection 644. It should be understood that vertical manipulation of the selection 644 is optionally relative to... Figures 6AL to 6AM The descriptions are similar or identical.

[0232] exist Figure 6AP to Figure 6AT The example illustrates hover events pointing to selectable icons and / or user interface objects that include selectable icons, as well as the display of additional selectable options.

[0233] exist Figure 6AP In the user interface 609, a region 614 including multiple selectable options (e.g., "Breaking," "Sports," "USA," and "World") is displayed. These selectable options can be chosen to initiate one or more actions, such as displaying the corresponding webpage, launching an application, and / or adding a webpage-based bookmark. Figure 6AQ In this device, stylus 600 moves within a hover threshold 601 of touchscreen 504, as shown by icon 603, and is positioned corresponding to an area 614 that includes the selectable options. In response, device 500 displays multiple navigation arrows, including arrow 624. In some embodiments, arrow 624 can be selected to scroll the multiple selectable options (e.g., to the right). For example, device 500 may optionally, in response to selection of arrow 624, display one or more currently undisplayed selectable options, deselect one or more currently displayed selectable options, and / or modify the positioning of the corresponding currently displayed selectable option.

[0234] exist Figure 6AR In the process, device 500 detects a hover event, which includes the stylus 600 moving to correspond to the position of navigation arrow 624 while held within a hover threshold 601 (e.g., without touching touchscreen 504). In response to this hover event, navigation arrow 624 is displayed with visual emphasis (e.g., modifications to scale, color opacity, shadow, border, and / or lighting effects) having a first appearance (e.g., a first scale, color opacity, shadow, border, and / or lighting effects), such as... Figure 6AR As shown. In Figure 6AS In this process, device 500 detects a selection of navigation arrow 624 (e.g., stylus 600 contacts touchscreen 504 at a location corresponding to navigation arrow 624). It should be understood that in some embodiments, selection of navigation arrow 624 may optionally be one or more alternative inputs (e.g., hand gestures and / or double-clicks between stylus 600 and the surface). Figure 6AT In response to Figure 6AS Upon input, device 500 scrolls right through selectable options in area 614 to display the selectable option "Alliums" in area 614. Furthermore, stylus 600 moves beyond hover threshold 601, and in response to such movement, device 500 stops displaying the navigation arrows, as... Figure 6AT As shown. In some embodiments, the navigation arrow 624 remains displayed even though the stylus 600 is moved beyond the hover threshold 601. In some embodiments, after the stylus 600 is moved beyond the hover threshold 601, the navigation arrow is displayed for a threshold amount of time (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds), and after the time threshold is exceeded, the device 500 stops displaying the navigation arrow.

[0235] exist Figure 6AU to Figure 6AY In this context, user interface 609 corresponds to a web browser interface (e.g., Safari web browser) and exemplifies hover events corresponding to multiple tabs of the web browser.

[0236] exist Figure 6AU In the user interface 609, the content corresponding to the first label 640A is not displayed, while the content corresponding to the second label 640B is displayed (for example, the second label 640B is...). Figure 6AU The currently selected tag in the context of the user. In some implementations, content corresponding to the corresponding tag is displayed (e.g., content from two or more tags is displayed simultaneously), and in other implementations, content corresponding to the corresponding tag is not displayed (e.g., a default login page is displayed, such as a page including the user's bookmarks links). Figure 6AU As shown by icon 603, the stylus 600 is outside the hover threshold 601. Figure 6AV In this process, electronic device 500 detects a hover event, which includes the stylus 600 moving to a hover threshold 601 and to a location corresponding to label 640A (e.g., an area of ​​user interface 609 corresponding to the display portion of label 640A). In response to the hover event, device 500 displays selectable options 682A for label 640A. In some embodiments, in response to the hover event, visual emphasis (e.g., scale, color, opacity, shadow, border, and / or lighting effects) of the area corresponding to the first label 640A is modified or displayed. In some embodiments, in response to a hover event corresponding to selectable options 682A and / or 682B, visual emphasis of selectable options 682A and / or 682B is displayed, or the currently displayed visual emphasis is modified, as described later. Figure 6AWIn this process, the stylus 600 moves from a position corresponding to the first label 640A to a position corresponding to the area associated with the second label 640B. In response to determining that the position of the stylus 600 does not correspond to the first label 640A, the device 500 stops displaying the selectable option 682A, such as... Figure 6AW As shown. In some embodiments, the visual emphasis corresponding to the first label 640A is additionally or alternatively stopped based on such determination. Furthermore, in Figure 6AW In response to determining that the stylus 600 is hovering over the touchscreen 504 at a position corresponding to the second label 640B, the display of the second label 640B is modified. For example, visual emphasis on the second label 640B is displayed, and / or the device 500 initiates the display of selectable option 682B; in some embodiments, the details of the displayed label 640B and / or selectable option 682B are compared with those of the second label 640B. Figure 6AU The same or similar (e.g., the visual emphasis and / or display of option 682A). In Figure 6AX In this process, device 500 detects a selection of selectable option 682B (e.g., detects an indication of contact between stylus 600 and touchscreen 504 at a location corresponding to selectable option 682B), and optionally modifies the visual emphasis of selectable option 682B (e.g., modifications to aspects such as scale, color, opacity, shadow, border, and / or lighting effects). In some embodiments, the input for selecting selectable option 682B is a hand gesture and / or one or more gestures on an input device.

[0237] exist Figure 6AY In response to Figure 6AX When the selectable option 682B is selected, device 500 stops displaying the content corresponding to the second label 640B. Furthermore, in response to stopping the display of the content corresponding to the second label 640B, the content corresponding to the first label 640A is started, such as... Figure 6AY As shown. In some implementations, upon determining that the user interface 609 includes a single tab, default content (e.g., a login page) is displayed in response to a request to stop displaying content corresponding to that single tab.

[0238] exist Figures 6AZ to 6BF The image shows hover events corresponding to requests that interact with user interface objects.

[0239] exist Figure 6AZIn this context, user interface 609 corresponds to a drawing user interface including a control palette 630, which includes selectable options for modifying handwriting produced by the stylus 600 in the drawing user interface. For example, selectable options 632A, 632B, and 632C can be selected to modify the currently selected writing and / or drawing tool for the stylus 600. Based on the positioning of the stylus 600 relative to the touchscreen 504 and the currently selected writing and / or drawing tool, a virtual shadow 662 is displayed in user interface 609. (This is in contrast to method 900 and...) Figures 8A to 8C The behavior and appearance of the virtual shadow are further described. The currently selected user interface object 642 is displayed in the content input area of ​​the user interface 609, including a grabber 650A, which can optionally be selected to modify the user interface object 642. In some embodiments, a corresponding "grabber" is an optional option that can be selected to modify the corresponding virtual object. For example, detecting a selection of a corresponding grabber and detecting a modification of the corresponding grabber while maintaining that selection can optionally modify (e.g., scale, pan, and / or expand) the virtual object associated with the corresponding grabber in some way (e.g., in the direction of scaling, panning, and / or expanding the virtual object) based on that modification.

[0240] exist Figure 6BA In this context, the stylus 600 moves to a position relative to the touchscreen 504 corresponding to the gripper 650A but outside the hover threshold 601, as shown in icon 603. Because the stylus 600 is outside the hover threshold 601, the device 500 does not modify the display of object 642 and / or the various grippers of the user interface object 642. Figure 6BB In this embodiment, the stylus 600 moves within a hover threshold 601 and to a position corresponding to the gripper 650A, and in response, the device 500 displays a directional arrow 648A. In some embodiments, in response to hovering over the gripper 650A, one or more visual indicators are displayed associated with the direction of manipulation (e.g., scaling, deforming, and / or expanding) of the user interface object 642. In some embodiments, the directional arrow indicates a possible direction of manipulation. For example, the directional arrow 648B is oriented to extend horizontally to the left and right, thereby indicating that manipulation (e.g., expanding a visual object) towards the left and / or right of the gripper 650 is possible. In some embodiments, the directional arrow indicates a single direction of manipulation (e.g., up only, down only, right only, or left only).

[0241] exist Figure 6BC In this context, when the positioning of the stylus 600 corresponds to the positioning of the gripper 650A, the device 500 receives a selection instruction including the stylus 600 contacting the touchscreen 504. In response to this contact, the device 500 initiates an operation associated with manipulating the user interface object 642. For example, in... Figure 6BD In the middle, while maintaining relative Figure 6BC Upon described contact, a movement instruction is received (e.g., the stylus 600 slides to the left along the touchscreen 504). In response to the operation manipulating the user interface object 642, the device 500 scales the user interface object 642 according to the movement (e.g., scales the object 642 to the left). In some embodiments, such manipulation corresponds to scaling (e.g., stretching, enlarging, and / or shrinking) the user interface object to a degree related to (e.g., positively or negatively related to) the amount of movement of the stylus 600.

[0242] exist Figure 6BE In the display, a user interface object 642 is shown, along with directional arrows 648B corresponding to different manipulations of the user interface object 642. Specifically, in response to a hover event, including a stylus 600 hovering above the touchscreen 504 at a location corresponding to the positioning of the corresponding gripper (e.g., the upper-left gripper of object 642) corresponding to the navigation arrow 648B, the device 500 optionally displays the navigation arrow 648B. In some embodiments, the navigation arrow 648B indicates that the selection and modification of the positioning of the gripper 650B can modify the user interface object 642 in two non-parallel directions (e.g., scaling, deforming, and / or expanding the object toward the upper-left and / or lower-right of the touchscreen 504). Similarly, in Figure 6BF In response to a hover event, including a positional event where the stylus 600 moves above the touchscreen 504 corresponding to the positioning of a corresponding gripper (e.g., the upper gripper of object 642) corresponding to the navigation arrow 648C, a directional arrow 648C is displayed corresponding to the manipulation of the user interface object in the vertical direction. In some embodiments, the navigation arrow 648C indicates that the selection and modification of the positioning of the gripper 650C can vertically modify the user interface object 642 (e.g., scaling, deforming, and / or expanding the object toward the top and / or bottom of the touchscreen 504). In some embodiments, in response to a hover event where the stylus 600 is detected above other corresponding grippers of the user interface object 642, the device 500 similarly displays visual indications corresponding to the manipulation direction of these grippers, such as relative to... Figures 6AZ to 6BF As stated above.

[0243] Figure 7A Figure 7J is a flowchart illustrating a method 700 for displaying additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device. Method 700 is optionally performed on an electronic device (such as device 100, device 300, and device 500), as referenced above. Figures 1A to 1B , Figures 2 to 3 , Figures 4A to 4B and Figures 5A to 5IThe operations in method 700 may be optionally combined, and / or the order of some operations may be optionally changed.

[0244] As described below, method 700 provides a way to display additional controls and / or information when an input device, such as a stylus, hovers over a user interface displayed by an electronic device. This method reduces the cognitive burden on the user when interacting with the device user interface of this disclosure, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charging cycles.

[0245] In some embodiments, method 700 is performed at an electronic device that communicates with a display generating component and one or more sensors (e.g., touch-sensitive surfaces). For example, a mobile device (e.g., tablet, smartphone, media player, or wearable device) or computer that optionally communicates with one or more of the following: a mouse (e.g., external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), and / or a controller (e.g., external). In some embodiments, the display generating component is a display integrated with the electronic device (optionally touch-sensitive and / or touchscreen display), an external display such as a monitor, projector, television, and / or a hardware component (optionally integrated or external) for projecting a user interface or making the user interface visible to one or more users.

[0246] In some embodiments, the electronic device displays (702a) a user interface including a first user interface object, such as a display generating component. Figure 6C The user interface 609 includes a search box. For example, this user interface may optionally be the system user interface of an electronic device (e.g., a home screen interface, such as...). Figure 4A This can be illustrated as a user interface for a content creation application (e.g., a drawing user interface), a note-taking application, a content browsing user interface, or a web browsing user interface. In some embodiments, the first user interface object is a selectable option within that user interface that can be selected to perform a corresponding function optionally associated with that user interface. For example, the first user interface object may optionally be a button on a home screen user interface that can be selected to display an application icon via a display generation component, a button in a drawing user interface that can be selected to display a palette including options for changing the characteristics of handwriting generated in the drawing user interface using a stylus, or a representation of media within a content browsing interface that can be selected to initiate playback of the corresponding media.

[0247] In some embodiments, when a user interface including a first user interface object is displayed via a display generating component, the electronic device detects (702b) a corresponding object approaching but not touching a surface associated with the user interface via one or more sensors, such as detecting... Figure 6C The stylus 600, or Figure 6F and / or Figure 6G The hand 605 in the image (e.g., a touch-sensitive surface, a physical surface on which the user interface is projected, or a virtual surface corresponding to at least a portion of the user interface). For example, the corresponding object may optionally be a finger of a user's hand interacting with the surface. In some embodiments, the object is a stylus communicating with an electronic device, or optionally a stylus not communicating with an electronic device. In some embodiments, one or more signals transmitted between the corresponding object, the electronic device, and / or the surface are used to determine proximity between the corresponding object and the surface. For example, the corresponding object may optionally be a stylus having one or more sensors configured to detect one or more signals transmitted from the surface. In some embodiments, the received signal strength of the one or more signals is used as a criterion for determining proximity. In some embodiments, the one or more signals include data encoding one or more relative distances between the stylus and the surface. In some implementations, when the object is greater than a first threshold distance from the surface (e.g., 0.0 cm, 0.01 cm, 0.03 cm, 0.05 cm, 0.1 cm, 0.3 cm, 0.5 cm, or 1 cm) and less than a second threshold distance greater than the first threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm), the object is determined to be close to but not in contact with the surface. Otherwise, the object is optionally determined to be not near the surface or to be in contact with the surface.

[0248] In some implementations, in response to detecting a corresponding object approaching but not touching the surface (702c), depending on whether the corresponding object approaching the surface is an input device (such as an input device for communicating with an electronic device)... Figure 6C The stylus 600 in the middle) and the positioning of the input device corresponds to the first user interface object (such as Figure 6C The electronic device displays (702d) a first selectable option in the user interface that can be selected to perform a first operation associated with the first user interface object, such as displaying a stylus 600 corresponding to the search box. Figure 6CThe text insertion cursor preview is 690. For example, the input device is optionally a stylus device. In some embodiments, the input device is a wearable device (e.g., a glove or thimble). Communication between the input device and the electronic device optionally includes sending and / or receiving one or more data streams through the input device and the electronic device. In some embodiments, determining that the location of the input device corresponds to a first user interface object (e.g., including a selection indicator and / or highlight) optionally includes determining that the input device (and / or the vertical or other projection of the input device on the touch-sensitive surface) is within a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm) of a corresponding location of the first user interface object displayed by the display generating component or a corresponding location corresponding to that user interface object. As mentioned herein, determining that the location of the input device corresponds to a first user interface object and that the input device is within a threshold distance of a corresponding portion of the first user interface object or a corresponding portion corresponding to the first user interface object is optionally a "hover event". Similarly, the state of the input device within a threshold distance is optionally referred to as "hovering" (e.g., above a surface). In some embodiments, selectable options are not displayed until the input device is within a threshold distance of a corresponding location of the first user interface object or a corresponding location corresponding to the first user interface object. For example, the input device, electronic device, and / or touch-sensitive surface optionally determines that the magnitude of a vector extending from a portion of the input device (e.g., a tip) toward or from the surface toward the input device is less than a threshold magnitude. In some embodiments, selectable options are buttons (e.g., play, pause, rewind, or fast forward buttons) that control media associated with playback of the displayed media. In some embodiments, selectable options can be selected to initiate a scrolling process of one or more displayed elements. For example, selectable options optionally control scrolling operations on a list of text and / or icons. In some embodiments, the user interface includes a text input box, and selectable options can be selected to delete or select one or more characters displayed in that text input box. In some embodiments, the display of a first selectable option is abandoned if the location of the input device does not correspond to the first user interface object, even if the input device is close to but not in contact with the surface. In some implementations, in response to the detection of each hover event in a sequence of multiple hover events, the electronic device responds to the hovered input device in one or more of the various methods described herein. In some implementations, upon hovering, subsequent input (e.g., the input device contacting a surface at the location of a first selectable option) is detected to perform a first operation.

[0249] In some implementations, based on the determination that the corresponding object approaching the surface is not an input device for communicating with an electronic device (such as... Figures 6F to 6GThe electronic device abandons (702e) the display of a first selectable option in the user interface that can be selected to perform a first operation associated with the first user interface object, such as referencing (hand 605). Figures 6F to 6G The described text insertion cursor preview 690 is not displayed. For example, the object is optionally a user's finger near the surface. In some embodiments, the object is an input device (e.g., a stylus) that does not communicate with an electronic device. Displaying selectable options for performing additional operations when the input device is near the user interface object reduces the amount of input required to access those operations.

[0250] In some implementations, in response to detecting a corresponding object approaching but not touching a surface, based on determining that the corresponding object approaching the surface is an input device communicating with the electronic device and that the positioning of the input device corresponds to a first user interface object (e.g., the vertical projection of the tip of the input device onto the surface within a threshold distance of the first user interface object, such as 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm), the electronic device modifies (704) the visual characteristics of the first user interface object to indicate that the first user interface object is selectable, such as... Figure 6N The display of icon 604 is modified. As previously described with respect to step 702, one or more functions are optionally performed in response to determining that the input device has initiated a hover event. One such embodiment includes modifying one or more visual characteristics of the first user interface object. For example, the modification includes displaying one or more visual indicators to indicate the interactivity of the user interface object (e.g., a button). The visual indicators optionally include changing color, shading, hue, saturation, lighting effects, displaying or modifying the border around the user interface object, initiating or modifying animation of the first user interface object, changing the shadow associated with the first user interface object, scaling one or more portions of the user interface object, and / or modifying the perceived positioning (e.g., depth) of the first user interface object in the user interface. In some embodiments, the visual indicators optionally include highlighting the first user interface object. In some embodiments, different visual indicators are displayed in response to a hover event corresponding to a corresponding portion of the first user interface object. For example, hovering over a portion of the first user interface object corresponding to the top boundary optionally results in the display of one or more arrows (e.g., extending upwards and / or downwards). Similarly, in response to hovering over the lateral boundaries of the first user interface object, one or more arrows (e.g., extending to the left and / or right) are optionally displayed. Displaying one or more modifications to one or more visual characteristics of the first user interface object to indicate that the user interface object is interactive effectively conveys that the first user interface object is interactive and reduces errors in the interaction with the first user interface object.

[0251] In some implementations, the first user interface object is associated with a selection of a first region of the user interface and not with a selection of a second region of the user interface that is different from the first region of the user interface, and the first user interface object is interactive to modify the region of the user interface selected by the first user interface object (706a), such as Figure 6AK Selection 644. For example, the user interface is a font-based handwriting and / or drawing user interface, and a first user interface object (e.g., a selection indicator and / or highlight) indicates the selection of a first portion of the content displayed in the user interface. It should be understood that the user interface optionally includes individual fonts, drawings, and / or handwritten content or forms some combination, and optionally includes graphic content (e.g., images and / or hand-drawn shapes or curves). In some embodiments, the first user interface object corresponds to a highlighted portion of the user interface. For example, the user's first portion corresponds to a selected portion of content including text and / or handwritten content displayed for visual distinction (e.g., a partially transparent background with a first color or fill pattern based on the positioning and / or size of the underlying selected content).

[0252] In some implementations, in response to detecting a corresponding object approaching but not touching a surface, based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to a first user interface object (e.g., as previously described with respect to step 704), the electronic device displays (706b) a visual indication via a display generating component associated with one or more directions of modification of the area of ​​the user interface selected by the first user interface object, such as... Figure 6AL and Figure 6AOThe display of arrows 648A or 648B. One or more visual elements, such as circles or spheres, are optionally displayed at the location of a visually distinct area (e.g., overlaid on a border) to convey the interactivity of the selected portion (e.g., before the input device hovers over the first user interface object), and in response to a hover event corresponding to the respective visual element, one or more arrows are optionally displayed indicating one or more directions of modification of the selection. For example, the selection is optionally a semi-rectangular highlight containing font-based text, and the hover event optionally causes left and / or right arrows to be displayed on the lateral edges to indicate that the highlight can be laterally expanded. In some embodiments, interaction with the first user interface object modifies the selection to include a second portion of the content instead of the first portion. For example, touching the surface with the input device and subsequently moving the input device along the surface optionally expands the text selection (e.g., the text selection expands vertically and / or laterally depending on the vertical and / or lateral movement of the input device while in contact with the surface). In some implementations, the hover event depends on determining that the input device corresponds to a first user interface object (e.g., the vertical projection of the tip of the input device onto a surface within a threshold distance of the first user interface object, such as 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm). In some implementations, one or more visual indicators are displayed in response to a hover event associated with a corresponding portion of the first user interface object. For example, hovering over a portion of the first user interface object corresponding to its top boundary optionally results in the display of one or more vertical arrows (e.g., extending upwards and / or downwards). Similarly, in response to hovering over a lateral boundary of the first user interface object, one or more horizontal arrows (e.g., extending to the left and / or right) are optionally displayed. Displaying visual indicators in response to hover events reduces erroneous input from the user and prevents the continuous display of such visual indicators, thus reducing the power consumption and computational load required for such operations.

[0253] In some implementations, the first user interface object corresponds to the first part of the user interface, such as Figure 6AW Label 640B, and the first selectable option is the first part of the stop user interface (such as...) Figure 6AWThe display of the selectable option (682B) is associated with (708a). For example, the first user interface object is a label corresponding to a first portion of content (e.g., a webpage, or other content "page") in the application's user interface. For example, in response to detecting a selection of the first user interface object, the electronic device displays content corresponding to the first user interface object in the application's user interface without displaying content corresponding to the second user interface object, and in response to detecting a selection of the second user interface object, the electronic device displays content corresponding to the second user interface object in the application's user interface without displaying content corresponding to the first user interface object. In some embodiments, the first selectable option is a text and / or graphic object that can be selected to stop the display of a first portion of the content, and optionally is not displayed until a hover event pointing to the first user interface object is detected.

[0254] In some implementations, when a first selectable option is displayed based on determining that a corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to a first user interface object (e.g., as previously described relative to step 704), the electronic device receives (708b) information via one or more sensors regarding the selection of the first selectable option (such as...). Figure 6AX One or more inputs corresponding to the selection of option 682B.

[0255] In some implementations, in response to receiving one or more inputs corresponding to a selection of a first selectable option, the electronic device stops (708c) the display of a first portion of the user interface (and optionally a first user interface object), such as stopping the display of the portion of a web browser user interface corresponding to label 640B, as... Figure 6AYAs shown. For example, in response to detecting a hover event over any portion of a first user interface object, the first user interface object is visually emphasized (e.g., with changes in color, bolding, and / or scale), and / or a first selectable option is displayed within or associated with the first user interface object. When hovering over a surface, in response to receiving a selection of the first selectable option (e.g., contacting the surface with the tip of an input device at the location of the first selectable option), the electronic device optionally stops the display of a first portion of the content. For example, hovering over an “X” included in a label corresponding to a webpage displayed in the user interface optionally causes the electronic device to visually distinguish the “X,” and a subsequent selection of the “X” (e.g., contacting the surface with an input device) causes the electronic device to stop the display of the webpage. In some embodiments, the electronic device also stops displaying the first selectable option (e.g., “X”) (e.g., stops displaying a label including “X”). Displaying the first selectable option in response to hover prevents the continuous display of the first selectable option and avoids clutter in the user interface, thus reducing the power and computational load required for such display.

[0256] In some implementations, the first user interface object includes a content input area, which includes content such as... Figure 6J The text includes a search box (602), and a first selectable option is associated with stopping the display of content in the content input area (710a), such as... Figure 6J The optional option 621 is provided. In some embodiments, the content input area is a text-based area (e.g., a text box), or another content area that includes handwriting and / or characters (e.g., handwritten and / or font-based text and / or graphic objects). A first optional option can optionally be selected to stop the display of all or a subset of the content included in the content input area. For example, the first optional option is a button or icon for stopping the display of text in the text input area (e.g., deleting or clearing text).

[0257] In some implementations, when a first selectable option is displayed based on determining that a corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to a first user interface object (e.g., as previously described relative to step 704), the electronic device receives (710b) information via one or more sensors regarding the selection of the first selectable option (such as...). Figure 6K One or more inputs corresponding to the selection of optional option 621.

[0258] In some implementations, in response to receiving one or more inputs corresponding to a selection of a first selectable option, the electronic device stops (710c) the display of content in the content input area, such as by... Figure 6K The display of text 602 is shown by stopping the display. For example, in response to a hover event corresponding to a portion of the text input area, a first selectable option is displayed. In response to receiving a selection of the first selectable option (e.g., by contacting the surface with the tip of the input device at the location of the first selectable option), the display of the entire or a subset of the text input area is optionally stopped. It should be understood that although the embodiments described herein are directed to text input areas, such functionality may optionally be applied to other content input areas (e.g., stopping the display of handwriting, including font-based and handwriting-based content, and / or graphic objects). Displaying the first selectable option in response to hover prevents the continuous display of the first selectable option and avoids clutter in the user interface, thus reducing the power and computational load required for such display.

[0259] In some implementations, the first user interface object is associated with the presented media content, such as Figure 6Z The media player 608 in the middle, and the first selectable option is associated with modifying the playback of media content (712a), such as Figure 6AB The selectable option 620A is provided. For example, the first user interface object includes, or is for, a media player for video and / or audio content. The first selectable option is optionally associated with navigation of such media content, such as searching forward or backward or refreshing to traverse the media content. In some embodiments, selecting the first selectable option starts, stops, or resumes playback of the media content. In some embodiments, the playback speed is increased or decreased, or navigation is performed between another content item and the current content item (e.g., the previous or next in the media content queue). All such operations are contemplated in relation to media playback and navigation.

[0260] In some implementations, when a first selectable option is displayed based on determining that a corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to a first user interface object (e.g., as previously described relative to step 704), the electronic device receives (712b) information via one or more sensors regarding the selection of the first selectable option (such as...). Figure 6AC One or more inputs corresponding to the selection of optional option 620B.

[0261] In some implementations, in response to receiving one or more inputs corresponding to the selection of a first selectable option, the electronic device modifies (712c) the playback of media content, such as Figure 6ADAs shown in media player 608. In some embodiments, when hovering over a location corresponding to the first user interface object, one or more of the selectable options described relative to the first selectable option are displayed. In response to receiving a selection of the first selectable option when the corresponding selectable option is displayed (e.g., during hovering) (e.g., by contacting the surface with the tip of the input device at the location of the first selectable option), execution of an associated operation is optionally initiated. Displaying the first selectable option in response to hover prevents the continuous display of the first selectable option, thus reducing the power and computational load required for such display.

[0262] In some implementations, in response to detecting a corresponding object approaching but not touching a surface, based on determining that the corresponding object approaching the surface is an input device communicating with an electronic device and that the location of the input device corresponds to a first user interface object (714a) (e.g., as previously described with respect to step 704), based on determining that the location of the input device meets one or more criteria, including criteria satisfied when the location of the input device corresponds to the first user interface object for a period of time longer than a threshold time, such as relative to Figure 6Y As shown in the stylus 600, the electronic device displays (714b) information associated with the first user interface object via a display generating component, such as... Figure 6Y Tooltip 631, as shown in icon 604, is displayed in response to a hover event involving an input device and a surface. For example, information describing a function corresponding to a first user interface object (e.g., information indicating what function will be executed if the first user interface object and / or a first selectable option are selected) is displayed in response to the input device hovering over the surface at the position corresponding to the user interface object for a period exceeding a threshold time (e.g., 0.05s, 0.1s, 0.25s, 0.5s, 0.75s, 1s, 2.5s, or 5s). This information optionally includes the name of the function associated with the user interface object and / or the selectable option, and optionally describes one or more inputs required to activate the function. In some embodiments, the first user interface object is visually emphasized (e.g., with color, bolding, and / or scaling) in response to the detection of a hover event.

[0263] Displaying information associated with the first user interface object in response to hover prevents users from accidentally initiating functions or unnecessarily reviewing documentation to understand the functions bound to the first user interface object, thus reducing the computational load and power consumption required for such operations.

[0264] In some implementations, the first user interface object includes a content (e.g., text) input area (716a), such as Figure 6CThe search box includes text 602. In some embodiments, in response to detecting a corresponding object approaching but not touching the surface, based on determining that the corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to a first user interface object (e.g., as previously described relative to step 704), the electronic device displays (716b) a visual indication of a content (e.g., text) insertion cursor at a location corresponding to the input device in the content input area via a display generation component. Figure 6C The insertion cursor preview 690 shown is displayed. For example, hovering the input device over a portion of the user interface corresponding to a user interface object causes a shadow corresponding to the text insertion cursor to be displayed. Optionally, the shadow of the text insertion is displayed with a first visual appearance (e.g., color, saturation, hue, opacity, and / or with a first animation) to convey the suggested positioning of the text insertion cursor to the user. In some examples, the suggested positioning of the text insertion cursor corresponds to a portion of the input device (e.g., a tip). For example, the suggested positioning corresponds to the positioning of the tip of the input device projected (e.g., vertically projected) onto a surface, and the projected surface positioning optionally corresponds to the suggested positioning in the content input area. In some embodiments, new content input (e.g., text input) will not cause the corresponding content to be displayed at the indication of the text insertion cursor until the text insertion cursor is placed at the indication of the text insertion cursor. In some embodiments, when the indication of the text insertion cursor is displayed at the suggested positioning described above, the text insertion cursor is displayed at different positions in the content input area (e.g., new content input (e.g., text input) will cause the corresponding content to be displayed at the position of the text insertion cursor) and / or is not displayed in the content input area.

[0265] In some implementations, when displaying a visual indicator, the electronic device receives (716c) one or more inputs via one or more sensors corresponding to the selection of the visual indicator (and / or the selection of an area within a threshold distance of the visual indicator, such as 0.1cm, 0.3cm, 0.5cm, 1cm, 3cm, 5cm, or 10cm). Figure 6D The stylus 600 is shown in the diagram. In some embodiments, in response to receiving one or more inputs corresponding to a selection of a visual instruction, the electronic device displays (716d) a text insertion cursor, such as a text insertion cursor, at a location corresponding to the input device in the text input area via a display generating component. Figure 6DThe text insertion cursor 692 is displayed. When hovering over a surface, in response to receiving a selection of a first selectable option (e.g., contacting the surface with the tip of the input device at the location of the first selectable option), the text insertion cursor is optionally inserted at and / or moved to the suggested location. This insertion and / or movement optionally includes displaying a text insertion cursor with a second visual appearance (e.g., corresponding to a first visual appearance with different corresponding visual characteristics, such as lower opacity and / or darker color) at the suggested location. In some embodiments, in response to a selection of a visual indication, the shadow of the text insertion cursor is no longer displayed. In some embodiments, new content input (e.g., text input) will cause the corresponding content to be displayed at the location of the text insertion cursor. Displaying a visual indication corresponding to the text insertion cursor prevents text editing from being entered in unwanted locations within the content (e.g., text) input area, thus reducing the computational load and power consumption required to handle and display erroneous operations.

[0266] In some implementations, the first user interface object includes content (718a), such as Figure 6AH The text 612 in the content. For example, the content is text displayed in a content (e.g., text) display area. In some embodiments, in response to detecting a corresponding object approaching but not touching the surface, based on determining that the corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to the content (718b) (e.g., as previously described with respect to step 704), based on determining that the content is non-editable content (e.g., the text is part of an image and is not text that can be edited (e.g., cannot be deleted or changed) in response to input such as from a virtual keyboard), the electronic device displays (718c) a visual indication of a content (e.g., text) selection cursor at the location in the content corresponding to the input device via a display generation component, for example... Figure 6AH The selection cursor preview 615 is displayed at the tip of the stylus 600. In some examples, the positioning of the content selection cursor corresponds to a portion of the input device (e.g., the tip). For example, this positioning corresponds to the tip of the input device projected (e.g., vertically projected) onto a surface, and the projected surface positioning optionally corresponds to a positioning within the content display area. Even if the text in the content is optionally not editable, it can optionally be selected (e.g., via highlighting) for subsequent operations (e.g., copy, paste, and / or cut). Displaying the content selection cursor in response to hover prevents continuous display of the content selection cursor, thus reducing the power and computational load required for such display.

[0267] In some implementations, in response to detecting a corresponding object approaching but not touching a surface, based on determining that the corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to the content (720a) (e.g., as previously described with respect to step 704), and based on determining that the content is editable content (e.g., text editable in response to input such as from a virtual keyboard), such as text 602, the electronic device abandons (720b) ​​displaying a visual indication of the content (e.g., text) selection cursor at the location in the content corresponding to the input device via a display generation component, such as... Figure 6AH The stylus 600 is hovering over text 602 instead of text 612. For example, the editable content is font-based or handwritten text and is displayed in an area including the content input area. In some embodiments, the content selection cursor is displayed when its location corresponds to non-editable text within the content, and the display of the content selection cursor stops as the input device moves to a location corresponding to the editable text (e.g., hovering above the surface). In some embodiments, a second cursor different from the content selection cursor, such as a text insertion cursor and / or a shadow of the text insertion cursor (e.g., as described relative to step 716), is displayed in response to the input device hovering over the editable content. Relinquishing the display of the content selection cursor in response to hovering over the editable text indicates that the text is editable, thereby reducing errors in the interaction between the input device and the text.

[0268] In some implementations, before detecting a corresponding object approaching but not touching the surface, a first user interface object is displayed with a first separation amount from the back panel (e.g., the back panel and / or background of the user interface), such as... Figure 6M The icon 604 is separated by a first amount from the back panel (e.g., the background of user interface 609) above which is displaying the content of user interface 609. In some embodiments, in response to detecting a corresponding object approaching but not touching the surface, based on determining that the corresponding object approaching the surface is an input device communicating with the electronic device and that the positioning of the input device corresponds to the first user interface object (e.g., as previously described relative to step 704), the electronic device displays (722b) the first user interface object with a second separation amount from the back panel greater than the first separation amount, such as... Figure 6NThe icon 604 is separated from the aforementioned back panel by a second amount. For example, the back panel of the user interface shares the same plane as the plane of the user interface displayed in a two-dimensional (or near-two-dimensional) environment (e.g., on a display device such as a computer monitor or smartphone). In some embodiments, displaying the first user interface object with a first separation amount from the back panel includes a non-zero or zero separation amount from the plane of the user interface. Displaying a non-zero separation optionally includes displaying shadows, borders (e.g., widths in different portions of the border), scales, applied lighting effects, and / or other visual characteristics of the first user interface object to convey the sense of separation and / or depth difference of the first user interface object relative to the plane of the user interface. In some examples, the user interface is a three-dimensional environment, and the first user interface object is separated from a flat or curved plane by a first separation amount (e.g., 0 cm, 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm). For example, the first user interface object is displayed in a mixed reality environment and occupies a space or position having an interval perceived as corresponding to the physical distance between a real-world object and a real-world plane (e.g., a flat or curved plane). In some implementations, in response to the detection that an input device is hovering over a first user interface object and / or a location on a surface determined to correspond to that user interface object, the first user interface object is displayed with a larger or smaller separation from the plane (e.g., 0 cm, 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm) from the plane (e.g., a flat or curved plane). If the corresponding object is not an input device and / or is not in the location corresponding to the first user interface object, the first user interface object is optionally abandoned from being displayed with a second separation from the back panel (e.g., and optionally, the first user interface object is displayed with a first separation from the back panel). Displaying the first user interface object with a variable separation indicates to the user the interactivity of the first user interface object, thus preventing erroneous input pointing to user interface areas outside the user interface object, thereby reducing the computational load and power consumption required to handle erroneous input.

[0269] In some embodiments, when a user interface including a first user interface object is displayed via a display generation component, the first user interface object has a first visual appearance, in which first visual characteristics (e.g., color, texture, fill, border, animation, shadow, and / or lighting effects) have first values, such as Figure 6S The visual appearance of icon 604 in the image, the electronic device detects (724a) the movement of the cursor from a position away from the first user interface object to the first input corresponding to the cursor moving from the first user interface object, such as from... Figure 6S to Figure 6TThe input is used to move the cursor 613 to the icon 604. For example, the first user interface object is an optional representation of a function that can be selected to activate the device, such as a button or icon. Such a function may optionally be to launch an application, display a menu for inputting content into a content input area, add an image or font-based text to the content input area, or display a menu for changing the markings made by simulated handwriting in the content input area. In some embodiments, the first visual appearance includes displaying the first user interface object with default colors, shading, fill, borders, animations, shadows, and / or lighting effects. In some examples, the cursor control device is a computer mouse, touchpad, stylus, hand, or finger worn with a peripheral device, and / or a gaze / gesture detection unit configured to cause movement or interaction with the peripheral device to change the positioning of the cursor optionally displayed in the user interface. The displayed cursor is optionally positioned at a location in two-dimensional or three-dimensional space within the user interface, and is optionally moved by subsequent movement or indication of the cursor control device to correspond to the location of the first user interface object displayed in the user interface (e.g., covering the location, within a threshold distance of the location (e.g., 0cm, 0.1cm, 0.3cm, 0.5cm, 1cm, 3cm, 5cm, 10cm, 30cm, 50cm, or 100cm), and / or occupying the location).

[0270] In some implementations, in response to detecting the first input, the electronic device moves (724b) the cursor to a first user interface object, such as from... Figure 6S to Figure 6T As shown, and displaying the first user interface object with a second visual appearance, in which the first visual characteristics have a second value different from the first value, such as Figure 6T The visual appearance of icon 604 in the interface. In some embodiments, in response to moving the cursor to correspond to a first user interface object, the first user interface object is optionally displayed with a second visual appearance (e.g., color, shading, fill, border, animation, shadow, and / or lighting effects). For example, before the cursor is moved to a button displayed in the user interface, the button is initially displayed as including a first set of graphics and / or fonts with various colors, and having an initial amount of shadow on an initial background (e.g., transparent, semi-transparent, or solid color background). In response to moving the cursor to correspond to the button, the background of the button is optionally displayed with a second set of visual values, including at least one or more different visual values ​​(e.g., shadow, lighting effects, background, and / or line width) for the same first visual characteristics. For example, borders and fills of different opacities but the same color are optionally displayed around and / or in the gaps within the first user interface object.

[0271] In some implementations, when the location of the input device corresponds to a first user interface object (e.g., as previously described relative to step 704), such as Figures 6N to 6O The positioning of the stylus 600 corresponds to icon 604, and the first user interface object is displayed with a third visual appearance (e.g., optionally the same as or different from the second visual characteristic), in which the first visual characteristic has a second value (724c). In some embodiments, if the positioning of an input device (e.g., optionally a device that is not context-dependent or generally unrelated to the cursor) corresponds to the first user interface object, the first user interface object is optionally displayed with a third visual appearance that includes a third set of visual values ​​(e.g., including the same set or subset of visual values ​​relative to the first and second visual values, but optionally including at least one or more different visual values). For example, visual values ​​include opacities different from the second and first opacity values, but with the same background color. In some examples, the third visual values ​​are a subset or superset of the second visual values, or vice versa. Displaying the user interface object with consistent visual changes across interactions with different input devices reduces erroneous interactions with electronic devices, and thus reduces the computational load and power consumption required to process these operations.

[0272] In some implementations, in response to the detection of a first input, the electronic device displays (726) the first user interface object with a parallax effect based on the movement of the cursor when the cursor is located on the first user interface object, such as... Figures 6U to 6V The parallax effect displayed relative to icon 604, wherein displaying the first user interface object in a third visual appearance based on determining that the input device's position corresponds to the first user interface object, does not include displaying the first user interface object in this parallax effect based on the movement of the input device when the input device's position corresponds to the first user interface object (e.g., as previously described relative to step 704), such as from Figures 6N to 6OThe displayed icon 604 lacks a parallax effect. For example, as described above, a third visual appearance including a third visual value includes a superset or subset of the second visual value. Possible visual appearances and characteristics have been previously described herein by way of non-limiting embodiments and are omitted for brevity. For example, the interaction of a stylus with a button (e.g., hovering an input device over a button) is the same as or nearly the same as the interaction of a comparable cursor with a button (e.g., hovering a cursor based on a cursor input device over a button). In some embodiments, hovering a cursor over a button includes displaying a first user interface object and / or its background area with a parallax effect, such that movement of the cursor causes movement of one or more portions of the button with a different amount than movement of another area of ​​the button (e.g., the background and / or border surrounding the button) in response to the movement of the cursor (e.g., to indicate that movement of the cursor is being detected, and further movement of the cursor will optionally cause the cursor to move away from the first user interface object). In some embodiments, when hovering an input device over a button, the visual appearance of the button is the same as when hovering with a cursor; however, no parallax effect is displayed in response to the detection of movement of the input device. In some implementations, the opposite is true: the parallax effect is not displayed when the cursor device is moved over the button, but is displayed when the input device is hovered over and moved at a location on the surface opposite the button. In some implementations, hovering the input device over the button includes displaying a different level of parallax effect than hovering the cursor device over the button. Although some implementations are described relative to the button, it should be understood that any suitable visual object (e.g., graphic objects, text objects, and animated objects) may optionally include such behavior. Displaying the parallax effect according to the type of input device reduces the computational power required to display such an effect for input device types that are not suitable for parallax effects, and avoids situations where the parallax effect increases the difficulty of selecting the button with a stylus (e.g., due to button movement).

[0273] In some implementations, in response to detecting a first input, the electronic device displays a first user interface (728) with a lighting effect based on the movement of the cursor when the cursor is over a first user interface object, such as relative to... Figures 6U to 6V Icon 604 in the diagram shows a lighting effect. For example, the lighting effect optionally includes a specular highlight applied to a portion of the first user interface object. This portion is optionally a border surrounding the area associated with the first user interface object.

[0274] In some implementations, displaying the first user interface object in a third-view appearance based on determining that the input device's position corresponds to the first user interface object does not include displaying the first user interface object with lighting effects based on the movement of the input device when the input device's position corresponds to the first user interface object (728) (e.g., as previously described relative to step 704), such as Figures 6N to 6O The icon 604 displayed lacks lighting effects. For example, as described relative to parallax effects, the lighting effects associated with a first user interface object (e.g., a button) optionally vary based on whether a cursor or input device is used to interact with the first user interface object. In some embodiments, the first user interface object displays lighting effects when the cursor corresponds to it, but optionally does not display such lighting effects when an input device corresponds to it. In some embodiments, the lighting effects (e.g., the brightness value of a portion of a simulated light source positioning or specular highlight) are modified in response to cursor movement. In some embodiments, the opposite is true: the lighting effects are displayed in response to interaction with the input device rather than interaction with the cursor. Optionally, the lighting effects are displayed in response to determining that the cursor device and the input device correspond to the first user interface object, respectively. In some embodiments, the lighting effect is a specular highlight applied to a portion (e.g., a border or section) of an area in the user interface that includes the first user interface object. Displaying lighting effects according to the type of input device reduces the computational power required to display lighting effects for input device types that are not suitable for such effects.

[0275] In some implementations, the first user interface object corresponds to a link to content (730a) (e.g., a web link to web-based content), such as Figure 6AF Link 610 in the document. In some embodiments, in response to detecting a corresponding object approaching but not touching the surface, based on determining that the corresponding object approaching the surface is an input device communicating with the electronic device and that the location of the input device corresponds to a first user interface object (e.g., as previously described relative to step 704), the electronic device modifies (730b) the visual appearance of the first user interface object, such as... Figure 6AFThe visual appearance of link 610 is modified. For example, the content is a website, application, media, or other user interface environment associated with the link. In some embodiments, the first user interface object is a graphic or text object and is displayed with a modified visual appearance in response to a hover event, such as highlighting, bolding, underlining, and / or other appropriate visual emphasis. For example, the first user interface object is text associated with a hyperlink (e.g., pointing to a webpage), and the modification includes highlighting a portion of the user interface including the text (e.g., based on the outline of the text or an area of ​​surrounding text), bolding, and / or underlining the text. Additionally, the text may optionally be enlarged (e.g., scaled up or enlarged in font) to further convey potential interaction with the text. In some embodiments, while displaying the modified visual appearance, in response to receiving a selection of a first selectable option (e.g., contacting the surface with the tip of an input device at the location of the first selectable option), a function corresponding to the first user interface object is initiated. For example, launching an application from the system user interface (e.g., if the first selectable option is an application icon displayed on the home screen user interface of an electronic device, such as a reference). Figure 4A (as described above) and / or displaying web pages associated with graphics or text links (optionally stopping the display of the user interface that appears when a selection is detected). Displaying the first user interface object with a modified visual appearance conveys the interactivity of the underlying object, and thus reduces the input required to initiate operations associated with the user interface object and avoids errors in interaction with the user interface object, thereby reducing the computational load and power consumption originally required to initiate such operations.

[0276] It should be understood that, Figures 7A to 7G The specific order in which the operations described herein are presented is merely exemplary and not intended to indicate that the order is the only order in which the operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Furthermore, it should be noted that the details of other processes described herein with respect to other methods (e.g., methods 900, 1100, and 1300) are similarly applicable to the above descriptions with respect to other methods described herein (e.g., methods 900, 1100, and 1300). Figures 7A to 7G The method 700. For example, the interaction between the input device and the surface, the response of the electronic device, the virtual shadow of the input device and / or the input detected by the electronic device and / or the input detected by the input device optionally have one or more of the characteristics of the interaction between the input device and the surface, the response of the electronic device, the virtual shadow of the input device and / or the input detected by the electronic device and / or the input detected by the electronic device as described herein with reference to other methods (e.g., methods 900, 1100, and 1300). For the sake of brevity, these details will not be repeated here.

[0277] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., combined with...). Figures 1A to 1B , Figure 3 , Figures 5A to 5I (as described above) or a dedicated chip. Furthermore, see the above references. Figures 7A to 7G The operation is optionally performed by Figures 1A to 1B The components depicted are used to implement this. For example, display operations 702a and 702d and detection operation 702b are optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. The event handler 190 optionally utilizes or invokes a data updater 176 or an object updater 177 to update the application's internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will readily understand how this can be achieved based on... Figures 1A to 1B The components described herein are used to implement other processes.

[0278] Provide feedback on the posture of the input device.

[0279] Users interact with electronic devices in a variety of ways, including using input devices such as styluses. In some implementations, the electronic device receives input from such input devices based on the relative posture (e.g., orientation and / or positioning) of the input device with respect to a surface it interacts with (e.g., touches and / or hovers over). The implementations described below provide methods for the electronic device to provide feedback on the posture of the input device relative to a surface, thereby enhancing interaction with the device. Enhanced interaction with the device reduces the amount of time required for the user to perform actions, thereby reducing the device's power consumption and extending the battery life of battery-powered devices. It should be understood that people use devices. When a person uses a device, that person is optionally referred to as the user of the device.

[0280] Figures 8A to 8AF Exemplary methods for displaying an indication of the posture of an input device relative to a surface are illustrated in some embodiments of this disclosure. The embodiments in these figures are used to illustrate the processes described below, including references to... Figures 9A to 9K The process described.

[0281] Figure 8AA first set of exemplary simulated shadows 832 displayed by an electronic device are illustrated, corresponding to different orientations of the input device 800 (e.g., assuming a virtual light source is located above surface 852) relative to the surface. Surface 852 optionally corresponds to a touchscreen of the electronic device, but may also be other surfaces, such as those described in reference method 800. In portions 858a, 860a, and 862a, the z-axis 850 points outward from surface 852 (e.g., in a direction perpendicular to the plane of surface 852), the x-axis is parallel to surface 852, and the y-axis is perpendicular to the x-axis and also parallel to surface 852. Portions 858a, 860a, and 862a correspond to different orientations of the input device 800 relative to surface 852, and corresponding portions 858b, 860b, and 862b respectively illustrate example simulated shadows 832 displayed by the electronic device in response to detecting that the input device 800 is in these orientations. In some embodiments, the simulated shadow 832 displayed by the electronic device is characterized relative to one or more thresholds, such as thresholds 802 and 854. For example, if the input device 800 (or the tip of the input device 800) is more than a threshold distance 802 from the surface 852, the electronic device optionally does not display a simulated shadow. If the input device 800 (or the tip of the input device 800) is less than a threshold distance 802 from the surface 852, the electronic device optionally displays a simulated shadow of the input device 800 based on the position and / or orientation of the input device 800 relative to the surface 852.

[0282] For example, in Figure 8AIn portion 858a, the input device 800 is perpendicular to the surface (e.g., within a threshold angle 854 of the normal). When the input device 800 is perpendicular to the surface and / or within the threshold angle 854 of the normal, the electronic device optionally does not display an analog shadow of the input device, as shown in portion 858b. In contrast, in portion 860a, the tilt of the input device 800 is a first amount that is greater than the threshold angle 854 relative to the normal of the surface 852, such as 15 degrees, 20 degrees, 25 degrees, 30 degrees, or 35 degrees. In response, the electronic device displays an analog shadow 832 of the input device 800 having a first intensity level (e.g., a first blur level, a first shadow diffusion level, and / or a first opacity level), as shown in portion 860b. In portion 862a, the tilt of the input device 800 relative to the normal 850 is greater than the tilt of the input device 800 in portion 860a. Compared to the simulated shadow 832 in portion 860b corresponding to the tilt of input device 800 in portion 860a, the simulated shadow 832 in the corresponding portion 862 has lower intensity blur, less shadow spread, and increased opacity. In some embodiments, the intensity of the visual representation of the simulated shadow 832 varies in response to changes in the tilt of input device 800. For example, if the tilt of input device 800 in portion 862a is greater than that in portion 860a, its corresponding simulated shadow 832 in 862b is more intense (e.g., darker and / or more defined) than the simulated shadow 832 in 860b, while the simulated shadow 832 included in 860b is lighter and more blurred. Furthermore, in some embodiments, as the tilt of input device 800 relative to normal 850 decreases, the length of simulated shadow 832 shortens, such as... Figure 8A Parts 860b and 862b are shown in the figure.

[0283] Figure 8B A second set of exemplary simulated shadows 832 is illustrated, which change visually in response to changes in the distance between the input device 800 and the surface. Figure 8B In this implementation, the tilt of the input device 800 relative to the normal 850 remains constant, while the distance of the input device 800 relative to the surface 852 changes as the input device 800 moves away from the surface (e.g., from no distance (or contact with the surface) to distance 870 and then to distance 872). In some embodiments, if the input device 800 (e.g., the tip of a stylus) moves to a predefined threshold distance 802 or beyond, a simulated shadow 832 is not displayed (e.g., not included in the user interface), similar to... Figure 8A Part 858b is shown in the diagram. Figure 8BAs shown, the visual appearance of the simulated shadow 832 changes in response to changes in the distance from the input device 800. For example, in part 864a, the input device 800 is in contact with the surface 852 (e.g., there is little or no distance between the tip of the input device 800 and the surface 852). When the input device 800 is in contact with the surface 852, the electronic device displays a simulated shadow of the input device with a first intensity level (e.g., a first blur level, a first shadow diffusion level, and / or a first opacity level), as shown in part 864b. In contrast, in part 866a, there is a first distance between the tip of the input device 800 and the surface 852, such as 0.1 cm, 0.2 cm, 0.5 cm, 0.8 cm, 1 cm, 3 cm, or 5 cm. In response, the electronic device displays a simulated shadow 832 of the input device 800 with a second blur level greater than the first blur level, a second shadow diffusion level greater than the first shadow diffusion level, and / or a second opacity level greater than the first opacity level, as shown in part 866b. In part 868a, the distance 872 between the tip of the input device 800 and the surface 852 is greater than the distance 870 in part 866a. The intensity of the simulated shadow 832 corresponding to part 868a includes a third blur level greater than the second blur level, a third shadow diffusion level greater than the second shadow diffusion level, and / or a third opacity level greater than the second opacity level, as shown in part 868b. For example, in part 868b, the opacity of the simulated shadow 832 is lower when the tip of the input device 800 is at a distance 872 from the surface 852, while in part 866b, the opacity of the simulated shadow 832 is higher when the tip of the input device 800 is at a distance 870 from the surface 852. Compared to parts 868b and 866b, the opacity of the simulated shadow 832 is even higher when the input device 800 is in contact with the surface 852 (e.g., there is no distance between the tip of the input device and the surface), as shown in part 864b.

[0284] like Figure 8A , Figure 8B and / or Figure 8C As shown, the direction and / or orientation of the simulated shadow 832 will change as the orientation of the input device 800 relative to the surface 852 changes. Figure 8C A third set of exemplary simulated shadows 832 is illustrated, wherein the input device 800 is in contact with the surface 852 and positioned in a downward and rightward orientation, its orientation being consistent with... Figure 8B The downward and leftward orientations shown are different. For example... Figure 8B and Figure 8CAs shown, the orientation of the simulated shadow 832 changes accordingly with the orientation of the input device 800 relative to the surface 852 (for example, when the tip of the input device 800 is facing down and to the right, the tip of the simulated shadow 832 is also facing down and to the right, and when the tip of the input device 800 is facing down and to the left, the tip of the simulated shadow 832 is also facing down and to the left).

[0285] Figure 8C The variation in simulated shadow 832 is also illustrated as the tilt of input device 800 relative to normal 850 changes. For example, in section 874a, input device 800 is within a threshold angle 854 of the normal, although... Figure 8A The tilt angle of the input device 800 in portion 858a is different. When the input device 800 is within a threshold angle 854 of the normal 850, the electronic device optionally does not display an analog shadow 832 of the input device, as shown in portion 874b. In contrast, in portion 876a, the tilt angle of the input device 800 is a first amount that is greater than the threshold angle 854 relative to the normal 850 of the surface 852, such as 15 degrees, 20 degrees, 25 degrees, 30 degrees, or 35 degrees. In response, the electronic device displays an analog shadow 832 of the input device 800 with a first intensity level (e.g., a first blur level, a first shadow diffusion level, and / or a first opacity level), as shown in portion 876b. In portion 878a, the tilt angle of the input device 800 is greater than that of the input device 800 in portion 876a. Compared to the simulated shadow 832 in portion 876b corresponding to the tilt of the input device 800 shown in portion 876a, the simulated shadow 832 in portion 878b corresponding to portion 878a has lower intensity blur, less shadow spread, and / or increased opacity. For example... Figures 8A to 8C As illustrated in some implementations, the simulated shadow changes in blurriness, length, intensity, opacity, size, and / or color in response to one or more of the tilt, orientation, and / or distance of the input device 800 relative to the surface 852.

[0286] In some implementations, the electronic device displays details of the analog shadow 832 for the input device 800, such as... Figures 8A to 8C The descriptions and illustrations may optionally be applied to one or more or all of the simulated shadows illustrated and described in reference methods 700, 900, 1100 and / or 1300.

[0287] Figure 8D An exemplary device 500 (corresponding to object 500 of illustrative symbol 804) including a touchscreen 504 is illustrated. Device 500 is optionally... Figures 8A to 8C The electronic devices referenced in the description. For example... Figure 8DAs shown, electronic device 500 is displaying a main screen user interface 890. The main screen user interface 890 includes one or more virtual objects (e.g., virtual objects 826 and 828). See reference... Figures 8D to 8K The virtual objects 826 and 828 are representations of application icons, such as application icon 826 and application icon 828, which can be selected to cause device 500 to display and / or launch the corresponding application. Figure 8D A symbol 804, including a side view of device 500, is also illustrated. Symbol 804 indicates a relative posture, including the distance of input device 800 relative to a surface of device 500 (e.g., touchscreen 504). The symbol also includes two thresholds. Threshold 802 is a first distance threshold from the surface of device 500 (e.g., 0.3cm, 0.5cm, 1cm, 3cm, 5cm, 10cm, 20cm, 50cm, or 100cm). Threshold 830 is a second, smaller distance threshold from the surface of device 500 (e.g., 0.1cm, 0.3cm, 0.5cm, 1cm, 3cm, 5cm, 10cm, 20cm, or 50cm). As shown and described later, device 500 optionally displays a virtual shadow and / or other indications in response to the positioning of the input device relative to thresholds 802 and 830.

[0288] It should be understood that although virtual objects 826, 828 and user interface 890 are illustrated as being displayed on touchscreen 504, virtual objects 826, 828 and user interface 890 are optionally displayed on a head-mounted display including display generation components that present these items to the user in a computer-generated environment (e.g., an extended reality environment or a three-dimensional environment). In some embodiments, virtual objects 826, 828 and user interface 890 are displayed on a physical surface on which these items are projected, or on a virtual surface corresponding to at least a portion of these items.

[0289] exist Figure 8D In this case, the distance between input device 800 and device 500 exceeds the threshold 802, therefore device 500 will not display the virtual shadow corresponding to input device 800. Figure 8E An example is illustrated where input device 800 moves such that it falls within a threshold distance 802 on the surface of device 500. In response to device 500 detecting that input device 800 has moved below the threshold 802 in icon 804, device 500 displays a virtual shadow 832 with a visual appearance corresponding to the input device, as shown in the reference. Figures 8A to 8C As described above. In some embodiments, when the input device 800 (e.g., the tip of a stylus) is above a threshold 802, the device 500 stops displaying the virtual shadow 832 in the user interface 890, such as... Figure 8DAs shown. Therefore, in some embodiments, even if the input device 800 is located above the virtual object 826, the user interface displayed by the device 500 will not include the virtual shadow 832 if the input device is not within a threshold distance 802 of the surface. Back Figure 8E , Figure 8E It is also illustrated that device 500 presents virtual object 826 in a way that visually distinguishes it from other virtual objects (e.g., virtual object 828) (e.g., app icon 826 is enlarged) to indicate that the current focus of input device 800 is on virtual object 826 (e.g., virtual object 826 will be selected if device 500 detects that the input device is in contact with the surface of touch screen 504).

[0290] Figure 8F An example is illustrated where input device 800 moves beyond a lateral threshold distance (e.g., 0.1cm, 0.3cm, 0.5cm, 1cm, 3cm, 5cm, or 10cm) from virtual object 826, and in response to device 500 detecting that input device 800 has moved away from virtual object 826, device 500 displays virtual object 826 restored to its original size, such as... Figure 8F As shown. Therefore, in some embodiments, even if the input device 800 is within a threshold distance 802 of the surface shown by the icon 804, if the input device is not within a lateral threshold distance of the virtual object 826, the user interface 890 displayed by the device 500 does not include the virtual object 826 which has focus (indicated at least in part by its magnified visual appearance) (e.g., can be selected to initiate a process associated with the virtual object 826).

[0291] Figure 8F An example is also illustrated whereby device 500 displays a specific portion of virtual shadow 832 in user interface 890 when input device 800 is within (or below) a threshold distance 830 as illustrated by symbol 804. Figure 8G As shown, the virtual shadow 832 includes a first portion (e.g., virtual shadow portion 832a) corresponding to the body of the input device 800 and a second portion (e.g., virtual shadow portion 832b) corresponding to the tip of the input device 800. The virtual shadow portion 832b optionally indicates the position where the tip of the input device 800 will make contact in the user interface 890 if the input device 800 is brought closer to the surface of the touchscreen 504, thereby providing feedback to the user about how the input device 800 interacts with the device 500. Furthermore, the virtual shadow portion 832b is optionally visually distinguishable from the virtual shadow portion 832a (e.g., darker, more intense, and / or clearer).

[0292] Figure 8GAn example is illustrated where, when input device 800 is within a threshold distance 830, the input device 800 moves so that it is again within a lateral threshold distance of virtual object 826. In response to device 500 detecting that input device 800 has moved within the lateral threshold distance of virtual object 826, electronic device 500 changes the visual appearance of virtual shadow portion 832b to the shape of a cursor, such as a circular cursor (represented by a small circle on virtual object 826), indicating that the current focus of the input device is on virtual object 826. In some embodiments, Figure 8G The size or shape of the circular cursor is not based on the size or shape of the virtual object 826. Additionally or alternatively, in some embodiments, the electronic device 500 changes the visual appearance of the virtual shadow portion 832b based on the size and / or shape of the virtual object having focus on the input device 800. For example, as... Figure 8H As shown, the virtual object 828 has the focus of the input device 800 and is an enlarged square. Therefore, the device 500 changes the visual appearance of the tip of the virtual shadow portion to an enlarged square, providing a visual effect for the virtual object 828 including shadow and / or highlight display.

[0293] Figures 8I to 8K An example of a change in the tilt angle 836 of the input device 800 relative to the surface of the touchscreen 504 is illustrated, and in response to the change in tilt angle, the device 500 updates the visual appearance of the virtual shadow 832. Figures 8I to 8K In the process, the orientation, positioning, and distance of the input device 800 relative to the surface remain constant, while the tilt of the input device 800 changes as the tilt angle approaches perpendicularity to the surface (or perpendicularity to device 500). (e.g., from...) Figure 8I The inclination of the middle is 836 to Figure 8J The inclination of 836 to Figure 8K (without tilt). For example, such as Figures 8I to 8K As shown, as the tilt of the input device 800 gets closer to the normal (e.g., closer to the threshold angle 854), the device 500 alters the visual appearance of the virtual shadow 832 (e.g., from...). Figure 8I Virtual shadows in 832 Figure 8J Virtual Shadows 832 and beyond Figure 8KThe virtual shadow 832 is stopped from being displayed. In some embodiments, in response to detecting that the tilt of the input device decreases to zero (e.g., when the input device is greater than a threshold angle 854 relative to the normal), the device 500 gradually weakens and / or reduces the visual appearance of the virtual shadow 832 until the virtual shadow 832 is no longer displayed in the user interface. Conversely, in some embodiments, in response to detecting that the tilt of the input device relative to the normal increases to 90 degrees as the tilt of the input device becomes closer to parallel to the surface, the device 500 gradually increases the intensity and / or visual appearance of the virtual shadow 832. Figures 8I to 8K As shown, in some implementations, as the input device 800 is tilted to an angle closer to the normal of the touchscreen 504 surface (but greater than the threshold angle 854 of the normal), the virtual object 832 changes its opacity (e.g., decreases opacity), size (e.g., decreases size), and / or color (e.g., lightens in color)). For more information on the visual appearance changes of the virtual shadow 832 based on the pose changes of the input device 800 relative to the surface, please refer to [link to relevant documentation]. Figures 8A to 8C .

[0294] In some implementations, user interface 890 is a user interface for a drawing application, or a user interface for content drafting using input device 800. In some implementations, the drawing application is an application installed on device 500. Figures 8L to 8AF As shown, the user interface 890 includes one or more virtual objects (e.g., virtual object 844). Figure 8L The virtual object 844 is a content input palette that includes one or more selectable options associated with the content. For example, the content input palette 844 includes options for selecting a drawing tool (e.g., a content input tool) simulated by the input device 800, options for undoing or redoing (e.g., re-performing) a recent content input-related operation, options for changing the content color, and / or options for selecting a virtual keyboard for entering text. In some embodiments, possible drawing tools for the input device 800 include a text input tool, a pen input tool, a highlighter (or marker) input tool 810, a pencil input tool, an eraser tool, and / or a content selection tool.

[0295] In some implementations, when the input device 800 (e.g., the tip or other representative portion of the input device) is more than a threshold 830 away from the surface, the device 500 displays a virtual shadow 832 with a first visual appearance; when the input device 800 is less than the threshold 830 away from the surface, the device 500 displays a virtual shadow 832 with a second visual appearance different from the first visual appearance. For example, in Figure 8LWhen device 500 detects that input device 800 is below threshold distance 802 but above threshold distance 830, device 500 displays a virtual shadow 832 including a first virtual shadow portion 832a corresponding to the cylinder of input device 800. Figure 8M An example is illustrated whereby device 500 displays a second virtual shadow portion 832b corresponding to the tip of input device 800 when device 500 detects that input device 800 is below a threshold distance 830. In some embodiments, the second virtual shadow portion 832b indicates the location on the user interface 890 that the input device will touch (or mark) before the input device 800 touches (or contacts) the surface of touchscreen 504. In some embodiments, device 500 presents a second virtual shadow portion 832b having a shape and / or color corresponding to the tip of a currently selected drawing tool, which is simulated by input device 800. For example, in Figure 8M In this context, the currently selected drawing tool is the marker input tool 810, and the selected color is black, as indicated by the color picker tool 814. In response to device 500 detecting that input device 800 has moved below threshold 830, because device 500 has determined that the currently selected drawing tool is the marker input tool 810, device 500 presents a second virtual shadow portion 832b of virtual shadow 832 with a visual appearance corresponding to the marker input tool. For example, the second virtual shadow portion 832b is a black rectangle corresponding to the active color (e.g., selected from the color picker tool 814) and the chisel tip of the marker input tool 810, as shown... Figure 8M As shown. In some embodiments, when the tip of the input device 800 contacts and moves on the surface, the device 500 updates the user interface to display handwriting and / or lines that are black in color and rectangular in shape (e.g., corresponding to the color and / or shape of the second virtual shadow portion 832b).

[0296] In some implementations, device 500 detects a gesture indication (e.g., one or more taps) on input device 800 and interprets the gesture indication as a request to initiate an operation. For example, in Figure 8N In the process, device 500 detects a gesture indication (e.g., indicated by 816) detected by input device 800, which corresponds to a request to change the currently selected drawing tool from marker tool 810 to pen input tool 818, and in response to detecting the request to change the currently selected drawing tool, device 500 selects pen tool 818 as the currently selected drawing tool, as shown below. Figure 8OAs shown. In some implementations, the input for changing the currently selected drawing tool is any appropriate input for achieving such a change, such as voice input, touch input on touchscreen 504, etc. In response to selecting the pen input tool 818, the device displays a virtual object 844 in the user interface 890, wherein the pen input tool 818 is the currently selected (e.g., active) tool, and a second virtual shaded portion 832b has a visual appearance corresponding to the pen input tool 818 (e.g., a rounded bullet shape), such as... Figure 8O As shown. In addition to changing the shape of the second virtual shadow portion 832b, the device 500 also changes the color of the second virtual shadow portion 832b to a color corresponding to the active color (e.g., selected from the color picker 814), which is gray. In some embodiments, when the tip of the input device 800 contacts and moves across the surface, the device 500 updates the user interface to display the handwriting and / or lines in the active color (e.g., the color corresponding to the second virtual shadow portion 832b).

[0297] Go to Figure 8P In some implementations, when the input device 800 is detected to be within a threshold distance 830, the device 500 detects a gesture indication (or other suitable input, such as voice input, touch input on the touchscreen 504, etc.) on the input device 800, which corresponds to a request to change one or more drawing settings of the input device 800. In response to the detected request to change one or more drawing settings of the input device 800, the device 500 displays a content input user interface element 840 (optionally positioned based on the positioning of the input device 800 relative to a surface) at or near the second virtual shaded area 832b, such as... Figure 8P As shown. The content input user interface element 840 includes one or more selectable options for changing one or more drawing settings (e.g., opacity and / or thickness level) associated with the currently selected drawing tool. Device 500 detects input that changes the line thickness level from level 846 (e.g., thinnest) to level 848 (thickest), as shown. Figure 8Q As shown. In response to the determined change, device 500 changes the visual appearance of the second virtual shadow portion 832b to correspond to the change in line thickness level (e.g., from...). Figure 8P The tiny tip in the middle Figure 8Q (The thick tip in the middle). In some embodiments, when the tip of the input device 800 contacts and moves on the surface, the device 500 updates the user interface to display the handwriting and / or lines with active line thickness (e.g., corresponding to the size and / or thickness of the second virtual shading portion 832b), and / or propagates the drawing settings to the already drawn handwriting and / or lines.

[0298] exist Figure 8R In the process, device 500 detects a gesture instruction 816 (or other suitable input, such as voice input, touch input on touchscreen 504, etc.) corresponding to a request to change the color of the currently selected drawing tool from gray to black, and in response to detecting the request to change the color of the currently selected drawing tool, device 500 changes the active color from gray to black. Figure 8R The gray shown in the color picker tool 814 is changed to Figure 8S The black shown in the color picker 814 is even if the input device 800 is more than a threshold distance 802 from the surface of the touchscreen 504. Therefore, compared to Figure 8Q The second virtual shading portion 832b is displayed when the tip of the input device 800 is below the threshold 830 and when input that changes the line thickness of the currently selected drawing tool is detected. Figure 8R and Figure 8S In this case, the input device 800 is more than a threshold distance 802 away from the surface of the touch screen 504, so the device 500 will not display the virtual shadow 832 of the input device 800 when it detects an input that changes the color of the currently selected drawing tool.

[0299] exist Figure 8T In this process, device 500 detects that input device 800 is below threshold 830 and within a lateral threshold distance of selectable virtual object 822, which can be selected to create a new drawing in a drawing application. In response, device 500 displays a virtual shadow 832 of input device 800 as previously described, and the visual appearance of the virtual shadow portion 832b is based on the shape and / or size of the selectable virtual object 822. Figure 8T As shown, since the selectable virtual object 822 has a focal point (e.g., the input device 800 is within a lateral threshold distance of the selectable virtual object 822), the visual appearance of the virtual shadow portion 832b is changed to a shape similar to or based on the shape of the selectable virtual object 822 (e.g., a square), thereby presenting a visual appearance in which the selectable virtual object 822 includes shadows and / or highlights. In some embodiments, the visual appearance of the first portion of the virtual shadow corresponding to the cylinder of the input device 800 (e.g., virtual shadow portion 832a) is not based on the currently selected drawing tool (e.g., having the same visual appearance across different selected drawing tools).

[0300] Figure 8UAn example is illustrated where, when the input device 800 is within a lateral threshold distance of the selectable virtual object 822, the input device 800 moves such that the tip of the input device 800 contacts the surface at a position corresponding to the position of the selectable virtual object 822. In response to the device 500 detecting that the input device 800 is in contact with the surface at the position corresponding to the position of the selectable virtual object 822, the electronic device 500 alters the visual appearance of the selectable virtual object 822 and / or the virtual shadow portion 832b (by comparison). Figure 8T The visual appearance of the selectable virtual object 822 is indicated by deeper shadows and / or highlights to indicate selection of the virtual object 822. In response to such selection, the device 500 optionally displays a blank drawing canvas in the user interface 890.

[0301] exist Figure 8V to Figure 8AF In this embodiment, the user interface 890 includes a content input area 812. In some embodiments, the content input area 812 is configured to receive handwritten input (e.g., drawing input via input device 800) and display a representation of that handwritten input (e.g., if drawing input is provided) and / or display font-based text (e.g., if font-based text input is provided and / or if the handwritten input is converted to font-based text based on the currently selected drawing tool of input device 800). In some embodiments, such as Figure 8V As shown, selecting text input tool 820 as the currently selected drawing tool will cause device 500 to enter text input mode. In this text input mode, handwritten input drawn in content input area 812 will be analyzed into text characters, recognized, and converted into font-based text in content input area 812. Figure 8V In this case, the input device 800 is located within a threshold distance 802 of the surface of the touch screen 504, but is more than a threshold distance 830 from the surface of the touch screen. Therefore, the device 500 displays a virtual shadow 832 of the input device 800.

[0302] Figure 8WAn example is illustrated where, when the input device is below threshold 830, the device continues to display a virtual shadow 832 corresponding to the currently selected drawing tool (i.e., text input tool 820), but does not display the tip portion of the virtual shadow as described above (e.g., virtual shadow portion 832b, which has a shape and / or color corresponding to the tip of the selected drawing tool simulated by input device 800). In some embodiments, when the currently selected drawing tool is text input tool 820, handwritten input and the corresponding (e.g., converted) font-based text are displayed with default color and / or line thickness. Therefore, when text input tool 820 is the currently selected drawing tool, the color and / or line thickness of the provided handwritten input are optionally independent of text input tool 820, and device 500 does not display a virtual shadow portion 832b having a color and / or shape corresponding to text input tool 820.

[0303] Figure 8X An example is illustrated where, when input device 800 is below threshold 830, the input device 800 is moved such that it is within a lateral threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, or 10 cm) of the content (e.g., text) within the content input area 812. In response to device 500 detecting that input device 800 is within the threshold distance of the content, electronic device 500 changes the visual appearance of the second virtual shadow portion 832b of virtual shadow 832 to a text insertion cursor. For example, when the tip of input device 800 is located within the lateral threshold distance of the content within the content input area 812, the second virtual shadow portion 832b is optionally replaced with an indication of a text insertion cursor (e.g., a simulated shadow on the cursor). In some embodiments, the vertical positioning of the virtual shadow portion 832b in the content is aligned to the content and / or text line nearest the tip of input device 800, such as... Figure 8X As shown. Furthermore, the horizontal positioning of the virtual shadow portion 832b within the content optionally corresponds to the horizontal positioning of the tip of the input device 800.

[0304] For example, in Figure 8Y In the process, device 500 detects that input device 800 has moved upward in user interface 890, and accordingly displays the virtual shaded portion 832b, including the text insertion cursor, from the lower line text in content input area 812 to the middle line text. For example... Figure 8X and Figure 8YAs shown, the movement of the text insertion cursor within the content corresponds to the movement of the input device 800. For example, the text insertion cursor is displayed when the tip of the input device 800 is above a specific character of the font-based text in the content input area 812 (e.g., the text insertion cursor is positioned to the nearest character of the font-based text without requiring the user to move the tip of the input device 800 more precisely to the specific character). Therefore, in some embodiments, the vertical and / or horizontal positioning of the virtual shading portion 832b and / or the text insertion cursor is separated from the actual positioning in the user interface 890 corresponding to the tip positioning of the input device 800, because the device 500 automatically aligns the virtual shading portion 832b and / or the text insertion cursor to the nearest line and / or character of the content in the content input area 812.

[0305] exist Figures 8Z to 8AF In this embodiment, user interface 890 includes one or more virtual objects (e.g., virtual objects 842 and 842a). Virtual object 842 is a content-aligned user interface element (e.g., a virtual ruler), which includes one or more virtual objects 842a (e.g., guide points). In some embodiments, device 500 aligns a second virtual shadow portion 832b to the nearest virtual object 842a to facilitate the automatic drawing of aligned lines based on the content-aligned user interface element. Figure 8Z In the illustrated example, device 500 detects that input device 800 is below threshold 802 (e.g., the tip of the input device is within a threshold distance 802 from the surface), and in response, device 500 displays virtual shadow 832, as previously described. Figure 8AA An example is illustrated where the input device 800 is moved to a position below a threshold 830 (e.g., the tip of the input device is within a threshold distance 830 from the surface). In response to device 500 detecting that the input device 800 is below the threshold 830, electronic device 500 alters the visual appearance of virtual shadow 832 to include a second virtual shadow portion 832b corresponding to the tip of the currently selected drawing tool (e.g., pen tool 818), as previously referenced. Figures 8N to 8Q As stated above.

[0306] Figures 8AB to 8AD This illustrates the contact and movement of the tip of input device 800 along the guide line of virtual object 842. In response to the contact and movement of input device 800, device 500 generates a line at the position of the guide line of virtual object 842 according to the movement of input device 800, such as... Figures 8AB to 8ADAs shown. In some embodiments, during the movement of the input device 800, the virtual shaded portion 832b is vertically aligned with the horizontal guide line (e.g., when the tip of the input device 800 is within a lateral threshold distance of the horizontal guide line, such as 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, or 5 cm) to facilitate the drawing of a straight line according to the movement of the input device 800, even if the tip of the input device 800 is not located at any of the positions on the guide line.

[0307] from Figures 8AD to 8AE The input device 800 further moves to the right to object 842a. In response, such as... Figure 8AE As shown, the second virtual object portion 832b corresponding to the tip of the currently selected drawing tool is aligned to the nearest virtual object 842a because this virtual object is the closest guide point to the position of the input device 800, and / or because the tip of the input device 800 has moved within a lateral threshold distance (e.g., 0.1cm, 0.3cm, 0.5cm, 1cm, 3cm, or 5cm) of the virtual object 842a. In some embodiments, in conjunction with aligning the virtual shading portion 832b to the object 842a, the device 500 also completes the drawing of a line from the left square to the object 842a via the input device 800, and this line is optionally aligned based on the alignment provided by the alignment object 842 (e.g., relative to the actual path traversed by the input device 800). In some embodiments, the device 500 maintains the display of the second virtual object portion 832b at the virtual object 842a as long as the device 500 detects movement of the input device 800 within a lateral threshold distance of the virtual object 842a. Figures 8AE to 8AF As shown. In some embodiments, when device 500 detects input device 800 within a lateral threshold distance of one or more guide points, device 500 traverses one or more guide lines to automatically generate lines aligned with the guide lines along one or more guide points (e.g., alignment relative to the actual path traversed by input device 800). Therefore, in some embodiments, even if the second virtual object portion 832b is a partial virtual object 832, if the input device is within a lateral threshold distance of a content alignment guide (e.g., virtual object 842a), device 500 displays the second virtual object portion 832b that is offset from (or separated from) virtual object 832.

[0308] Figures 9A to 9K This is a flowchart illustrating a method 900 for providing feedback on the posture of an input device relative to a surface. Method 900 is optionally performed on electronic devices (such as device 100, device 300, and device 500), as referenced above. Figures 1A to 1B , Figures 2 to 3 , Figures 4A to 4B and Figures 5A to 5IThe operations in method 900 may be optionally combined, and / or the order of some operations may be optionally changed.

[0309] As described below, method 900 provides a way to provide feedback on the posture of an input device relative to a surface. This method reduces the cognitive burden on the user when interacting with the device user interface of this disclosure, thereby creating a more efficient human-machine interface. For battery-powered electronic devices, improving the efficiency of user interaction with the user interface saves power and increases the time between battery charging cycles.

[0310] In some embodiments, method 900 is performed at an electronic device that communicates with a display generating component, one or more sensors (e.g., a touch-sensitive surface), and an input device. For example, the electronic device is a mobile device (e.g., a tablet, smartphone, media player, or wearable device) including a touchscreen and wireless communication circuitry, or a computer including one or more of a keyboard, mouse, touchpad, and touchscreen, and wireless communication circuitry, and optionally has one or more of the characteristics of the electronic device of method 700. In some embodiments, the display generating component has one or more characteristics of the display generating component in method 700. In some embodiments, the input device has one or more characteristics of one or more input devices in method 700. The touch-sensitive surface (e.g., a touch-sensitive surface) is used in the process of display generating components. Figure 1A The touch-sensitive display system 112 or Figure 4A In some embodiments of the touchscreen 112, an input device (e.g., a stylus communicating with an electronic device, such as the stylus described in reference methods 700, 900, 1100 and / or 1300) is detected hovering over the touch-sensitive surface so that when the input device is detected hovering over the touch-sensitive surface, movement of the input device relative to the touch-sensitive surface changes the representation of a virtual shadow corresponding to the input device, as described in method 900 herein. In some embodiments, one or more sensors are optionally included. Figure 1A One or more sensors in the system.

[0311] In some embodiments, the electronic device displays a user interface (902a) via a display generating component, such as... Figures 8D to 8AFThe user interface 890 in the method 700. For example, the user interface of an application installed and / or running on the electronic device, or the user interface of the electronic device's operating system. In some embodiments, the user interface is the main screen user interface of the electronic device, or the user interface of an application accessible to the electronic device's operating system, such as a word processing application, note-taking application, image management application, digital content management application, drawing application, presentation application, word processing application, spreadsheet application, messaging application, web browsing application, and / or email application. In some embodiments, the user interface simultaneously includes one or more applications and / or multiple user interfaces of the electronic device's operating system. In some embodiments, the user interface has one or more characteristics of the user interface of method 700.

[0312] In some implementations, when the user interface is displayed via the display generating component, the electronic device detects (902b) a first gesture (e.g., positioning and / or orientation) of the input device (e.g., a stylus) relative to a surface (e.g., a touch-sensitive surface, a physical surface of the projected user interface, or a virtual surface corresponding to at least a portion of the user interface), such as Figure 8A The posture of the input device 800 in part 860a. For example, the electronic device and / or touch-sensitive surface acquires information including positioning / attitude (pitch, yaw and / or roll), orientation, tilt, path, force, distance and / or position of the input device relative to the surface through one or more sensors of the input device, one or more electrodes in the surface, one or more planar surfaces of a physical object (or physical area) in the physical environment, other defined coordinate systems, other sensors and / or other input devices (e.g., an input touchpad, which includes a specialized surface configured to translate motion and posture information of the input device).

[0313] In some implementations, in response to detecting a first posture of the input device relative to a surface, and based on determining that the first posture of the input device relative to the surface includes the input device being within a threshold distance of the surface (e.g., 0 cm, 0.01 cm, 0.05 cm, 0.1 cm, 0.2 cm, 0.5 cm, 0.8 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm), the electronic device displays (902c) a user interface via a display generation component as a representation including a virtual shadow corresponding to the input device, such as... Figure 8A The virtual shadow 832 in part 860b, wherein the representation of the virtual shadow in the user interface has a first visual appearance and a first positioning based on a first pose of the input device relative to a surface, such as Figure 8AThe appearance and / or positioning of the virtual shadow 832 in part 860b. For example, the electronic device optionally determines that the input device is within a threshold distance of a surface (in some embodiments, not in contact with the surface; in some embodiments, in contact with the surface), and based on this determination, displays a virtual shadow having a first visual appearance corresponding to a first intensity (e.g., a first degree of tinting, a first shape, a first size, a first degree of transparency, a first angle, a first distance, a first degree of blur, and / or a first other characteristic of the virtual shadow), which is virtually projected onto the user interface by the input device. For example, the first positioning of the representation of the virtual shadow projected onto the user interface is based on gestural information of the physical position and / or orientation of the input device relative to the surface. In some embodiments, the positioning and / or visual appearance of the virtual shadow corresponding to the input device is as if one or more external light sources at one or more locations relative to the electronic device were illuminating the surface and / or the surface of the input device and / or the display generating component, thereby displaying the virtual shadow.

[0314] In some embodiments, when the user interface is displayed via a display generation component as a representation including a virtual shadow corresponding to the input device, wherein the representation of the virtual shadow has a first visual appearance and a first positioning in the user interface, the electronic device detects (902d) movement of the input device relative to the surface from a first posture to a second posture (e.g., positioning and / or orientation) different from the first posture, such as from Figure 8A The orientation of input device 800 in part 860a Figure 8A The movement of the pose of the input device 800 in part 862a. For example, the electronic device detects input from a user (e.g., finger manipulation on the input device, gesture on the input device, and / or rotational or translational movement of the input device) to move the positioning and / or orientation of the input device relative to the surface from a first pose to a second pose. For example, the second pose of the input device is optionally within a threshold distance of the surface and optionally vertically oriented relative to a reference axis, compared to the first pose where the input device is optionally horizontally oriented relative to the reference axis. In some embodiments, the transition from the first pose to the second pose includes changing the distance of the input device from the surface without changing the orientation of the input device relative to the surface; in some embodiments, the transition from the first pose to the second pose includes changing the orientation of the input device relative to the surface without changing the distance from the surface; in some embodiments, the transition from the first pose to the second pose includes changing both the orientation of the input device relative to the surface and the distance from the surface.

[0315] In some embodiments, in response to detecting a movement of the input device relative to a surface from a first posture to a second posture and based on determining that the second posture relative to the surface includes the input device being within a threshold distance of the surface, the electronic device displays (902e) a user interface via a display generation component as a representation including a virtual shadow corresponding to the input device, the representation having a second visual appearance in the user interface that differs from the first visual appearance based on the second posture of the input device relative to the surface and a second positioning that differs from the first positioning, such as... Figure 8A The appearance and / or positioning of the virtual shadow 832 in part 862b. For example, the virtual shadow visually alters the appearance of the simulated shadow with a second tinting intensity, a second shape, a second size, a second transparency, a second angle, a second distance, a second blur level, and / or otherwise. In some embodiments, once the input device is within a threshold distance of the surface, the electronic device displays (or presents) a representation of the virtual shadow, which varies based on the positioning and / or orientation information of the physical position and / or orientation of the input device relative to the surface; for example, the virtual shadow is more intense (sharper and / or darker) at a second positioning (closer to the surface) compared to a first positioning (farther from the surface but within the threshold distance). In some embodiments, the positioning and / or visual appearance of the virtual shadow corresponding to the input device in a second posture is as if the aforementioned one or more external light sources at one or more of the aforementioned positioning relative to the electronic device were illuminating the surface and / or surface of the input device and / or display generating component, thereby displaying the virtual shadow. Displaying a virtual shadow of the input device that changes based on its posture provides an indication of the input device's posture, distance from the surface, and / or distance from the target user interface element. This allows the user to place the input device precisely, thereby reducing errors in the interaction between the input device and / or the surface (e.g., preventing accidental handwriting from the input device in the user interface) and reducing the amount of input required to correct such errors.

[0316] In some implementations, the representation of the virtual shading corresponding to the input device includes a first portion (such as) corresponding to the cylinder of the currently selected drawing tool on the input device. Figure 8M Part 832a) and the second part (904) corresponding to the tip of the currently selected drawing tool (such as... Figure 8M(as per part 832b). For example, the input device simulates one or more virtual drawing tools (e.g., a pen, pencil, brush, and / or highlighter), and when it is determined that the input device is within a threshold distance from the surface, the user interface optionally includes an indication of a first portion corresponding to the barrel of the currently selected drawing tool and an indication of a second portion corresponding to the tip of the currently selected drawing tool. In some embodiments, the indication of the first portion corresponding to the barrel of the currently selected drawing tool indicates one or more of the following: the distance of the input device relative to the surface, the orientation of the input device relative to the surface, and / or the tilt of the input device relative to the surface. In some embodiments, the indication of the second portion corresponding to the tip of the currently selected drawing tool indicates one or more of the following: the proximity of the input device to the surface, and / or the location where the user interface will draw (or render) the handwriting if and / or when the input device provides handwriting input to the user interface (e.g., movement of the input device when the tip of the input device contacts the surface). In some embodiments, the representation of the virtual shadow, including the first and second portions, is not displayed (e.g., because the user interface of an application installed on the electronic device already presents a visual indication of the input device, because the user interface of the application installed on the electronic device does not support (is not configured to) present a representation of the virtual shadow, and / or because the input device is outside a threshold distance from the surface). In some embodiments, the user interface includes only a portion of the first portion corresponding to the barrel of the currently selected drawing tool of the input device, but not the entire first portion corresponding to the barrel of the currently selected drawing tool of the input device (e.g., because the remainder of the first portion would extend beyond the display boundary of the display generating component). In some embodiments, the user interface includes the first portion corresponding to the barrel of the currently selected drawing tool of the input device, but excludes the second portion, as described in more detail later. In some embodiments, the user interface includes the second portion corresponding to the tip of the currently selected drawing tool of the input device, but excludes the first portion, as described in more detail later with reference to step 950. In some embodiments, the user interface first includes the first portion corresponding to the barrel of the currently selected drawing tool of the input device, and then includes the second portion corresponding to the tip of the currently selected drawing tool of the input device, as described in more detail later with reference to step 948. By presenting a virtual shadow with two distinct parts, electronic devices enable users to precisely position the input device relative to the surface and understand the type and / or characteristics of the virtual drawing tool before providing handwriting input on the user interface, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in the user interface) and reducing the amount of input required to correct such errors.

[0317] In some implementations, when displaying a representation of a virtual shadow corresponding to the input device, the electronic device detects (906a) a movement of the input device relative to the surface from a second posture to a third posture different from the second posture, such as the input device 800 moving from... Figure 8Q to Figure 8R The movement. For example, the third pose is outside the aforementioned threshold distance on the surface.

[0318] In some implementations, in response to detecting a movement of the input device relative to the surface from a second posture to a third posture, and based on determining that the third posture relative to the surface includes the input device being outside a threshold distance from the surface (e.g., 0 cm, 0.01 cm, 0.05 cm, 0.1 cm, 0.2 cm, 0.5 cm, 0.8 cm, 1 cm, 3 cm, 5 cm, 10 cm, 30 cm, 50 cm, or 100 cm), the electronic device stops (906b) displaying a representation of a virtual shadow corresponding to the input device, such as not displaying... Figure 8R The virtual shadow 832. For example, when an input device is detected to be outside a threshold distance from the surface, the representation of the virtual shadow disappears from the user interface (e.g., is not included). In some embodiments, the threshold distance at which the electronic device stops displaying the representation of the virtual shadow corresponding to the input device (“hidden distance”) is different from the threshold distance at which the electronic device begins displaying the representation of the virtual shadow corresponding to the input device (“display distance”) (e.g., threshold hysteresis to avoid jitter when displaying the representation of the virtual shadow when the input device is at or very close to the threshold distance). In some embodiments, the “hidden distance” is greater than the “display distance”; in some embodiments, the “hidden distance” is less than the “display distance”. In some embodiments, the representation of the virtual shadow disappears only after the entire input device is detected to be outside the threshold distance from the surface. In some embodiments, the representation of the virtual shadow disappears only after a majority (e.g., greater than 70%, 75%, 80%, 85%, 90%, or 95%) of the input device is detected to be outside the threshold distance from the surface. In some embodiments, the representation of the virtual shadow disappears only after the tip of the input device (or other specific portion of the input device) is detected to be outside the threshold distance from the surface. Stopping the display of virtual shadows means that input from input devices will not be detected by electronic devices, thereby reducing errors in the interaction between input devices and / or surfaces (e.g., preventing input devices from creating accidental handwriting in the user interface) and reducing the amount of input required to correct such errors.

[0319] In some implementations, the first posture relative to the surface includes a first distance of the input device from the surface, such as... Figure 8L The distance to the input device 800 in the first visual appearance includes the intensity of the representation of the virtual shadow as a first intensity (908), such as Figure 8LThe intensity of the virtual shadow 832 in the image. For example, a virtual shadow with a first intensity is represented by a first degree of shading, a first shape, a first size, a first degree of transparency, a first angle, a first offset (e.g., the gap between the virtual shadow and the input device), a first degree of blur, and / or other first characteristics of the virtual shadow. In some embodiments, the second pose relative to the surface includes a second distance from the input device to the surface that is different from the first distance, such as... Figure 8M The distance to the input device 800, and the second visual appearance including the representation of virtual shadows with an intensity different from the first intensity (908), such as Figure 8M The intensity of the virtual shadow 832 in the image. The electronic device detects a change in distance between the input device and the surface (or relative to the surface) and, in response to this change in distance, optionally updates the representation of the virtual shadow. For example, if the second distance is less than the first distance (e.g., closer to the surface), the representation of the virtual shadow with the second intensity is displayed with a second tint that is denser than the first tint (e.g., darker), a second shape that is shorter than the first shape, a second size that is smaller than the first size, a second transparency that is lower than the first transparency, a second angle that deviates from the input device by a larger angle than the first angle (e.g., closer to the input device after rotation), a second deviation that is smaller than the first deviation (e.g., a smaller gap between the virtual shadow and the input device), and / or a second blur that is smaller than the first blur (e.g., clearer). In some implementations, if the second distance is greater than the first distance (e.g., further away from the surface), the representation of the virtual shadow includes a third intensity displayed with a third tint that is lighter than the first tint (e.g., paler), a second shape that is longer than the first shape, a third size that is larger than the first size, a third transparency that is higher than the first transparency, a third angle that is smaller than the first angle relative to the input device (e.g., further away from the input device after rotation), a third deviation that is smaller than the first deviation (e.g., a larger gap between the virtual shadow and the input device), and / or a third blur that is greater than the first blur (e.g., more blurred). Displaying a virtual shadow of the input device with intensity that changes based on changes in the posture of the input device provides an indication of the relative positioning of the input device relative to the surface, the distance to the surface, and / or the distance to the target user interface element, and enables the user to accurately position the input device, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in the user interface) and reducing the amount of input required to correct such errors.

[0320] In some implementations, the first posture relative to the surface includes the input device having a first orientation relative to the surface, such as... Figure 8CThe orientation of the input device 800 in part 876a, and the first visual appearance including the intensity of the representation of the virtual shadow is a first intensity (e.g., as described in more detail with reference to step 908), wherein the intensity of the representation of the virtual shadow is based on the orientation of the input device relative to the surface (910), such as Figure 8C The intensity of the virtual shadow 832 in part 876b. The input device has a first orientation relative to the surface, optionally including a first tilt (or angle) relative to the normal (perpendicular) of the surface, as described in more detail later with reference to step 914. In some embodiments, the electronic device detects a change in tilt relative to the normal of the surface, and in response to the change in tilt relative to the normal of the surface, the electronic device updates the representation of the virtual shadow, as described in more detail later with reference to step 914. Displaying a virtual shadow of the input device with varying intensity based on changes in the orientation of the input device provides an indication of the tilt of the input device relative to the surface and allows the user to accurately position the input device, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in the user interface) and reducing the amount of input required to correct such errors.

[0321] In some embodiments, when displaying a representation of a virtual shadow corresponding to the input device while the input device has a first orientation relative to the surface, the electronic device detects (912a) a movement of the input device relative to the surface from a first posture to a third posture different from the first posture, such as the input device moving from... Figure 8C Part 876a to Figure 8C The movement of part 878a in the middle. For example, the third pose is within a threshold distance of the surface (e.g., 0cm, 0.01cm, 0.05cm, 0.1cm, 0.2cm, 0.5cm, 0.8cm, 1cm, 3cm, 5cm, 10cm, 30cm, 50cm or 100cm).

[0322] In some embodiments, in response to detecting a movement of the input device relative to the surface from a first posture to a third posture, and based on determining that the third posture includes a second orientation of the input device relative to the surface within a first orientation range (e.g., within 1 degree, 2 degrees, 3 degrees, 5 degrees, 10 degrees, 15 degrees, 20 degrees, 30 degrees, 35 degrees, 40 degrees, or 45 degrees, and / or between 0 degrees and 45 degrees, between 3 degrees and 30 degrees, or between 5 degrees and 20 degrees), (e.g., the third posture includes a third tilt (e.g., 15 degrees from the normal), which is less than the first orientation), the third tilt is less than the second orientation. A first tilt angle of a pose (e.g., 45 degrees from the normal), and a representation of the virtual shadow having a third visual appearance different from the first visual appearance (e.g., an intensity level lower than the first visual appearance), an electronic device display (912b) displays a representation of the virtual shadow corresponding to the input device in a third visual appearance different from the first visual appearance, wherein the third visual appearance includes a representation of the virtual shadow having a third intensity that varies based on the orientation of the input device relative to the surface within a first orientation range (e.g., as described in more detail with reference to step 908), such as Figures 8A to 8C In this embodiment, the intensity of the virtual shadow 832 varies based on the orientation of the input device 800 within a threshold angle 854 of the normal 850. For example, the intensity of the virtual shadow representation optionally varies gradually when the orientation of the input device relative to the surface is within a first orientation range. For example, the intensity of the virtual shadow decreases as the tilt of the input device decreases. In some embodiments, the intensity of the virtual shadow representation gradually decreases as the tilt of the input device decreases from 20 degrees to 5 degrees from the normal. For example, when the orientation of the input device includes a 20-degree tilt from the normal, the electronic device optionally displays a virtual shadow representation with reduced intensity (e.g., shorter, smaller, and / or blurred shape). As the tilt continues to decrease from 20 degrees to, for example, 10 degrees from the normal, the electronic device optionally displays a virtual shadow representation with weaker intensity than when the input device includes a 20-degree tilt from the normal (e.g., shorter, smaller, and / or more blurred shape). In some embodiments, the intensity of the virtual shadow representation decreases until a second orientation range is reached, as described in more detail with reference to step 912. Displaying a virtual shadow of the input device, whose intensity gradually changes based on the orientation of the input device, provides an indication of the tilt of the input device relative to the surface and allows the user to place the input device precisely, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in the user interface) and reducing the amount of input required to correct such errors.

[0323] In some implementations, in response to detecting a movement of the input device relative to the surface from a first posture to a third posture, and based on determining that the third posture includes the input device having a third orientation relative to the surface within a second orientation range different from the first orientation range (e.g., within 0, 1, 2, 3, 5, 10, 15, 20, 30, 35, 40, or 45 degrees from the normal, and / or between 30 and 0 degrees from the normal, between 20 and 0 degrees from the normal, or between 5 and 0 degrees from the normal), such as in Figures 8A to 8C Within a threshold angle 854 of the normal 850, the electronic device stops (914) displaying a representation of a virtual shadow corresponding to the input device, such as... Figure 8A Part of 858b and Figure 8C As shown in section 874b. For example, the intensity of the virtual shadow representation decreases until a second orientation range is reached, within which the representation of the virtual shadow optionally gradually weakens. In some embodiments, the second orientation range is closer to the vertical direction than the first orientation range. For example, when the third orientation of the input device includes zero tilt, which is within the second orientation range, the representation of the virtual shadow disappears from the user interface (e.g., is not included). In some embodiments, when the orientation of the input device is within the second orientation range, changes in the orientation of the input device remain within the second orientation range without causing changes in the display in the user interface (e.g., not displaying aspects of the representation of the virtual shadow). Stopping the display of the virtual shadow when the input device is substantially perpendicular to the surface provides an indication that the input device is substantially perpendicular to the surface, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding accidental handwriting in the user interface by the input device) and reducing the input required to correct such errors.

[0324] In some embodiments, the first pose relative to the surface includes the input device being in a first orientation relative to the surface, and the first visual appearance includes the shape of the virtual shadow being a first shape (916), such as Figure 8AThe shape of the virtual shadow 832 in portion 860b. For example, a first orientation is perpendicular or substantially perpendicular to the surface (e.g., perpendicularity within 1 degree, 3 degrees, 5 degrees, 10 degrees, 15 degrees, or 20 degrees). When the input device is perpendicular or substantially perpendicular to the surface, the representation of the virtual shadow optionally includes the first shape. In some embodiments, when the input device is parallel or more or substantially parallel to the surface, the electronic device optionally displays a representation of the virtual shadow having the first shape, which is less visible than (e.g., shorter and / or smaller) a second shape of the representation of the virtual shadow, as described in more detail later with reference to step 914. In some embodiments, the representation of the virtual shadow includes a second portion corresponding to the tip of the currently selected drawing tool of the input device, but does not include a first portion corresponding to the barrel of the currently selected drawing tool of the input device. In some embodiments, when the first orientation of the input device is perpendicular or substantially perpendicular to the surface, the representation of the virtual shadow includes a first corresponding portion of the first portion corresponding to the barrel of the currently selected drawing tool of the input device, but does not include a second corresponding portion of the first portion of the virtual shadow.

[0325] In some embodiments, the second orientation relative to the surface includes the input device being in a second orientation relative to the surface that is different from the first orientation, and the second visual appearance includes the shape of the virtual shadow being a second shape that is different from the first shape (916), such as Figure 8A The shape of the virtual shadow 832 in portion 862b. For example, the first orientation is parallel or substantially parallel to the surface (e.g., within 1 degree, 3 degrees, 5 degrees, 10 degrees, 15 degrees, or 20 degrees of parallelism). When the input device is parallel or substantially parallel to the surface, the representation of the virtual shadow optionally includes a second shape. In some embodiments, the electronic device displays a representation of the virtual shadow with the second shape, which is more visible than a representation of the virtual shadow with the first shape. For example, the second shape is optionally longer than the first shape, and / or the second shape is optionally larger than the first shape. In some embodiments, when the first orientation of the input device is parallel or substantially parallel to the surface, the representation of the virtual shadow includes most (or all) of the first portion corresponding to the barrel of the currently selected drawing tool of the input device, compared to when the input device is perpendicular or substantially perpendicular to the surface, only a portion of the first portion corresponding to the barrel of the currently selected drawing tool of the input device is included. The virtual shadow of the input device, which changes based on the orientation of the input device, provides an indication of the orientation of the input device relative to the surface and allows the user to place the input device precisely, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in the user interface) and reducing the amount of input required to correct such errors.

[0326] In some implementations, the first pose relative to the surface includes the input device being in a first orientation relative to the surface, and the first visual appearance includes the orientation of the representation of the virtual shadow being in a first corresponding orientation relative to the user interface (918), such as Figure 8A The orientation of the virtual shadow 832 in portion 860b. For example, in a first orientation, the cylindrical end of the input device (e.g., the part furthest from the tip of the input device) points toward an edge of the user interface (e.g., the top edge, left edge, right edge, or left edge). In some embodiments, the electronic device displays a representation of the virtual shadow in a corresponding orientation relative to the user interface based on the detected orientation of the input device (e.g., the end of the first portion corresponding to the cylindrical part of the currently selected drawing tool of the input device points toward the top edge of the user interface, rather than the bottom edge, right edge, or left edge). In some embodiments, the tip of the input device points toward an edge of the user interface (e.g., the top edge, left edge, right edge, or left edge), and the electronic device displays a representation of the virtual shadow in a corresponding orientation relative to the user interface based on the detected orientation of the input device (e.g., the end of the second portion corresponding to the tip of the currently selected drawing tool of the input device points toward the top edge of the user interface, rather than the bottom edge, right edge, or left edge).

[0327] In some embodiments, the second orientation relative to the surface includes the input device being in a second orientation relative to the surface that differs from the first orientation (e.g., in the second orientation, the cylindrical end of the input device points toward the bottom edge of the user interface in a direction opposite to (or different from) the first orientation pointing toward the top edge of the user interface), and the second visual appearance includes the orientation of the representation of the virtual shadow being a second corresponding orientation relative to the user interface that differs from the first corresponding orientation (918), such as Figure 8CThe orientation of the virtual shadow 832 in part 876b. For example, the user interface includes a representation of the virtual shadow in a second corresponding orientation relative to the user interface (e.g., the end of a first portion corresponding to the barrel of the currently selected drawing tool of the input device points towards the bottom edge of the user interface, rather than towards the top, right, or left edge of the user interface). The representation of the virtual shadow in the second corresponding orientation is optionally in the opposite direction (or a different direction) from the representation of the virtual shadow in the first corresponding orientation, because the representation of the virtual shadow in the first corresponding orientation optionally includes the end of a first portion corresponding to the barrel of the currently selected drawing tool of the input device pointing towards the top edge of the user interface. In some embodiments, the tip of the input device points towards the bottom edge of the user interface, and the electronic device displays a representation of the virtual shadow in a corresponding orientation relative to the user interface (e.g., the end of a second portion corresponding to the tip of the currently selected drawing tool of the input device points towards the bottom edge of the user interface, rather than towards the top, right, or left edge of the user interface). The virtual shadow of the input device, which changes orientation based on the orientation of the input device, provides an indication of whether the input device is pointing to a specific edge or boundary of the surface and enables the user to place the input device precisely, thereby reducing errors in the interaction between the input device and / or the surface (e.g., avoiding accidental handwriting from the input device in the user interface) and reducing the amount of input required to correct such errors.

[0328] In some implementations, the user interface is the user interface of a drawing application (920), such as Figure 8LThe user interface 890 is described. For example, the representation of the virtual shadow is included in the user interface of a drawing application. The drawing application is optionally an application in which handwriting input received from an input device is displayed in the user interface as handwriting on a drawing canvas. In some embodiments, the representation of the virtual shadow is included in the user interface of various applications accessible to the electronic device, such as word processing applications, photo management applications, spreadsheet applications, presentation applications, website creation applications, email applications, or other content creation applications. In some embodiments, the electronic device displays the virtual shadow in the user interface of a drawing application (e.g., an application configured to receive drawing input from an input device), but does not display the virtual shadow in other types of applications (e.g., applications not configured to receive drawing input from an input device, such as calendar applications, TV / movie browsing applications, digital wallet applications, and / or map / navigation applications). In some embodiments, the electronic device does not display the virtual shadow in the system user interface (e.g., as described with reference to step 922). Displaying virtual shadows in drawing applications allows users to precisely position input devices when providing drawing input, thereby reducing errors in the interaction between input devices and / or surfaces (e.g., preventing input devices from creating accidental handwriting in the user interface) and reducing the amount of input required to correct such errors.

[0329] In some implementations, displaying a representation of a virtual shadow with a first visual appearance includes (922a) displaying a representation of the virtual shadow in a first shape corresponding to the first drawing tool, based on determining that the currently selected drawing tool for the input device is a first drawing tool (922b), such as... Figure 8N The shape of the virtual shadow 832. For example, if the first drawing tool is a virtual pen (e.g., the input device is used as a virtual pen), the first shape of the virtual shadow corresponds to the round bullet tip of the virtual pen. In another example, the second part corresponding to the tip of the currently selected drawing tool of the input device (a representation of the virtual shadow with the first shape) is optionally a round bullet tip (corresponding to the round bullet tip of the virtual pen).

[0330] In some embodiments, displaying a representation of a virtual shadow having a first visual appearance includes, based on determining that the currently selected drawing tool for the input device is a second drawing tool different from the first drawing tool, displaying a representation of the virtual shadow in a second shape corresponding to the second drawing tool, wherein the second shape is different from the first shape (922b), such as... Figure 8OThe shape of the virtual shadow 832. For example, if the second drawing tool is a virtual highlighter different from the virtual pen, the second shape of the virtual shadow corresponds to the flat tip of the virtual pen, which is different from the first shape of the virtual shadow corresponding to the round bullet tip. In another example, the second portion (of the representation of the virtual shadow with the second shape) corresponding to the tip of the currently selected drawing tool for the input device is optionally a flat tip (which corresponds to the flat tip of the virtual highlighter). In some embodiments, the shape (and / or color) of the representation of the virtual shadow corresponds to the shape (and / or color) of the tip of the virtual drawing tool simulated by the input device, as described in more detail later with reference to step 920. In some embodiments, the first shape and / or the second shape corresponds to the shape of a selectable representation of a corresponding tool displayed in a tool palette of the user interface (e.g., a tool palette such as that described with reference to step 934), wherein the tool palette is interactive to select a drawing tool for the input device. Before detecting handwriting input on the user interface, a virtual shadow change is presented based on the currently selected drawing tool to indicate to the user the type and / or characteris...

Claims

1. A method, the method comprising: At the electronic device that communicates with the display generation component and one or more sensors: The display generation component displays a user interface including a first user interface object; When the user interface including the first user interface object is displayed via the display generation component, a corresponding object that approaches but does not touch the surface associated with the user interface is detected via the one or more sensors; as well as In response to detecting the corresponding object approaching but not touching the surface: Based on the determination that the corresponding object near the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object, a first selectable option that can be selected to perform a first operation associated with the first user interface object is displayed in the user interface; as well as Based on the determination that the corresponding object near the surface is not an input device communicating with the electronic device, the display of the first selectable option in the user interface, which can be selected to perform the first operation associated with the first user interface object, is abandoned.

2. The method according to claim 1, further comprising: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, the visual characteristics of the first user interface object are modified to indicate that the first user interface object is selectable.

3. The method of claim 1, wherein the first user interface object is associated with selection of a first region of the user interface and not with selection of a second region of the user interface that is different from the first region of the user interface, and the first user interface object is interactive to modify the region of the user interface selected by the first user interface object, the method further comprising: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, a visual indication associated with one or more directions of the modification of the area of ​​the user interface selected by the first user interface object is displayed via the display generation component.

4. The method of claim 1, wherein the first user interface object corresponds to a first portion of the user interface, and the first selectable option is associated with stopping the display of the first portion of the user interface, the method further comprising: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the first selectable option, the first portion of the user interface is stopped from being displayed.

5. The method of claim 1, wherein the first user interface object includes a content input area, the content input area includes content, and the first selectable option is associated with stopping the display of the content in the content input area, the method further comprising: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the first selectable option, the display of the content in the content input area is stopped.

6. The method of claim 1, wherein the first user interface object is associated with presenting media content, and the first selectable option is associated with modifying the playback of the media content, the method further comprising: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the first selectable option, the playback of the media content is modified.

7. The method according to claim 1, further comprising: In response to detecting a corresponding object approaching but not touching the surface, and determining that the corresponding object approaching the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object: Based on the determination that the positioning of the input device meets one or more criteria, information associated with the first user interface object is displayed via the display generation component, the one or more criteria including those that are met when the positioning of the input device corresponds to the first user interface object for a period of time longer than a threshold.

8. The method of claim 1, wherein the first user interface object includes a content input area, and the method further includes: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, a visual indication of a content insertion cursor is displayed via the display generation component at a position corresponding to the input device in the content input area; When displaying the visual indication, one or more inputs corresponding to the selection of the visual indication are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the visual instruction, the display generation component displays the content insertion cursor at the position corresponding to the input device in the content input area.

9. The method of claim 1, wherein the first user interface object includes content, and the method further includes: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the content: Based on the determination that the content is uneditable, the display generation component displays a visual indication of the content selection cursor at a position in the content corresponding to the input device.

10. The method according to claim 9, further comprising: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the content: Based on the determination that the content is editable, the visual indication of displaying the content selection cursor at the position in the content corresponding to the input device via the display generation component is abandoned.

11. The method of claim 1, wherein the first user interface object is displayed with a first separation amount from the back panel before the corresponding object is detected approaching but not touching the surface, wherein the back panel is the background of the user interface, the method further comprising: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed with a second separation from the back panel that is greater than the first separation.

12. The method according to claim 1, further comprising: When the user interface including the first user interface object is displayed via the display generation component, wherein the first user interface object has a first visual appearance, and a first visual characteristic has a first value in the first visual appearance, and a first input corresponding to the cursor moving from a position away from the first user interface object to the first user interface object is detected via a cursor control input device communicating with the electronic device; as well as In response to detecting the first input, the cursor is moved to the first user interface object, and the first user interface object is displayed with a second visual appearance, in which the first visual characteristic has a second value different from the first value. When the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed with a third visual appearance, in which the first visual characteristic has the second value.

13. The method according to claim 12, further comprising: In response to detecting the first input, the first user interface object is displayed with a parallax effect based on the movement of the cursor when the cursor is located on the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the parallax effect based on the movement of the input device when the positioning of the input device corresponds to the first user interface object.

14. The method according to claim 12, further comprising: In response to detecting the first input, the first user interface object is displayed with a lighting effect based on the movement of the cursor when the cursor is located on the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the lighting effect based on the movement of the input device when the positioning of the input device corresponds to the first user interface object.

15. The method of claim 1, wherein the first user interface object corresponds to a link pointing to content, the method further comprising: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object, the visual appearance of the first user interface object is modified.

16. An electronic device, the electronic device comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the following steps: A user interface, including a first user interface object, is displayed via a display generation component; When the user interface including the first user interface object is displayed via the display generation component, a corresponding object that approaches but does not touch the surface associated with the user interface is detected via one or more sensors; as well as In response to detecting the corresponding object approaching but not touching the surface: Based on the determination that the corresponding object near the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object, a first selectable option that can be selected to perform a first operation associated with the first user interface object is displayed in the user interface; as well as Based on the determination that the corresponding object near the surface is not an input device communicating with the electronic device, the display of the first selectable option in the user interface, which can be selected to perform the first operation associated with the first user interface object, is abandoned.

17. The electronic device of claim 16, wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, the visual characteristics of the first user interface object are modified to indicate that the first user interface object is selectable.

18. The electronic device of claim 16, wherein the first user interface object is associated with selection of a first region of the user interface and not with selection of a second region of the user interface that is different from the first region of the user interface, and the first user interface object is interactive to modify the region of the user interface selected by the first user interface object, wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, a visual indication associated with one or more directions of the modification of the area of ​​the user interface selected by the first user interface object is displayed via the display generation component.

19. The electronic device of claim 16, wherein the first user interface object corresponds to a first portion of the user interface, and the first selectable option is associated with stopping the display of the first portion of the user interface, wherein the one or more programs further include instructions for performing the following steps: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; and In response to receiving one or more inputs corresponding to the selection of the first selectable option, the first portion of the user interface is stopped from being displayed.

20. The electronic device of claim 16, wherein the first user interface object includes a content input area, the content input area including content, and the first selectable option is associated with stopping the display of the content in the content input area, wherein the one or more programs further include instructions for performing the following steps: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; and In response to receiving one or more inputs corresponding to the selection of the first selectable option, the display of the content in the content input area is stopped.

21. The electronic device of claim 16, wherein the first user interface object is associated with presenting media content, and the first selectable option is associated with modifying the playback of the media content, wherein the one or more programs further include instructions for performing the following steps: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; and In response to receiving one or more inputs corresponding to the selection of the first selectable option, the playback of the media content is modified.

22. The electronic device of claim 16, wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and determining that the corresponding object approaching the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object: Based on the determination that the positioning of the input device meets one or more criteria, information associated with the first user interface object is displayed via the display generation component, the one or more criteria including those that are met when the positioning of the input device corresponds to the first user interface object for a period of time longer than a threshold.

23. The electronic device of claim 16, wherein the first user interface object includes a content input area, and wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, a visual indication of a content insertion cursor is displayed via the display generation component at a position corresponding to the input device in the content input area; When displaying the visual indication, one or more inputs corresponding to the selection of the visual indication are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the visual instruction, the display generation component displays the content insertion cursor at the position corresponding to the input device in the content input area.

24. The electronic device of claim 16, wherein the first user interface object includes content, and wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the content: Based on the determination that the content is uneditable, the display generation component displays a visual indication of the content selection cursor at a position in the content corresponding to the input device.

25. The electronic device of claim 24, wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the content: Based on the determination that the content is editable, the visual indication of displaying the content selection cursor at the position in the content corresponding to the input device via the display generation component is abandoned.

26. The electronic device of claim 16, wherein the first user interface object is displayed with a first separation amount from the back panel before the corresponding object is detected approaching but not touching the surface, wherein the back panel is the background of the user interface, and wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed with a second separation from the back panel that is greater than the first separation.

27. The electronic device of claim 16, wherein the one or more programs further include instructions for performing the following steps: When the user interface including the first user interface object is displayed via the display generation component, wherein the first user interface object has a first visual appearance, a first visual characteristic having a first value in the first visual appearance, and a first input corresponding to the cursor moving from a position away from the first user interface object to the first user interface object is detected via a cursor control input device communicating with the electronic device; and In response to detecting the first input, the cursor is moved to the first user interface object, and the first user interface object is displayed with a second visual appearance, in which the first visual characteristic has a second value different from the first value. When the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed with a third visual appearance, in which the first visual characteristic has the second value.

28. The electronic device of claim 27, wherein the one or more programs further include instructions for performing the following steps: In response to detecting the first input, the first user interface object is displayed with a parallax effect based on the movement of the cursor when the cursor is located on the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the parallax effect based on the movement of the input device when the positioning of the input device corresponds to the first user interface object.

29. The electronic device of claim 27, wherein the one or more programs further include instructions for performing the following steps: In response to detecting the first input, the first user interface object is displayed with a lighting effect based on the movement of the cursor when the cursor is located on the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the lighting effect based on the movement of the input device when the positioning of the input device corresponds to the first user interface object.

30. The electronic device of claim 16, wherein the first user interface object corresponds to a link pointing to content, and wherein the one or more programs further include instructions for performing the following steps: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object, the visual appearance of the first user interface object is modified.

31. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform a method comprising the following steps: A user interface, including a first user interface object, is displayed via a display generation component; When the user interface including the first user interface object is displayed via the display generation component, a corresponding object that approaches but does not touch the surface associated with the user interface is detected via one or more sensors; as well as In response to detecting the corresponding object approaching but not touching the surface: Based on the determination that the corresponding object near the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object, a first selectable option that can be selected to perform a first operation associated with the first user interface object is displayed in the user interface; as well as Based on the determination that the corresponding object near the surface is not an input device communicating with the electronic device, the display of the first selectable option in the user interface, which can be selected to perform the first operation associated with the first user interface object, is abandoned.

32. The non-transitory computer-readable storage medium of claim 31, wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, the visual characteristics of the first user interface object are modified to indicate that the first user interface object is selectable.

33. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object is associated with selection of a first region of the user interface and not with selection of a second region of the user interface that is different from the first region of the user interface, and the first user interface object is interactive to modify the region of the user interface selected by the first user interface object, wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, a visual indication associated with one or more directions of the modification of the area of ​​the user interface selected by the first user interface object is displayed via the display generation component.

34. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object corresponds to a first portion of the user interface, and the first selectable option is associated with stopping the display of the first portion of the user interface, wherein the method further comprises: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the first selectable option, the first portion of the user interface is stopped from being displayed.

35. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object includes a content input area, the content input area including content, and the first selectable option is associated with stopping the display of the content in the content input area, wherein the method further comprises: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the first selectable option, the display of the content in the content input area is stopped.

36. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object is associated with presenting media content, and the first selectable option is associated with modifying playback of the media content, wherein the method further comprises: When the first selectable option is displayed based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, one or more inputs corresponding to the selection of the first selectable option are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the first selectable option, the playback of the media content is modified.

37. The non-transitory computer-readable storage medium of claim 31, wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and determining that the corresponding object approaching the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object: Based on the determination that the positioning of the input device meets one or more criteria, information associated with the first user interface object is displayed via the display generation component, the one or more criteria including those that are met when the positioning of the input device corresponds to the first user interface object for a period of time longer than a threshold.

38. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object includes a content input area, and wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is the input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, a visual indication of a content insertion cursor is displayed via the display generation component at a position corresponding to the input device in the content input area; When displaying the visual indication, one or more inputs corresponding to the selection of the visual indication are received via the one or more sensors; as well as In response to receiving one or more inputs corresponding to the selection of the visual instruction, the display generation component displays the content insertion cursor at the position corresponding to the input device in the content input area.

39. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object includes content, and wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the content: Based on the determination that the content is uneditable, the display generation component displays a visual indication of the content selection cursor at a position in the content corresponding to the input device.

40. The non-transitory computer-readable storage medium of claim 39, wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the content: Based on the determination that the content is editable, the visual indication of displaying the content selection cursor at the position in the content corresponding to the input device via the display generation component is abandoned.

41. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object is displayed with a first separation amount from the back panel before the corresponding object is detected approaching but not touching the surface, wherein the back panel is the background of the user interface, wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed with a second separation from the back panel that is greater than the first separation.

42. The non-transitory computer-readable storage medium of claim 31, wherein the method further comprises: When the user interface including the first user interface object is displayed via the display generation component, wherein the first user interface object has a first visual appearance, and a first visual characteristic has a first value in the first visual appearance, and a first input corresponding to the cursor moving from a position away from the first user interface object to the first user interface object is detected via a cursor control input device communicating with the electronic device; as well as In response to detecting the first input, the cursor is moved to the first user interface object, and the first user interface object is displayed with a second visual appearance, in which the first visual characteristic has a second value different from the first value. When the positioning of the input device corresponds to the first user interface object, the first user interface object is displayed with a third visual appearance, in which the first visual characteristic has the second value.

43. The non-transitory computer-readable storage medium according to claim 42, wherein the method further comprises: In response to detecting the first input, the first user interface object is displayed with a parallax effect based on the movement of the cursor when the cursor is located on the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the parallax effect based on the movement of the input device when the positioning of the input device corresponds to the first user interface object.

44. The non-transitory computer-readable storage medium of claim 42, wherein the method further comprises: In response to detecting the first input, the first user interface object is displayed with a lighting effect based on the movement of the cursor when the cursor is located on the first user interface object, wherein displaying the first user interface object with the third visual appearance based on determining that the positioning of the input device corresponds to the first user interface object does not include displaying the first user interface object with the lighting effect based on the movement of the input device when the positioning of the input device corresponds to the first user interface object.

45. The non-transitory computer-readable storage medium of claim 31, wherein the first user interface object corresponds to a link pointing to content, wherein the method further comprises: In response to detecting a corresponding object approaching but not touching the surface, and based on the determination that the corresponding object approaching the surface is an input device communicating with the electronic device and the location of the input device corresponds to the first user interface object, the visual appearance of the first user interface object is modified.

Images