Apparatus and method for proximity-based interaction with user interface objects

Abstract

This invention relates to devices and methods for interacting with user interface objects based on proximity. The invention provides an electronic device having a touch-sensitive surface, a display, one or more first sensors for detecting the proximity of an input object above the touch-sensitive surface, and one or more second sensors for detecting the strength of contact between the input object and the touch-sensitive surface. The electronic device displays a user interface object at a first position and detects selection of the user interface object at the first position on the display. When the user interface object is selected, the device detects lateral movement of the input object above the touch-sensitive surface when the input object meets a hover criterion. In response, the device moves the selected user interface object according to the lateral movement. While moving the selected user interface object, the device also dynamically changes a first visual characteristic of the selected user interface object according to the current hover proximity parameter of the input object.

Description

[0001] This application is a divisional application of the invention patent application with the application date of September 22, 2017, application number 201780053133.7, and name "Device, method and user interface for interacting with user interface objects through proximity-based and contact-based input". Technical Field

[0002] The present invention generally relates to electronic devices with touch-sensitive surfaces, including but not limited to electronic devices with touch-sensitive surfaces that facilitate user interface interactions through proximity-based and contact-based input. Background Art

[0003] The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Exemplary touch-sensitive surfaces include touch pads and touch screen displays. Such surfaces are widely used to manipulate user interface objects on displays.

[0004] Some touch-sensitive surfaces also include sensors that detect input provided by an object (e.g., a stylus) that is not in direct contact with the touch-sensitive surface, but is in close proximity to the touch-sensitive surface. Proximity-based input provides an additional way to manipulate user interface objects on the display. However, contact-based input and proximity-based input often do not work seamlessly together and may interfere with each other, leading to user confusion and frustration.

[0005] Exemplary manipulations of user interface objects include adjusting the position and / or size of one or more user interface objects or activating a button or opening a file / application represented by a user interface object, as well as associating metadata with one or more user interface objects or otherwise manipulating the user interface. Exemplary user interface objects include digital images, videos, text, icons, and control elements (such as buttons) and other graphics. In some cases, a user needs to perform such operations on user interface objects in, for example, a note-taking application, a file management program, an image management application, a digital content management application, a drawing application, a presentation application, a word processing application, or a spreadsheet application.

[0006] However, the methods used to perform these manipulations are cumbersome and inefficient. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-driven devices. Summary of the invention

[0007] Therefore, there is a need for electronic devices with faster, more efficient methods and interfaces for interacting with user interfaces to perform operations using contact-based input and proximity-based input. Such methods and interfaces optionally supplement or replace traditional methods of interacting with user interfaces using contact-based and proximity-based input. Such methods and interfaces reduce the amount, degree and / or nature of input from the user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0008] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface can be reduced or eliminated by means of the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI mainly through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, website creation, disk editing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note recording, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] According to some embodiments, a method is performed at an electronic device having a touch-sensitive surface and a display, wherein the electronic device includes one or more first sensors to detect the proximity of an input object above the touch-sensitive surface, and optionally includes one or more second sensors to detect the contact intensity of the input object with the touch-sensitive surface. The method includes displaying a user interface object at a first position on the display, and detecting the selection of the user interface object at the first position on the display. When the user interface object is selected, the method also includes detecting lateral movement of the input object above the touch-sensitive surface when the input object meets the hovering criteria. In response to detecting lateral movement of the input object above the touch-sensitive surface when the input object meets the hovering criteria, the method includes moving the selected user interface object according to the lateral movement of the input object above the touch-sensitive surface. When moving the selected user interface object according to the lateral movement of the input object above the touch-sensitive surface, the method also includes dynamically changing a first visual characteristic of the selected user interface object according to a current hovering proximity parameter of the input object.

[0010] According to some embodiments, a method is performed at an electronic device having a touch-sensitive surface and a display, wherein the device includes one or more first sensors to detect the proximity of an input object above the touch-sensitive surface, and optionally includes one or more second sensors to detect the contact strength of the input object with the touch-sensitive surface. The method includes displaying a user interface object at a first position on the display. When the user interface object is displayed at the first position on the display, the method also includes detecting a first portion of input made by the input object, wherein detecting the first portion of the input includes detecting the input object at a position above the touch-sensitive surface corresponding to the first position of the user interface object on the display when the input object meets the hovering criteria. In response to detecting the first portion of the input, the method includes dynamically changing the appearance of the user interface object in a first manner according to a current hovering proximity parameter of the input object. After dynamically changing the appearance of the user interface object in a first manner according to the current hovering proximity parameter of the input object, the method also includes detecting a second portion of the input through the input object, wherein detecting the second portion of the input includes detecting that the input object is in contact with the touch-sensitive surface at an initial contact position corresponding to the first position on the display; and, in response to detecting the second portion of the input, dynamically changing the appearance of the user interface object in a second manner according to the current strength of the contact of the input object on the touch-sensitive surface.

[0011] According to some embodiments, a method is performed at an electronic device having a touch-sensitive surface and a display, wherein the electronic device includes one or more first sensors to detect the proximity of an input object above the touch-sensitive surface, and optionally includes one or more second sensors to detect the intensity of contact with the touch-sensitive surface. The method includes displaying a position indicator within text on the display when the input object above the touch-sensitive surface meets the hovering criteria. When the position indicator is displayed within the text on the display, the method includes detecting a first movement of the input object including a movement component parallel to the touch-sensitive surface, wherein the input object continues to meet the hovering criteria during the first movement. In response to detecting the first movement of the input object above the touch-sensitive surface, the method includes moving the position indicator within the displayed text according to the first movement of the input object. After moving the position indicator within the displayed text according to the first movement of the input object, the method includes detecting a second movement of the input object including a movement component parallel to the touch-sensitive surface; and, in response to detecting the second movement of the input object across the touch-sensitive surface: based on determining that the second movement of the input object is detected when the input object is in contact with the touch-sensitive surface, while the input object remains in contact with the touch-sensitive surface, selecting text according to the second movement of the input object across the touch-sensitive surface.

[0012] A method is performed on a device having a display and a touch-sensitive surface, wherein the device includes one or more first sensors for detecting proximity of an input object above the touch-sensitive surface. The method includes displaying a user interface on the display; detecting a first input through the input object corresponding to the start of a first user interface operation; detecting lateral movement of the input object after detecting the first input corresponding to the start of the first user interface operation and while the input object satisfies a hover proximity criterion; in response to detecting the lateral movement of the input object while the input object satisfies the hover proximity criterion, updating the user interface to indicate a preview of the first user interface operation; while the user interface is updated to indicate the preview of the first user interface operation, detecting that the input object no longer satisfies the hover proximity criterion; and, in response to detecting that the input object no longer satisfies the hover proximity criterion: canceling the first user interface operation based on a determination that the hover proximity criterion is no longer satisfied because the input object has exited a hover proximity range of the touch-sensitive surface; and confirming the first user interface operation based on a determination that the hover proximity criterion is no longer satisfied because the input object has made contact with the touch-sensitive surface.

[0013] A method is performed on a device having a display and a touch-sensitive surface, wherein the device includes one or more first sensors for detecting proximity of an input object above the touch-sensitive surface. The method includes displaying a first user interface object at a first position on the display; detecting a predefined interaction of the input object with the first user interface object; in response to detecting the predefined interaction of the input object with the first user interface object, displaying a second user interface object associated with the first user interface object; while displaying the second user interface object, detecting a change in a hover proximity parameter of the input object; and, in response to detecting the change in the hover proximity parameter of the input object: maintaining display of the second user interface object based on determining that the input object satisfies a continue display criterion, wherein the continue display criterion requires that the input object continue to satisfy the hover proximity criterion in order to satisfy the continue display criterion; and ceasing display of the second user interface object based on determining that the input object satisfies a stop display criterion, wherein the stop display criterion requires that the input object no longer satisfies the hover proximity criterion in order to satisfy the stop display criterion.

[0014] A method is performed on a device having a display and a touch-sensitive surface, wherein the device includes one or more first sensors for detecting proximity of an input object above the touch-sensitive surface. The method includes: displaying a first user interface on the display; while displaying the first user interface on the display, detecting that the input object satisfies a first hover proximity criterion, wherein the first hover proximity criterion requires that a current value of a hover proximity parameter of the input object is within a first value range in order to satisfy the first hover proximity criterion; in response to detecting that the input object satisfies the first hover proximity criterion, displaying first visual feedback associated with the first hover proximity criterion; while displaying the first visual feedback associated with the first hover proximity criterion, detecting a change in the current value of the hover proximity parameter of the input object, and detecting that after the change the input object satisfies a second hover proximity criterion, wherein the second hover proximity criterion requires that the current value of the hover proximity parameter of the input object is within a second value range within the first value range in order to satisfy the second hover proximity criterion; and in response to detecting that the input object satisfies the second hover proximity criterion, displaying second visual feedback that is different and discontinuous from the first visual feedback, the second visual feedback being associated with a second hover proximity range.

[0015] A method is performed on a device having a display and a touch-sensitive surface, wherein the device includes one or more first sensors for detecting proximity of an input object above the touch-sensitive surface. The method includes: displaying a plurality of user interface objects including at least a first user interface object in a first user interface; while displaying the plurality of user interface objects, detecting an input object that satisfies a first hover proximity criterion at a first hover position above the first user interface object; after detecting the input object that satisfies the first hover proximity criterion at the first hover position above the first user interface object, detecting movement of the input object away from the first hover position; in response to detecting movement of the input object away from the first hover position: performing a first operation associated with movement of the input object based on determining that the input object satisfies a first enhanced hover proximity criterion, wherein the first enhanced hover proximity criterion requires that the input object already satisfies the first hover proximity criterion and enhances additional requirements of the first hover proximity criterion before the input object leaves the first position; and abandoning performance of the first operation based on determining that the input object does not satisfy the first enhanced hover proximity criterion.

[0016] A method is performed on a device having a display and a touch-sensitive surface, wherein the device includes: one or more first sensors for detecting proximity of an input object above the touch-sensitive surface; and one or more second sensors for detecting intensity of contact with the touch-sensitive surface. The method includes: displaying a plurality of user interface objects including a first user interface object on the display; detecting a first portion of an input through the input object, including detecting the input object at a first hover position above the touch-sensitive surface when the input object meets a first hover proximity criterion, wherein the first hover position above the touch-sensitive surface corresponds to a position of the first user interface object on the display; in response to detecting the first portion of the input, including detecting the input object at the first hover position above the touch-sensitive surface when the input object meets the first hover proximity criterion: based on determining that the first user interface object has a different response to the intensity-based input, displaying first visual feedback associated with the first user interface object to indicate that the first user interface object has a different response to the intensity-based input; and based on determining that the first user interface object does not have a different response to the intensity-based input, forgoing displaying the first visual feedback associated with the first user interface object.

[0017] A method is performed on a device having a display and a touch-sensitive surface, wherein the device includes one or more first sensors for detecting proximity of an input object above the touch-sensitive surface. The method includes: displaying a content selection object within selectable content, wherein the content selection object includes a first edge of the content selection object and a second edge of the content selection object, and wherein content located between the first edge of the content selection object and the second edge of the content selection object is selected; detecting a first portion of input through the input object, including detecting the input object at a first hover position above the touch-sensitive surface corresponding to the first edge of the content selection object; and in response to detecting the first portion of input through the input object: based on determining that the first portion of the input satisfies a first criterion, wherein the first criterion requires that the input object satisfies a hover proximity criterion when the input object is detected at a first hover position above the touch-sensitive surface corresponding to a position of the first edge of the content selection object on the display in order to satisfy the first criterion, thereby changing an appearance of the first edge relative to the second edge of the content selection object in a first manner to indicate that the first edge of the content selection object is to be selected to move relative to the second edge of the content selection object when the input object satisfies the second criterion.

[0018] A method is performed on a device having a display and a touch-sensitive surface, wherein the device includes one or more first sensors for detecting proximity of an input object above the touch-sensitive surface. The method includes: displaying a plurality of user interface objects on the display, wherein the plurality of user interface objects includes a first user interface object and a second user interface object, and wherein the first user interface object and the second user interface object are displayed at different locations on the display; detecting the input object at a first hovering position above a first portion of the touch-sensitive surface when the input object meets a hovering proximity criterion; in response to detecting the input object at the first hovering position above the first portion of the touch-sensitive surface when the input object meets the hovering proximity criterion, placing input focus on the first user interface object; and detecting a corresponding movement of the input object from the first hovering position above the first portion of the touch-sensitive surface to a corresponding hovering position above a corresponding portion of the touch-sensitive surface different from the first portion of the touch-sensitive surface while the first user interface object has input focus and the input object continues to meet the hovering proximity criterion. and while the input object continues to satisfy the hover proximity criteria, in response to detecting a corresponding movement of the input object: based on determining that the corresponding movement is a first movement from a first hover position to a second hover position above a second portion of the touch-sensitive surface: based on determining that the first movement satisfies a first focus retention criteria, wherein the first focus retention criteria requires that the first movement include lateral movement less than a first threshold amount in order to satisfy the first focus retention criteria, and wherein during the first movement of the input object from the first hover position to the second hover position, the first threshold amount of lateral movement is determined based on the value of a characteristic hover proximity parameter of the input object to maintain the input focus on the first user interface object; and based on determining that the first movement satisfies a first focus shift criteria, wherein the first focus shift criteria requires that the first movement include at least a first threshold amount of lateral movement in order to satisfy the first focus shift criteria, thereby moving the input focus from the first user interface object to the second user interface object.

[0019] According to some embodiments, an electronic device includes a display, a touch-sensitive surface, one or more sensors for detecting the proximity of an input object above the touch-sensitive surface, and optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, one or more processors, a memory, and one or more programs; one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any method described herein. According to some embodiments, a computer-readable storage medium stores therein instructions that, when executed by an electronic device having a display, a touch-sensitive surface, one or more sensors for detecting the proximity of an input object above the touch-sensitive surface, and optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, cause the device to perform the operations of any method described herein or cause the operations of any method described herein to be performed. According to some embodiments, a graphical user interface on an electronic device having a display, a touch-sensitive surface, one or more sensors for detecting the proximity of an input object above the touch-sensitive surface, optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, a memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements displayed in any method described herein, and the one or more elements are updated in response to the input, as described in any method described herein. According to some embodiments, an electronic device includes a display, a touch-sensitive surface, one or more sensors for detecting the proximity of an input object above the touch-sensitive surface, and optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface; and a device for performing or causing the performance of the operations of any of the methods described herein. According to some embodiments, an information processing device used in an electronic device having a display and a touch-sensitive surface, one or more sensors for detecting the proximity of an input object above the touch-sensitive surface, and optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface includes a device for performing or causing the performance of the operations of any of the methods described herein.

[0020] Thus, electronic devices having a display, a touch-sensitive surface, one or more sensors for detecting the proximity of an input object above the touch-sensitive surface, and optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface are provided with faster, more efficient methods and interfaces for interacting with a user interface to perform operations through proximity-based and contact-based inputs, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for interacting with a user interface through proximity-based and contact-based inputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0022] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0023] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0024] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0025] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0026] Figure 4 is a block diagram of an exemplary electronic stylus according to some embodiments.

[0027] FIG. 5A to FIG. 5B The position state of the stylus relative to the touch-sensitive surface according to some embodiments is shown.

[0028] Fig. 6A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0029] Figure 6B An example user interface for a multifunction device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0030] FIG. 7A to FIG. 7U An exemplary user interface for interacting with user interface objects through proximity-based and contact-based input (e.g., pulling, picking up, dragging, and dropping user interface objects) is shown according to some embodiments.

[0031] Figure 7V to Figure 7AJ An exemplary user interface for interacting with user interface objects through proximity-based and contact-based input (e.g., pushing, pressing, releasing, and pulling user interface objects) is shown according to some embodiments.

[0032] Figure 7AK to Figure 7AU An exemplary user interface for interacting with user interface objects through proximity-based and contact-based input (e.g., moving a cursor and selecting text) is shown according to some embodiments.

[0033] Figures 8A to 8VAn exemplary user interface is shown for displaying a preview of a move operation via hover move input, and canceling or confirming the move operation based on determining whether the input object is in contact with a touch-sensitive surface before being lifted out of a hover proximity range, according to some embodiments.

[0034] 9A to 9O An exemplary user interface is shown for displaying a user interface object following a predefined user interface interaction with an input object, and conditionally maintaining display of the user interface object while the input object continues to meet hover proximity criteria, according to some embodiments.

[0035] FIG. 10A to FIG. 10Y An exemplary user interface is shown for displaying different visual feedback when an input object is detected to meet different levels of hover proximity criteria according to some embodiments.

[0036] FIG. 11A to FIG. 11J An exemplary user interface is shown for entering an information display mode after an input object meets enhanced hover proximity criteria, and while in the information display mode, displaying information of additional objects when the input object meets non-enhanced hover proximity criteria, according to some embodiments.

[0037] FIG. 12A to FIG. 12R An exemplary user interface is shown for displaying visual feedback indicating whether an object has different object-specific responses to intensity-based input when an input object hovers over the object according to some embodiments.

[0038] FIG. 13A to FIG. 13N An exemplary user interface is shown for indicating which selection handle of a selection object is selected for movement when an input object hovers over a selection object according to some embodiments.

[0039] FIG. 14A to FIG. 14M Maintaining input focus on an object during hover motion input is shown according to some embodiments, wherein during hover motion input, the criteria for maintaining input focus on the object are dynamically changed based on characteristic hover proximity parameters of the input object.

[0040] FIG. 15A to FIG. 15C is a flowchart illustrating a method of interacting with a user interface object through proximity-based and contact-based input according to some embodiments.

[0041] FIG. 16A to FIG. 16E is a flowchart illustrating a method of interacting with a user interface object through proximity-based and contact-based input according to some embodiments.

[0042] 17A to 17C is a flowchart illustrating a method of interacting with a user interface object through proximity-based and contact-based input according to some embodiments.

[0043] FIG. 18A to FIG. 18G is a flow chart illustrating a method of moving an object via hover movement input according to some embodiments.

[0044] FIG. 19A to FIG. 19E is a flowchart illustrating a method of maintaining display of a transient user interface object using hover input according to some embodiments.

[0045] FIG. 20A to FIG. 20I is a flow chart illustrating a method of displaying different visual feedback when a hover input meets different levels of hover proximity criteria according to some embodiments.

[0046] FIG. 21A to FIG. 21D is a flow chart illustrating a method of performing operations after enhanced hover proximity criteria are met, according to some embodiments.

[0047] FIG. 22A to FIG. 22G is a flowchart illustrating a method for displaying visual feedback in response to hover input to identify user interface objects that have different responses to intensity-based input according to some embodiments.

[0048] FIG. 23A to FIG. 23E is a flowchart illustrating a method of visually indicating an edge of a selection object to be selected for movement in response to a hover input according to some embodiments.

[0049] FIG. 24A to FIG. 24E is a flowchart illustrating a method for maintaining input focus on an object based on whether the input object has moved laterally beyond a threshold amount according to some embodiments, wherein the threshold amount of lateral movement is determined based on characteristic hover proximity parameters of the input object. DETAILED DESCRIPTION

[0050] Proximity-based input and contact-based input are provided to manipulate user interface objects. Visual, tactile, and / or audio feedback is provided based on whether the user interface object is affected by proximity-based input (e.g., the user interface object is in a hovering state under the influence of a hovering input) or by a contact-based input (e.g., the user interface object is in a contacting state under the influence of a press input or a swipe input). Visual, tactile, and / or audio feedback is optionally provided when the user interface object transitions between a hovering state and a contacting state. The device dynamically changes the feedback provided to the user based on input metrics such as a hovering proximity parameter and contact strength.

[0051] In some embodiments, a user interface object is moved by a hover input, and the device changes the appearance of the user interface object based on a hover proximity parameter of the hover input. In some embodiments, when the continuous input includes one or more proximity-based inputs and one or more contact-based inputs, the device changes the appearance of the user interface object according to the hover proximity parameter of the proximity-based input and according to the intensity of the contact-based input, so that when the user interface object transitions between a hover state and a contact state, the change in at least one feature is a smooth change. In some embodiments, an input object (e.g., a finger or stylus) places a cursor at a position in text based on lateral movement of the input object over a touch-sensitive surface, and begins to expand a text selection from the cursor position based on lateral movement of the input object across the touch-sensitive surface.

[0052] under, Figure 1A to Figure 1B , Figure 2 and Figure 3 A description of an exemplary apparatus is provided. Figure 4 A description of an exemplary electronic stylus is provided. FIG. 5A to FIG. 5B The position state of the stylus relative to the touch-sensitive surface is shown. FIG. 6A to FIG. 6B , Figures 7A to 7AU , Figures 8A to 8V , 9A to 9O , FIG. 10A to FIG. 10Y , FIG. 11A to FIG. 11J , FIG. 12A to FIG. 12R , FIG. 13A to FIG. 13N as well as FIG. 14A to FIG. 14L An exemplary user interface for interacting with user interface objects via proximity-based input and contact-based input is shown. FIG. 15A to FIG. 15C , FIG. 16A to FIG. 16E , 17A to 17C , FIG. 18A to FIG. 18G , FIG. 19A to FIG. 19E , FIG. 20A to FIG. 20I , FIG. 21A to FIG. 21D , FIG. 22A to FIG. 22G , FIG. 23A to FIG. 23E and 24A to Fig.24E A flow chart of a method of interacting with a user interface object through proximity-based and contact-based input is shown. FIG. 6A to FIG. 6B , Figures 7A to 7AU , Figures 8A to 8V , 9A to 9O , FIG. 10A to FIG. 10Y , FIG. 11A to FIG. 11J , FIG. 12A to FIG. 12R , FIG. 13A to FIG. 13N as well as FIG. 14A to FIG. 14L The user interface in FIG. 15A to FIG. 15C , FIG. 16A to FIG. 16E , 17A to 17C , FIG. 18A to FIG. 18G , FIG. 19A to FIG. 19E , FIG. 20A to FIG. 20I , FIG. 21A to FIG. 21D , FIG. 22A to FIG. 22G , FIG. 23A to FIG. 23E and 24A to Fig.24E process.

[0053] Exemplary Devices

[0054] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0055] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0056] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0057] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0058] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0059] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0060] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0061] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0062] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0063] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0064] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0065] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory 102 by other components of the device 100 (such as one or more CPUs 120 and peripheral device interface 118) is optionally controlled by a memory controller 122.

[0066] Peripheral interface 118 may be used to couple the device's input and output peripherals to memory 102 and one or more CPUs 120. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0067] In some embodiments, peripherals interface 118, one or more CPUs 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0068] RF (radio frequency) circuit 108 receives and transmits RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memory, and the like. RF circuit 108 optionally communicates with networks and other devices through wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence using Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0069] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0070] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0071] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some visual outputs or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" is a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input pointing to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0072] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic / tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary embodiment, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0073] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0074] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0075] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0076] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0077] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, an optical sensor is located on the rear portion of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0078] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0079] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0080] Device 100 optionally also includes one or more tactile output generators 163. Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. One or more tactile output generators 163 optionally include one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on the device). The tactile output generator 163 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile outputs on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front of the device 100.

[0081] The device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripheral device interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of device 100.

[0082] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a position module (or instruction set) 131, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in Figures 1A and 1B. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0083] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0084] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a device compatible with some of Apple Inc. (Cupertino, California). iPod A multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in iPod devices. In some embodiments, the external port is compatible with some of Apple Inc. (Cupertino, California) iPod A Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in iPod devices.

[0085] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0086] The contact / motion module 130 optionally detects gesture input from a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0087] In some embodiments, detecting a finger tap gesture depends on detecting the length of time between a finger press event and a finger lift event, but is independent of the strength of the finger contact between the finger press event and the finger lift event. In some embodiments, a tap gesture is detected based on determining that the length of time between the finger press event and the finger lift event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the strength of the finger contact during the tap reaches a given strength threshold (greater than a nominal contact detection strength threshold), such as a light press or deep press strength threshold. Therefore, a finger tap gesture can meet a specific input criterion that does not require the characteristic strength of the contact to meet a given strength threshold to meet the specific input criterion. For clarity, the finger contact in a tap gesture is generally required to meet a nominal contact detection strength threshold to detect a finger press event, and the contact will not be detected below the nominal contact detection strength threshold. A similar analysis applies to detecting a tap gesture by a stylus or other contact. In the case where the device is capable of detecting contact with a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0088] The same concepts apply in a similar manner to other types of gestures. For example, a swipe gesture, a pinch gesture, an expand gesture, and / or a long press gesture may be optionally detected based on satisfying criteria that are independent of the strength of the contacts included in the gesture or that do not require one or more contacts performing the gesture to reach a strength threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a zoom gesture is detected based on the movement of two or more contacts toward each other; a zoom gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of a contact on a touch-sensitive surface having less than a threshold amount of movement. Therefore, the statement that a particular gesture recognition criterion does not require the strength of the contacts to satisfy a corresponding strength threshold in order to satisfy the particular gesture recognition criterion means that the particular gesture recognition criterion can be satisfied when the contacts in the gesture do not reach the corresponding strength threshold, and can also be satisfied when one or more contacts in the gesture reach or exceed the corresponding strength threshold. In some embodiments, a tap gesture is detected based on determining that a finger down event and a finger up event are detected within a predefined time period, regardless of whether the contact is above or below a corresponding strength threshold during the predefined time period, and a swipe gesture is detected based on determining that the contact moves greater than a predefined amount, even if the contact is above the corresponding strength threshold at the end of the contact movement. Even in specific implementations where detection of gestures is affected by the strength of the contact performing the gesture (e.g., the device detects a long press more quickly when the strength of the contact is above the strength threshold, or the device delays detection of a tap input when the strength of the contact is higher), detection of these gestures does not require the contact to reach a particular strength threshold (e.g., even if the amount of time required to recognize the gesture varies), as long as the criteria for recognizing the gesture can be met without the contact reaching the particular strength threshold.

[0089] In some cases, the contact intensity threshold, duration threshold, and movement threshold are combined in various combinations to create a heuristic algorithm to distinguish between two or more different gestures for the same input element or area, so that multiple different interactions with the same input element can provide a richer set of user interactions and responses. The statement that a particular set of gesture recognition criteria does not require the intensity of the contact to meet the corresponding intensity threshold to meet the particular gesture recognition criteria does not exclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures with criteria that are met when the gesture includes a contact with an intensity above the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the intensity of the contact to meet the corresponding intensity threshold to meet the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on the contact reaching the corresponding intensity threshold). In such a competition, if the second gesture recognition criterion for the second gesture is met first, the gesture is optionally not recognized as meeting the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before the contact moves a predefined amount of movement, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves the predefined amount of movement before the contact reaches the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in such cases, the first gesture recognition criteria for the first gesture still does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the first gesture recognition criteria because if the contact remains below the corresponding intensity threshold until the gesture ends (e.g., a swipe gesture with an intensity of the contact that does not increase above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criteria. Thus, a particular gesture recognition criterion that does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the particular gesture recognition criterion will (A) in some cases ignore the intensity of the contact relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases fail to satisfy the particular gesture recognition criterion (e.g., for a long press gesture) if a set of competing intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-related gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input, and in this sense still depends on the intensity of the contact relative to the intensity threshold (e.g., for a long press gesture competing with the deep press gesture for recognition).

[0090] In conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169, position module 131 optionally detects position information about the device, such as the attitude (roll, pitch, and / or yaw) of the device in a particular reference frame. Position module 130 includes software components for performing various operations related to detecting device position and detecting changes in device position. In some embodiments, position module 131 uses information received from a stylus used with the device to detect position information about the stylus, such as detecting a position state of the stylus relative to the device and detecting changes in the position state of the stylus.

[0091] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0092] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0093] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 163 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0094] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0095] The GPS module 135 determines the location of the device and provides such information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and maps / navigation widgets).

[0096] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0097] Contacts module 137 (sometimes called address book or contact list);

[0098] Telephone module 138;

[0099] Video conferencing module 139;

[0100] Email client module 140;

[0101] Instant messaging (IM) module 141;

[0102] Fitness support module 142;

[0103] A camera module 143 for still images and / or video images;

[0104] Image management module 144;

[0105] Browser module 147;

[0106] Calendar module 148;

[0107] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, and a user-created widget 149-6;

[0108] A widget creator module 150 for forming a user-created widget 149-6;

[0109] Search module 151;

[0110] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0111] Notepad module 153;

[0112] Map module 154; and / or

[0113] Online video module 155.

[0114] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0115] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0116] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a sequence of characters corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0117] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0118] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0119] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0120] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0121] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0122] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0123] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0124] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0125] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0126] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0127] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0128] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0129] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, etc. according to user instructions.

[0130] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0131] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0132] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0133] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0134] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0135] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0136] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192, which indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0137] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0138] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0139] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0140] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0141] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0142] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0143] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0144] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0145] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0146] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0147] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring in a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit (not shown) or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in the application views 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0148] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0149] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0150] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0151] In some embodiments, event definition 187 includes a definition of an event for a corresponding user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and an object that triggers the hit test.

[0152] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0153] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0154] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0155] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0156] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0157] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0158] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0159] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining an event to be recognized.

[0160] Figure 2 A device having a touch screen (e.g., Figure 1AThe portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, the selection of one or more graphics will occur when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0161] The device 100 optionally also includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0162] In some embodiments, the device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for a user of the device 100.

[0163] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuits (sometimes referred to as chipsets) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 163), sensors 359 (e.g., similar to those described above) Figure 1A The memory 370 may include touch sensors, optical sensors, contact intensity sensors, proximity sensors, acceleration sensors, gesture and / or magnetic sensors of the sensors 112, 164, 165, 166, 167, 168 and 169). The memory 370 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) stored in memory 102 of portable multifunction device 100, or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0164] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to a group of instructions for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subgroup of the above-mentioned modules and data structures. In addition, memory 370 optionally stores other modules and data structures not described above.

[0165] Figure 4 is a block diagram of an exemplary electronic stylus 203 according to some embodiments. Electronic stylus 203 is sometimes referred to simply as a stylus. Stylus 203 includes memory 402 (which optionally includes one or more computer-readable storage media), memory controller 422, one or more processing units (CPUs) 420, peripheral device interface 418, RF circuit 408, input / output (I / O) subsystem 406, and other input or control devices 416. Stylus 203 optionally includes external port 424 and one or more optical sensors 464. Stylus 203 optionally includes one or more intensity sensors 465 for detecting the intensity of contact of stylus 203 on device 100 (e.g., when stylus 203 is used with a touch-sensitive surface such as touch-sensitive display system 112 of device 100) or on other surfaces (e.g., a table surface). Stylus 203 optionally includes one or more tactile output generators 463 for generating tactile output on stylus 203. These components optionally communicate via one or more communication buses or signal lines 403 .

[0166] In some embodiments, the term "tactile output" discussed above refers to a physical displacement of an accessory (e.g., stylus 203) of a device (e.g., device 100) relative to a previous position of the accessory, a physical displacement of a component of the accessory relative to another component of the accessory, or a displacement of a component relative to the center of mass of the accessory that will be detected by a user by virtue of the user's sense of touch. For example, in the case where an accessory or a component of the accessory is in contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the accessory or component of the accessory. For example, movement of a component (e.g., the housing of stylus 203) is optionally interpreted by the user as a "click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "click", even when a physical actuation button associated with the stylus that is physically pressed (e.g., displaced) by the user's movement does not move. Although such user interpretations of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular perception of a user (e.g., a “click”), unless otherwise stated, the tactile output generated corresponds to a physical displacement of the device or a component thereof that would generate that perception for a typical (or average) user.

[0167] It should be understood that stylus 203 is merely one example of an electronic stylus, and that stylus 203 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 4 The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0168] Memory 402 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more flash memory devices or other non-volatile solid-state memory devices. Access to memory 402 by other components of stylus 203 (such as one or more CPUs 420 and peripheral device interface 418) is optionally controlled by memory controller 422.

[0169] Peripheral device interface 418 can be used to couple the input peripherals and output peripherals of the stylus to one or more CPUs 420 and memory 402. The one or more processors 420 run or execute various software programs and / or instruction sets stored in memory 402 to perform various functions of stylus 203 and process data.

[0170] In some embodiments, peripherals interface 418, one or more CPUs 420, and memory controller 422 are optionally implemented on a single chip, such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0171] RF (radio frequency) circuit 408 receives and transmits RF signals, also referred to as electromagnetic signals. RF circuit 408 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with device 100 or device 300, a communication network, and / or other communication devices via electromagnetic signals. RF circuit 408 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 408 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communications protocol including communications protocols that have not been developed on the date of filing of this document.

[0172] The I / O subsystem 406 couples input / output peripherals on the stylus 203, such as other input or control devices 416, to a peripheral device interface 418. The I / O subsystem 406 optionally includes an optical sensor controller 458, an intensity sensor controller 459, a tactile feedback controller 461, and one or more input controllers 460 for other input or control devices. The one or more input controllers 460 receive / send electrical signals from / to other input or control devices 416. Other input or control devices 416 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, click wheels, etc. In some alternative embodiments, the one or more input controllers 460 are optionally coupled to any of the following (or none of the following): an infrared port and / or a USB port.

[0173] Stylus 203 also includes a power system 462 for powering various components. Power system 462 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device and / or portable accessory.

[0174] Stylus 203 optionally also includes one or more optical sensors 464 . Figure 4 An optical sensor is shown coupled to an optical sensor controller 458 in the I / O subsystem 406. The optical sensor 464 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The one or more optical sensors 464 receive light projected through one or more lenses from the environment and convert the light into data representing an image.

[0175] Stylus 203 optionally also includes one or more contact intensity sensors 465 . Figure 4 A contact force sensor is shown coupled to force sensor controller 459 in I / O subsystem 406. One or more contact force sensors 465 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a surface). Contact force sensor 465 receives contact force information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to the tip of stylus 203.

[0176] Stylus 203 optionally also includes one or more proximity sensors 466 . Figure 4 A proximity sensor 466 is shown coupled to the peripherals interface 418. Alternatively, the proximity sensor 466 is coupled to the input controller 460 in the I / O subsystem 406. In some embodiments, the proximity sensor determines the proximity of the stylus 203 to the electronic device (eg, device 100).

[0177] Stylus 203 optionally also includes one or more tactile output generators 463 . Figure 4 A tactile output generator coupled to a tactile feedback controller 461 in the I / O subsystem 406 is shown. One or more tactile output generators 463 optionally include one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on the device). One or more tactile output generators 463 receive tactile feedback generation instructions from the tactile feedback module 433 and generate tactile outputs on the stylus 203 that can be felt by the user of the stylus 203. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to the length of the stylus 203 (e.g., the body or housing), and optionally generates tactile outputs by moving the stylus 203 vertically (e.g., in a direction parallel to the length of the stylus 203) or laterally (e.g., in a direction perpendicular to the length of the stylus 203).

[0178] Stylus 203 optionally also includes one or more accelerometers 467 , gyroscopes 468 , and / or magnetometers 470 for obtaining information about the position and positional state of stylus 203 (eg, as part of an inertial measurement unit (IMU)). Figure 4 Sensors 467, 469, and 470 are shown coupled to peripherals interface 418. Alternatively, sensors 467, 469, and 470 are optionally coupled to input controller 460 in I / O subsystem 406. Stylus 203 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information about the location of stylus 203.

[0179] In some embodiments, the software components stored in memory 402 include an operating system 426, a communication module (or instruction set) 428, a contact / motion module (or instruction set) 430, a location module (or instruction set) 431, and a global positioning system (GPS) module (or instruction set) 435. In addition, in some embodiments, memory 402 stores device / global internal state 457, such as Figure 4Device / global internal state 457 includes one or more of the following: sensor state, including information obtained from various sensors and other input or control devices 416 of the stylus; position state, including information about the position of the stylus relative to the device (e.g., device 100) (e.g., position, orientation, tilt, rotation, and / or distance, such as Figure 5A and Figure 5B and location information about the stylus location (eg, determined by GPS module 435).

[0180] The operating system 426 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.), and facilitates communication between various hardware components and software components.

[0181] The communication module 428 optionally facilitates communication with other devices through one or more external ports 424, and also includes various software components for processing data received by the RF circuit 408 and / or the external port 424. The external port 424 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a device compatible with some of Apple Inc. (Cupertino, California). iPod A Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in iPod devices.

[0182] The contact / motion module 430 optionally detects contact with the stylus 203 and other touch-sensitive devices of the stylus 203 (e.g., buttons or other touch-sensitive components of the stylus 203). The contact / motion module 430 includes software components for performing various operations related to the detection of contact (e.g., detection of contact between the tip of the stylus and a touch-sensitive display such as the touch screen 112 of the device 100 or with another surface such as a table surface), such as determining whether contact has occurred (e.g., detecting a touch down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking across that movement (e.g., across the touch screen 112 of the device 100), and determining whether the contact has ceased (e.g., detecting a lift-off event or contact disconnection). In some embodiments, the contact / motion module 430 receives contact data from the I / O subsystem 406. Determining movement of a point of contact optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the point of contact, the movement of the point of contact being represented by a series of contact data. As described above, in some embodiments, one or more of these operations related to detection of contact are performed by the device using contact / motion module 130 (in addition to or in lieu of the stylus using contact / motion module 430).

[0183] The contact / motion module 430 optionally detects gesture input from the stylus 203. Different gestures performed with the stylus 203 have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single tap gesture includes detecting a touch down event, followed by detecting a lift-off event at the same location (or substantially the same location) as the touch down event (e.g., at the location of an icon). As another example, detecting a swipe gesture includes detecting a touch down event, followed by detecting one or more stylus drag events, and then followed by detecting a lift-off event. As described above, in some embodiments, gesture detection is performed by the device using the contact / motion module 130 (in addition to or in lieu of the stylus using the contact / motion module 430).

[0184] In conjunction with accelerometer 467, gyroscope 468 and / or magnetometer 469, position module 431 optionally detects position information about the stylus, such as the attitude (roll, pitch and / or yaw) of the stylus in a particular reference frame. In conjunction with accelerometer 467, gyroscope 468 and / or magnetometer 469, position module 431 optionally detects stylus movement gestures, such as flicks, taps, and rotations of the stylus. Position module 431 includes software components for performing various operations related to detecting the position of the stylus and detecting changes in the position of the stylus in a particular reference frame. In some embodiments, position module 431 detects a position state of the stylus relative to the device and detects changes in the position state of the stylus relative to the device. As described above, in some embodiments, device 100 or device 300 uses position module 131 to determine the position state of the stylus relative to the device and changes in the position state of the stylus (in addition to or instead of using position module 431 for the stylus).

[0185] Haptic feedback module 433 includes various software components for generating instructions used by one or more tactile output generators 463 to produce tactile output at one or more locations on stylus 203 in response to user interaction with stylus 203 .

[0186] GPS module 435 determines the location of the stylus and provides this information for use in various applications (eg, to applications that provide location-based services, such as applications for finding lost devices and / or accessories).

[0187] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 402 optionally stores a subset of the above modules and data structures. In addition, memory 402 optionally stores other modules and data structures not described above.

[0188] FIG. 5A to FIG. 5B The position state of the stylus 203 relative to the touch-sensitive surface (e.g., the touch screen 112 of the device 100) according to some embodiments is shown. In some embodiments, the position state of the stylus 203 corresponds to (or indicates): the projected position of the tip (or other representative portion) of the stylus on the touch-sensitive surface (e.g., Figure 5A 504), the orientation of the stylus relative to the touch-sensitive surface (e.g., Figure 5A 506), the tilt of the stylus relative to the touch-sensitive surface (e.g., Figure 5B 512) and / or the distance of the stylus relative to the touch-sensitive surface (e.g., Figure 5B In some embodiments, the position state of stylus 203 corresponds to (or indicates) the pitch, yaw, and / or roll of the stylus (e.g., the attitude of the stylus relative to a particular reference frame such as a touch-sensitive surface (e.g., touch screen 112) or the ground). In some embodiments, the position state includes a set of position parameters (e.g., one or more position parameters). In some embodiments, the position state is detected based on one or more measurement results from stylus 203 sent to an electronic device (e.g., device 100). For example, the stylus measures the tilt of the stylus (e.g., Figure 5B of tilt 512) and / or orientation (e.g., Figure 5A The device 100 may further include a touch screen 112 of the device 100 and a touch-sensitive surface 203. The touch-sensitive surface 203 may be used to detect the position state of the stylus 203 based on raw output from one or more electrodes in the stylus 203, such as the touch screen 112 of the device 100. The touch-sensitive surface 203 may be used to detect the position state of the stylus 203 based on raw output from one or more electrodes in the stylus 203, such as the touch screen 112 of the device 100. For example, the touch-sensitive surface may receive raw output from one or more electrodes in the stylus and calculate the tilt and / or orientation of the stylus based on the raw output (optionally in combination with position state information provided by the stylus based on sensor measurements generated by the stylus).

[0189] Figure 5A 2 shows stylus 203 relative to a touch-sensitive surface (e.g., touch screen 112 of device 100) from a perspective directly above the touch-sensitive surface in accordance with some embodiments. Figure 5A , the z-axis 594 points out of the page (i.e., in a direction perpendicular to the plane of the touch screen 112), the x-axis 590 is parallel to a first edge of the touch screen 112 (e.g., length), the y-axis 592 is parallel to a second edge of the touch screen 112 (e.g., width), and the y-axis 592 is perpendicular to the x-axis 590.

[0190] Figure 5AThe tip of stylus 203 is shown at (x, y) position 504. In some embodiments, the tip of stylus 203 is the end of the stylus that is configured to determine the proximity of the stylus to a touch-sensitive surface (e.g., touch screen 112). In some embodiments, the projection of the tip of the stylus onto the touch-sensitive surface is an orthogonal projection. In other words, the projection of the tip of the stylus onto the touch-sensitive surface is a point at the end of a line from the tip of the stylus to the surface of the touch-sensitive surface that is perpendicular to the touch-sensitive surface (e.g., (x, y) position 504, where the tip of the stylus would touch the touch-sensitive surface if the stylus was moved directly along a path perpendicular to the touch-sensitive surface). In some embodiments, the (x, y) position at the lower left corner of touch screen 112 is position (0, 0) (e.g., (0, 0) position 502) and other (x, y) positions on touch screen 112 are relative to the lower left corner of touch screen 112. Alternatively, in some embodiments, the (0,0) position is located at another location on the touch screen 112 (e.g., at the center of the touch screen 112) and the other (x,y) positions are relative to the (0,0) position of the touch screen 112.

[0191] in addition, Figure 5A Stylus 203 is shown having orientation 506. In some embodiments, orientation 506 is the orientation of a projection of stylus 203 on touch screen 112 (e.g., an orthogonal projection of the length of stylus 203 or a line corresponding to a line between projections of two different points of stylus 203 on touch screen 112). In some embodiments, orientation 506 is relative to at least one axis in a plane parallel to touch screen 112. In some embodiments, orientation 506 is relative to a single axis in a plane parallel to touch screen 112 (e.g., axis 508, with a clockwise rotation angle ranging from 0 degrees to 360 degrees starting from axis 508, as shown in FIG. 1 ). Figure 5A Alternatively, in some embodiments, orientation 506 is relative to a pair of axes in a plane parallel to touch screen 112 (e.g., x-axis 590 and y-axis 592, as shown in FIG. Figure 5A as shown, or a pair of axes associated with an application displayed on the touch screen 112).

[0192] In some embodiments, an indication (e.g., indication 516) is displayed on a touch-sensitive display (e.g., touch screen 112 of device 100). In some embodiments, indication 516 shows the location where the stylus will touch (or mark) the touch-sensitive display before the stylus touches the touch-sensitive display. In some embodiments, indication 516 is part of a mark drawn on the touch-sensitive display. In some embodiments, indication 516 is separate from the mark drawn on the touch-sensitive display and corresponds to a virtual "pen tip" or other element indicating the location where the mark will be drawn on the touch-sensitive display.

[0193] In some embodiments, indication 516 is displayed based on the position state of stylus 203. For example, in some cases, indication 516 is displaced from (x, y) position 504 (e.g., Figure 5A and Figure 5B ), and in other cases, indication 516 is not displaced from (x,y) position 504 (for example, when tilt 512 is zero degrees, indication 516 is displayed at or near (x,y) position 504). In some embodiments, indication 516 is displayed with varying colors, sizes (or radius or area), opacity, and / or other characteristics depending on the position state of the stylus. In some embodiments, the displayed indication takes into account the thickness of the glass layer on the touch-sensitive display so that the indication is implemented "on the pixels" of the touch-sensitive display rather than displaying the indication on the "glass" covering the pixels. As used herein, "indication" may also be used to refer to a location on a touch screen that is not visible when describing an exemplary user interface.

[0194] Figure 5B 2 shows stylus 203 relative to a touch-sensitive surface (e.g., touch screen 112 of device 100) viewed from a side view of the touch-sensitive surface according to some embodiments. Figure 5B , the z-axis 594 points in a direction perpendicular to the plane of the touch screen 112, the x-axis 590 is parallel to a first edge (e.g., length) of the touch screen 112, the y-axis 592 is parallel to a second edge (e.g., width) of the touch screen 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0195] Figure 5B Stylus 203 is shown having tilt 512. In some embodiments, tilt 512 is an angle relative to a normal to the surface of the touch-sensitive surface (also referred to simply as the normal to the touch-sensitive surface) (e.g., normal 510). Figure 5B As shown, when the stylus is perpendicular / perpendicular to the touch-sensitive surface (eg, when stylus 203 is parallel to normal 510), tilt 512 is zero, and increases as the stylus tilts closer to being parallel to the touch-sensitive surface.

[0196] in addition, Figure 5B 514 is shown relative to the touch-sensitive surface of the stylus 203. In some embodiments, the distance 514 is the distance from the tip of the stylus 203 to the touch-sensitive surface in a direction perpendicular to the touch-sensitive surface. Figure 5B , distance 514 is the distance from the tip of stylus 203 to (x,y) position 504 .

[0197] Although the terms "x-axis", "y-axis", and "z-axis" are used herein to illustrate certain directions in certain drawings, it should be understood that these terms do not refer to absolute directions. In other words, the "x-axis" can be any corresponding axis, and the "y-axis" can be a specific axis different from the x-axis. Typically, the x-axis is perpendicular to the y-axis. Similarly, the "z-axis" is different from the "x-axis" and the "y-axis", and is typically perpendicular to both the "x-axis" and the "y-axis".

[0198] in addition, Figure 5B A rotation 518 is shown - a rotation about the length (major axis) of stylus 203 .

[0199] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0200] Fig. 6A An exemplary user interface for a menu of applications on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 600 includes the following elements, or a subset or superset thereof:

[0201] One or more signal strength indicators 602 of one or more wireless communications such as cellular signals and Wi-Fi signals;

[0202] Time 604;

[0203] Bluetooth indicator 605;

[0204] Battery status indicator 606;

[0205] A tray 608 with icons for commonly used applications, such as:

[0206] o An icon 616 of the phone module 138 labeled "Phone," which optionally includes an indicator 614 of the number of missed calls or voice messages;

[0207] o an icon 618 of the email client module 140 labeled “Mail”, which optionally includes an indicator 610 of the number of unread emails;

[0208] o An icon 620 labeled "Browser" of the browser module 147; and

[0209] ○ Video and music player module 152 (also known as iPod (trademark of Apple Inc.)

[0210] module 152) and an icon 622 labeled "iPod"; and

[0211] Icons for other apps, such as:

[0212] o Icon 624 labeled "Text" of the IM module 141;

[0213] o An icon 626 labeled “Calendar” of the calendar module 148;

[0214] o Icon 628 labeled “Photos” of the image management module 144;

[0215] o An icon 630 labeled “Camera” of the camera module 143;

[0216] ○ Icon 632 labeled “Online Video” of the online video module 155;

[0217] ○ Icon 634 labeled “Stock Market” of the Stock Market Desktop Applet 149-2;

[0218] o An icon 636 labeled “Map” of the map module 154;

[0219] ○ Icon 638 labeled “Weather” of the weather widget 149-1;

[0220] ○ Icon 640 labeled as “clock” of the alarm clock desktop applet 169-6;

[0221] o An icon 642 labeled “Fitness Support” of the fitness support module 142;

[0222] o An icon 644 labeled "Notepad" of the Notepad module 153; and

[0223] ○ An icon 646 for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0224] It should be noted that Fig. 6A The icon labels shown in are exemplary only. For example, in some embodiments, icon 622 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0225] Figure 6B A touch-sensitive surface 651 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contacts on touch-sensitive surface 651, and / or one or more tactile output generators 359 for generating tactile outputs for a user of device 300.

[0226] Figure 6B A touch-sensitive surface 651 (eg, Figure 3 A device (e.g., a tablet or touchpad 355) Figure 3 In some embodiments, a touch-sensitive surface (e.g., Figure 6B 651) has a main axis (e.g., Figure 6B 653) corresponding to the main axis (for example, Figure 6B According to these embodiments, the device detects contact with the touch-sensitive surface 651 at a location corresponding to a corresponding location on the display (e.g., Figure 6B 660 and 662 in (e.g., in Figure 6B , 660 corresponds to 668 and 662 corresponds to 670). Thus, on a touch-sensitive surface (e.g., Figure 6B 651) and a display of a multi-function device (e.g., Figure 6B 650 in the figure is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 660 and 662 and their movement) is used by the device to manipulate the user interface on the display. Some of the following examples will be given with reference to a device that detects input on a touch-sensitive surface that is separated from the display, such as Figure 6B As shown. Some of the following examples will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and the display are combined). It should be understood that similar methods are optionally used for other user interfaces described herein.

[0227] In addition, although some of the following examples are given primarily with reference to finger input (e.g., finger contact, finger tap gesture, finger press gesture, and finger swipe gesture, etc.), it should be understood that in some embodiments, one or more finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus-based input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is located above the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0228] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 6B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 651 in FIG. 6A , the particular user interface element is adjusted according to the detected input. In the case that the touch-sensitive surface 651 in FIG. 6B includes a touch screen display (e.g., Figure 1A A touch-sensitive display system 112 or Fig. 6AIn some implementations of a touch screen in a touch screen display, a contact detected on the touch screen acts as a "focus selector" such that when an input (e.g., a press input by a contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted in accordance with the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without a corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the specific form that the focus selector takes, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to convey a user-desired interaction with the user interface (e.g., by indicating to the device an element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display). As used herein, a "focus selector" may be a visible "indicator" or an invisible "indicator" of a hovering input object (eg, a finger or stylus).

[0229] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a surrogate (proxy) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). The intensity of the contact is used as an attribute of the user input, thereby allowing the user to access additional device functions that the user cannot easily access on a smaller device with limited real estate for displaying indications (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control such as a knob or button).

[0230] In some embodiments, contact / motion module 130 and / or contact / motion module 430 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some embodiments, a software setting is provided to the user of the device for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0231] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more strength thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding option or give up performing the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0232] In some embodiments, a portion of a gesture is identified for determining a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an end position (e.g., a drag gesture), where the strength of the contact increases. In this embodiment, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the strength of the swipe contact to achieve the purpose of determining the characteristic strength.

[0233] The user interface diagrams described below (e.g., Figures 7A to 7AU ) optionally includes various intensity graphs showing the intensity of contacts on the touch-sensitive surface relative to one or more intensity thresholds (eg, contact detection intensity threshold IT 0 , Light press intensity threshold IT L , Deep compression intensity threshold IT D and / or one or more other intensity thresholds). This intensity graph is typically not part of the displayed user interface, but is provided to help interpret the graph. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from that typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a characteristic intensity is detected that is below the light press intensity threshold (e.g., and above the nominal contact detection intensity threshold IT 0 , contacts lower than the nominal contact detection intensity threshold are no longer detected), the device moves the focus selector based on the movement of the contact on the touch-sensitive surface without performing the actions associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface figures.

[0234] In some embodiments, the response of the device to the input detected by the device depends on a criterion based on the contact strength during the input. For example, for some "light press" inputs, the strength of the contact exceeding the first strength threshold during the input triggers a first response. In some embodiments, the response of the device to the input detected by the device depends on a criterion including both the contact strength during the input and a time-based criterion. For example, for some "deep press" inputs, as long as a delay time passes between satisfying the first strength threshold and satisfying the second strength threshold, the strength of the contact exceeding the second strength threshold greater than the first strength threshold of the light press during the input triggers a second response. The duration of the delay time is typically less than 200ms (e.g., 40ms, 100ms, or 120ms, depending on the magnitude of the second strength threshold, where the delay time increases as the second strength threshold increases). The delay time helps avoid accidental deep press inputs. As another example, for some "deep press" inputs, a period of reduced sensitivity will occur after reaching the first strength threshold. During this period of reduced sensitivity, the second strength threshold increases. This temporary increase in the second strength threshold also helps avoid accidental deep press inputs. For other deep press inputs, the response to detecting the deep press input is not dependent on time-based criteria.

[0235] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Exemplary factors are described in U.S. patent application serial numbers 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0236] The contact characteristic strength is below the light press strength threshold IT L The intensity increases to between the light pressure intensity threshold IT L and deep compression intensity threshold IT D The intensity between IT and the deep press intensity threshold is sometimes referred to as a "light press" input. D The intensity of the deep compression increases to above the deep compression intensity threshold IT D The strength of the contact feature is sometimes called a "deep press" input. The contact feature strength is below the contact detection strength threshold IT 0 The intensity increases to between the contact detection intensity threshold IT 0 With light pressure intensity threshold IT L The strength between is sometimes referred to as detecting contact on the touch surface. The characteristic strength of the contact is above the contact detection strength threshold IT 0 The intensity decreases to below the contact detection intensity threshold IT0 The strength of IT is sometimes referred to as detecting a contact lift-off from the touch surface. 0 In some embodiments, IT 0 Greater than zero In some examples, a shaded circle or ellipse is used to represent the intensity of a contact on a touch-sensitive surface. In some examples, an unshaded circle or ellipse is used to represent a corresponding contact on a touch-sensitive surface without specifying the intensity of the corresponding contact.

[0237] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the strength of the corresponding contact increases to above the press input strength threshold (e.g., the corresponding operation is performed on a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and the strength of the contact subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., the corresponding operation is performed on an "up stroke" of the corresponding press input).

[0238] In some embodiments, the device employs intensity hysteresis to avoid unexpected input, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold in a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units less than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., performing the corresponding operation on an "up stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from an intensity equal to or below a hysteresis strength threshold to an intensity equal to or above a press input strength threshold and, optionally, the contact strength subsequently decreases to an intensity equal to or below the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact strength or a decrease in contact strength, depending on the circumstances).

[0239] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input are optionally triggered in response to detecting that the intensity of the contact increases above the press input intensity threshold, the intensity of the contact increases from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, the intensity of the contact decreases below the press input intensity threshold, or the intensity of the contact decreases below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where operations are described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operations are optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold. As described above, in some embodiments, triggering of these operations is also dependent on satisfying a time-based criterion (e.g., a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold).

[0240] User interface and associated processes

[0241] Attention now turns to embodiments of a user interface ("UI") and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300 having a display, a touch-sensitive surface, and (optionally) one or more sensors for detecting intensity of contacts with the touch-sensitive surface.

[0242] Figures 7A to 7AU , Figures 8A to 8V , 9A to 9O , FIG. 10A to FIG. 10Y , FIG. 11A to FIG. 11J , FIG. 12A to FIG. 12R , FIG. 13A to FIG. 13N as well as FIG. 14A to FIG. 14L Exemplary user interfaces for interacting with user interface objects through proximity-based input and contact-based input according to some embodiments are shown. The user interfaces in these figures are used to illustrate the processes described below, including FIG. 15A to FIG. 15C , FIG. 16A to FIG. 16E , 17A to 17C , FIG. 18A to FIG. 18G , FIG. 19A to FIG. 19E , FIG. 20A to FIG. 20I , FIG. 21A to FIG. 21D , FIG. 22A to FIG. 22G , FIG. 23A to FIG. 23E and 24A to Fig.24E Although some of the following examples will be given with reference to input on a touch-sensitive surface 651 that is independent of the display 650, in some embodiments, the device detects input on a touch screen display (in which the touch-sensitive surface and the display are combined), such as Fig. 6AAlthough some of the examples that follow will be given with reference to input on a touch screen display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface 651 that is separate from the display 650, such as Figure 6B as shown in .

[0243] FIG. 7A to FIG. 7U The process of interacting with a user interface object (e.g., including picking up, moving, and dropping a user interface object) through proximity-based input and contact-based input by an input object (e.g., a finger or a stylus, such as stylus 203) is shown. Although the example is given with reference to input provided by stylus 203, in some embodiments, the device detects input provided by other types of input objects (such as fingers, sticks, handheld controllers, etc.) to perform the operations shown below.

[0244] Input parameters for input provided by an input object (and detected by an electronic device (e.g., device 100)) include a hover proximity parameter, which is a distance (e.g., distance 514) (or calculated based on the distance) between a predetermined portion of the input object (e.g., the tip of a finger or the tip of a stylus) and a touch-sensitive surface (e.g., touch screen 112 or touch-sensitive surface 651). In some embodiments, the input parameters include a lateral position (e.g., (x, y) position 504) of the input object (e.g., finger or stylus 203) relative to the touch-sensitive surface. In some embodiments, the input parameters include a three-dimensional position state and movement of the input object, such as relative to Figure 5A and Figure 5B In some embodiments, when the input object contacts the touch-sensitive surface, the input parameters include characteristic intensity and location of the contact of the input object (e.g., the tip of a finger or stylus) with the touch-sensitive surface.

[0245] exist Fig. 7A , user interface 702 is displayed on display 650. User interface 702 includes a plurality of user interface objects (e.g., represented by a row and column of circles), including user interface object 704 (e.g., an application launch icon, a button, etc.). When the tip of stylus 203 is maintained at a threshold hovering distance (e.g., distance 514) from the touch-sensitive surface, the device optionally does not display any visual indicator or focus selector corresponding to stylus 203 on the display.

[0246] Figure 7BThe device optionally displays a visual indicator (e.g., indicator 706 (e.g., hand pointer 706-a)) when the tip of stylus 203 moves toward the touch-sensitive surface and remains within a threshold hovering distance of the touch-sensitive surface. Indicator 706 is displayed in user interface 702 at a location corresponding to a location on the touch-sensitive surface (e.g., location 701) that is determined based on the lateral position of stylus 203 (e.g., (x, y) position 504) and, optionally, the vertical position and / or position state of stylus 203. Figure 7B As shown, position 701 deviates from (x,y) position 504 of stylus 203 when the tip of stylus 203 is above and not in contact with touch-sensitive surface 651. The offset is optionally determined based on vertical distance 514 and optionally based on the position state of stylus 203.

[0247] FIG. 7B to FIG. 7C It is shown that when stylus 203 moves laterally while being held above the touch-sensitive surface and within a threshold hovering distance of the touch-sensitive surface (in other words, when stylus 203 is within a hovering range above the touch-sensitive surface), indicator 706 moves according to the lateral movement of stylus 203. Figure 7C , lateral movement of stylus 203 stops when indicator 706 reaches user interface object 704. In response to detecting that stylus 203 is hovering over a location on the touch-sensitive surface that corresponds to the location of user interface object 704, the device changes the appearance of user interface object 704 (e.g., slightly enlarging user interface object 704).

[0248] FIG. 7D to FIG. 7F It is shown that as stylus 203 moves vertically up and down within the hovering range, the device dynamically changes the appearance of user interface object 704 according to the current hovering distance of stylus 203. In this example, the device enlarges the size of user interface object 704 as the hovering distance (e.g., represented by distance 514) decreases, and reduces the size of user interface object 704 as the hovering distance increases.

[0249] Figures 7G to 7HIt is shown that when stylus 203 is within a threshold hovering distance of the touch-sensitive surface, lateral movement of stylus 203 affects the shape of user interface object 704 when indicator 706 is within a threshold distance (e.g., three quarters of the diameter of the user interface object) from the original position of user interface object 704 (e.g., the position of the center of mass of the user interface object). In other words, lateral movement of indicator 706 away from the original position of user interface object 704 stretches user interface object 704 in the direction of movement of indicator 706 until indicator 706 moves laterally beyond the threshold distance from the original position of user interface object 704. In some embodiments (not shown), the threshold distance is shorter when stylus 203 is away from the touch-sensitive surface, and a smaller amount of stretch can be applied to object 704 before object 704 quickly resumes its original shape.

[0250] Fig.7I It is shown that when the indicator 706 moves beyond the threshold distance from the original position of the user interface object 704, the user interface object 704 quickly returns to its original position and shape. In some embodiments, the user interface object may be pulled away from its original position for a short distance as the indicator 706 moves, but once the indicator 706 moves beyond the threshold distance from the original position of the object 704, it quickly returns to its original position.

[0251] Figure 7JStylus 203 is shown to be in initial contact with touch-sensitive surface 651. When stylus 203 is in initial contact with the touch-sensitive surface, the device changes the appearance of indicator 706 to indicate that contact has been made and user interface object 704 is no longer in a hovering state, and interaction with the user interface object will be performed through contact between stylus 203 and the touch-sensitive surface. In addition, the device also changes the appearance of user interface object 704 (e.g., further enlarging user interface object 704 and highlighting user interface object 704 compared to the size before stylus 203 contacted the touch-sensitive surface) to indicate that the user interface object is now being manipulated by touch input rather than hover input. In some embodiments, the change in the size of object 704 is continuous during the transition between the object state before and after stylus 203 contacted the touch-sensitive surface. In some embodiments, the device generates an audio output (e.g., audio signal 708) to indicate that stylus 203 has been in initial contact with the touch-sensitive surface. The audio feedback (e.g., by audio signal 708) augments the visual feedback (e.g., magnifying and highlighting user interface object 704) to alert the user that object 704 has exited the hover state and entered the contact state, and that the user interface behavior will be different in the contact state compared to the hover state. In some embodiments, if stylus 203 moves laterally while maintaining contact with a touch-sensitive surface (not shown), indicator 706 moves in accordance with the lateral movement of stylus 203, and object 704 is dragged with it into user interface 702. In some embodiments, in order to distinguish the dragging of an object from the hover input provided by stylus 203, in response to the lateral movement of stylus 203 while maintaining contact with the touch-sensitive surface, object 704 moves at the same speed as indicator 706 and does not lag behind indicator 706.

[0252] Figures 7K to 7L shows when contact between stylus 203 and the touch-sensitive surface is maintained for more than a threshold amount of time (e.g., T 长按压 ) causes the device to register a long press input (e.g., stylus 203 has not moved more than a threshold amount before the long press input is registered), user interface object 704 is picked up by stylus 203. Upon detecting that stylus 203 remains in contact with the touch-sensitive surface for more than a long press time threshold, the device changes the appearance of indicator 706 (e.g., from an open hand 706-b to a grasping hand 706-c) to show that user interface object 704 is attached to indicator 706. The device provides a tactile output (e.g., a discrete tactile output 710) to indicate that user interface object 704 is picked up and is ready to be lifted and moved with stylus 203. In some embodiments, the device shows that user interface object 704 appears to jump from its original z position toward indicator 706. In some embodiments, the generation of tactile output 710 is synchronized with the attachment of user interface object 704 to indicator 706.

[0253] Figure 7M to Figure 7R 7 shows a process in which user interface object 704 is lifted and moves along with indicator 706 in response to movement of stylus 203 while stylus 203 is hovering over the touch-sensitive surface. FIG. 7M to FIG. 7O , after stylus 203 is lifted off the touch-sensitive surface, stylus 203 moves laterally while moving away from the touch-sensitive surface. As the distance between stylus 203 and the touch-sensitive surface increases (as shown by the increasing value of distance 514), the size of indicator 706 increases. Additionally, as the distance between stylus 203 and the touch-sensitive surface increases, the size of object 704 increases accordingly. Figure 7O to Figure 7P , it is shown that as the distance between the stylus 203 and the touch-sensitive surface decreases, the size of the indicator 706 decreases. In addition, as the distance between the stylus 203 and the touch-sensitive surface decreases, the size of the object 704 also decreases.

[0254] exist Figure 7Q In some embodiments, the stylus is raised above the touch-sensitive surface to a distance very close to the edge of the hovering range (e.g., a distance from the edge of the hovering range less than a threshold value (e.g., 10% of the threshold hovering distance)), and if stylus 203 is not immediately brought close to the touch-sensitive surface (e.g., within a threshold amount of time (e.g., 300 milliseconds)), the device generates visual feedback (e.g., a flashing edge 712 of indicator 702) indicating that stylus 203 is about to exit the hovering range and the hover input is about to terminate. In some embodiments, the device generates tactile output 714 (and / or an audio alert) and displays visual feedback to indicate that stylus 203 is about to exit the hovering range. Figure 7R The stylus is shown to be closer to the touch-sensitive surface without exiting the hovering range, and the appearance of indicator 706 is restored (eg, flashing edge 712 is no longer displayed). Object 704 is resized according to the current hovering distance of stylus 203 .

[0255] exist Figure 7M to Figure 7R , it is shown that the size of object 704 changes dynamically according to the change in the hovering distance of stylus 203, while object 704 moves laterally according to the lateral movement of indicator 706 (and according to the lateral movement of stylus 203 over the touch-sensitive surface). In addition, during the lateral movement of indicator 706, the movement of object 704 lags slightly behind the movement of indicator 704, so when indicator 706 changes its direction of movement, or suddenly slows down or accelerates, the relative position of indicator 706 and object 704 changes slightly to reflect the lag.

[0256] Figure 7S The stylus 203 is shown picking up and moving a user interface object 704 (eg, Figure 7M to Figure 7R704 ). After the stylus 203 contacts the touch-sensitive surface again to drop the user interface object 704, in some embodiments, upon detecting a second contact between the stylus 203 and the touch-sensitive surface, the device changes the appearance of the indicator 706 (e.g., from a grasping hand 706-c to an open hand 706-d) to indicate that the object 704 will be dropped at the current location. In some embodiments, the device shows that the object 704 is detached from the indicator 706. The device optionally generates a tactile output (tactile output 716) and the object 704 falling on the surface of the user interface 702.

[0257] Figure 7T 704 is shown, after the object 704 is dropped, the stylus 203 is lifted off the touch-sensitive surface and the appearance of the indicator 706 is changed to indicate that the stylus 203 is again in the hovering state. If the stylus 203 is now moved laterally while hovering over the touch-sensitive surface, the object 706 will be stretched by the movement indicator 706 but will not move with the indicator 706 (e.g., as previously shown in FIG. Figures 7G to 7H ).

[0258] Figure 7U It is shown that after object 704 is dropped and lifted off the touch-sensitive surface, stylus 203 is then lifted out of the hovering range so that stylus 203 is no longer in a hovering state. Indicator 706 is no longer displayed, and object 704 is displayed in its new position and resumes its original appearance.

[0259] Figure 7V to Figure 7AJThe process of device 100 changing the appearance of object 704 in response to continuous input is shown, the continuous input including a first part of the input, in which stylus 203 is hovering above the touch-sensitive surface, followed by a second part of the input, in which stylus 203 is in contact with the touch-sensitive surface, and then a third part of the input, in which stylus is lifted off the touch-sensitive surface and is again in a hovering state. In this example, the device dynamically changes the appearance of object 704 in a first manner based on the hovering distance of stylus 203 before stylus 203 initially contacts the touch-sensitive surface. After stylus 203 initially contacts the touch-sensitive surface, the device dynamically changes the appearance of object 704 in a second manner based on the contact intensity of stylus 203. In some embodiments, at least one change in appearance (e.g., a change in the size of object 704) is continuous during the transition between the hovering state and the contact state. In some embodiments, at least one change in appearance (e.g., a change in the color of object 704) is discontinuous during the transition between the hovering state and the contact state. Additionally, when stylus 203 ceases initial contact with the touch-sensitive surface, the device dynamically changes the appearance of object 704 based on the hovering distance of stylus 203. In some embodiments, at least one change in appearance (e.g., a change in the size of object 704) is continuous during the transition between the contact state and the hovering state. In some embodiments, at least one change in appearance (e.g., a change in the color of object 704) is discontinuous during the transition between the contact state and the hovering state.

[0260] Figure 7U to Figure 7V It is shown that when stylus 203 enters the hover range from over the touch-sensitive surface at a location corresponding to the location of object 704 in user interface 702 , the device displays indicator 706 at the location of object 704 in user interface 702 .

[0261] Figure 7V to Figure 7X As stylus 203 approaches the touch-sensitive surface from above, device 100 is shown increasing the size of object 704 as the hovering distance of stylus 203 decreases (e.g., as represented by decreasing distance 514) before stylus 203 makes initial contact with the touch-sensitive surface. In some embodiments, the device also increases the size of indicator 706 as the hovering distance of stylus 203 decreases.

[0262] Figure 7Y 2 shows initial contact of stylus 203 with the touch-sensitive surface. Figure 7Y , it is shown that once the stylus 203 contacts the touch-sensitive surface and the contact intensity satisfies the detection threshold IT 0, the device changes the appearance of indicator 706 (e.g., from hand pointer 706-a to open hand 706-b) to indicate that stylus 203 has exited the hovering state and is now in contact. In addition, when contact between stylus 203 and the touch-sensitive surface is detected (e.g., when the contact intensity is detected to meet the detection threshold IT 0 ), device 100 changes the appearance of object 704 (e.g., by highlighting object 704 or changing the color of object 704) to indicate that object 704 will now respond to changes in the contact intensity of contact between stylus 203 and the touch-sensitive surface. The transition from the non-highlighted state to the highlighted state is an abrupt change in the appearance of object 704, which occurs when stylus 203 makes contact with the touch-sensitive surface, or when the contact intensity reaches contact detection intensity threshold IT. 0 In some embodiments, when the stylus touches the touch-sensitive surface and the contact intensity is below the detection threshold intensity IT 0 In the previous intermediate stage, the device applies an animation transition to change the size of object 704 from the state before stylus 203 touches the touch-sensitive surface (e.g., to increase the size) to the state when the contact intensity between stylus 203 and the touch-sensitive surface meets the detection intensity threshold IT. 0 In other words, when the stylus 203 transitions from the hovering state to the contacting state, the change in the size of the object 704 (eg, the size increases) is a smooth transition.

[0263] Figure 7Z to Figure 7AB 7 shows that when stylus 203 contacts the touch-sensitive surface, device 100 dynamically changes the appearance of object 704 according to the characteristic intensity of the contact between stylus 203 and the touch-sensitive surface (e.g., shown in the intensity meter as contact intensity 655). Figure 7Z to Figure 7AB As shown, as the contact strength between stylus 203 and the touch-sensitive surface increases, the device increases the size of object 704. In some embodiments, when the contact strength between stylus 203 and the touch-sensitive surface increases, the device also optionally increases the size of indicator 706.

[0264] It can be seen that as stylus 203 approaches touch-sensitive surface 651, contacts touch-sensitive surface 651, and presses against touch-sensitive surface 651 with increasing intensity, the size of object 704 increases continuously. In other words, when the size of object 704 is changed first according to the reduced hovering distance in the hovering state and then according to the increased contact intensity in the contact state, the increased contact intensity in the contact state is viewed as a continuation of the reduced hovering distance in the hovering state. In contrast, when stylus 203 initially contacts the touch-sensitive surface, the color or highlighting of object 704 undergoes an abrupt transition. In other words, the device indicates the transition from the hovering state to the contact state by applying an abrupt change to the appearance of object 704.

[0265] Figure 7AB It is also shown that when the contact strength between the stylus 203 and the touch-sensitive surface reaches the light press strength threshold IT L When the contact intensity reaches the light press intensity threshold (and the contact does not move more than a threshold amount before the light press intensity threshold is satisfied), device 100 performs an operation associated with object 704, such as displaying menu 718 associated with object 704. In some embodiments, before the contact intensity reaches the light press intensity threshold (for example, after the contact intensity reaches a cue intensity threshold that is lower than the light press intensity threshold), a cue animation (for example, a visual change of object 704 or user interface 702 (for example, blurring and dimming of user interface 702 other than object 704)) is optionally displayed.

[0266] Figures 7AB to 7AD It is shown that when stylus 203 maintains contact with the touch-sensitive surface, stylus 203 gradually reduces the strength of contact with the touch-sensitive surface. When the contact strength between stylus 203 and the touch-sensitive surface gradually decreases, the device gradually reduces the size of object 704 according to the decrease in the contact strength between stylus 203 and the touch-sensitive surface. In some embodiments, as the contact strength between stylus 203 and the touch-sensitive surface decreases, the device also gradually reduces the size of indicator 706. In addition, in some embodiments, when the contact strength between stylus 203 and the touch-sensitive surface gradually decreases, menu 718 is still displayed.

[0267] Figure 7AE 2 shows that when the stylus 203 is about to be lifted off the touch-sensitive surface, and the contact strength between the stylus 203 and the touch-sensitive surface has been reduced to the contact detection threshold IT 0 , the device determines that stylus 203 will again transition from the contact state to the hover state. When stylus 203 transitions from the contact state to the hover state, the size of object 704 is the same as the size of object 704 when stylus 203 transitions from the hover state to the contact state immediately after initial contact with the touch-sensitive surface.

[0268] Figure 7AE to Figure 7AHIt is shown that after stylus 203 transitions again from the contact state back to the hovering state, the device increases the size of object 704 as the hovering distance of stylus 203 increases. The size of indicator 706 increases as the hovering distance of stylus 203 increases. The change in the indicator's appearance after stylus 203 is lifted off the touch-sensitive surface is consistent with the change in the appearance of indicator 706 before stylus 203 makes initial contact with the touch-sensitive surface. In other words, the size of indicator 706 increases as the hovering distance of stylus 203 increases, regardless of whether it is before or after stylus 203 makes initial contact with the touch-sensitive surface. In contrast to the behavior of indicator 706, the change in the size of object 704 as a function of the hovering distance after stylus 203 is lifted off the touch-sensitive surface is opposite to the change in the size of object 704 as a function of the hovering distance before stylus 203 makes initial contact with the touch-sensitive surface. In other words, after stylus 203 is lifted off the touch-sensitive surface, the size of object 704 increases as the hovering distance of stylus 203 increases (e.g., as shown in FIG. Figures 7AF to 7AH ), and before stylus 203 makes initial contact with the touch-sensitive surface, the size of object 704 increases as the hovering distance of stylus 203 decreases (e.g., as shown in FIG. 1 ). Figure 7V to Figure 7X shown).

[0269] Figure 7AI It is shown that when stylus 203 is within a threshold distance (e.g., 10% of the threshold hovering distance) away from reaching the edge of the hovering range, if stylus 203 does not immediately (e.g., within 300ms) approach the touch-sensitive surface, the device determines that stylus 203 is about to exit the hovering range. When device 100 detects that stylus 203 is about to exit the hovering range, device 100 provides visual feedback (e.g., a flashing edge 724 of object 704) to indicate that stylus 203 is about to exit the hovering range.

[0270] Figure 7AJ It is shown that when stylus 203 exits the hovering range, indicator 706 is no longer displayed, and object 704 breaks away from indicator 706 and falls to the surface of user interface 702. The size of object 704 also quickly returns to its original state. In some embodiments, in conjunction with object 704 returning to its original state, device 100 generates a tactile output (e.g., tactile output 726).

[0271] Figure 7AK to Figure 7AU A process is shown in which an input object (e.g., a finger) moves and positions a cursor within selectable text (e.g., editable or non-editable text) when the input object moves laterally in a hover state, and in which the input object selects text (e.g., expands a text selection from a current cursor position) when the input object moves laterally in a contact state.

[0272] exist Figure 7AK, a user interface (e.g., user interface 730) is displayed on touch screen 112. User interface 730 includes selectable text 732. When an input object, finger, or hand 734 is above a threshold hovering distance from touch screen 112 (e.g., distance 514 is greater than the threshold hovering distance), an indicator corresponding to the input object is not displayed in user interface 730.

[0273] Figure 7AL It is shown that when an input object (e.g., a finger or hand 734) is within a threshold hovering distance from the touch screen 112, an indicator (e.g., a cursor 738) is displayed within the optional text 732 at a position 701 on the touch screen 112 (e.g., at a position corresponding to the (x, y) position 504 on the touch screen 112). The position 701 deviates from the lateral position (x, y) position 504 of the finger 734, and the offset is optionally determined based on the hovering distance (e.g., represented by the distance 514) and / or the position state (e.g., tilt) of the finger 734. In some embodiments, in addition to the cursor 738, the device 100 also displays a magnifying glass (e.g., a magnifying glass 736) above the optional text 732. The magnifying glass 736 is displayed above the cursor 734 in the user interface 730 and moves with the cursor 732 in the user interface 730. Within the magnifying glass 736, a magnified version of a portion of the selectable text 732 is displayed, and depending on the position of the cursor 738 within the editable text 732, a copy 742 of the cursor 738 is also displayed at a position within the magnified text.

[0274] Figure 7AM It is shown that when the finger 734 moves laterally while remaining within the hovering range, the cursor 738 moves within the selectable text 732 according to the lateral movement of the finger 734. The magnifying glass 736 moves with the cursor 738 and displays the updated portion of the text 732 and the updated position of the cursor 738. Figure 7AM It is also shown that the hover distance does not have to be maintained at a constant value during movement of finger 734 in order to move cursor 738 within selectable text 732.

[0275] Figure 7AN 734 is shown as the cursor 738 moves to a new position within the selectable text 732 when the finger 734 moves laterally to another position on the touch screen 112. A touch press of the finger 734 is detected on the touch screen 112 (e.g., the contact intensity between the finger 734 and the touch screen 112 is above the contact detection threshold IT). 0). Before the finger 734 is moved laterally while in contact with the touch screen 112, the cursor 732 is displayed at its current location within the selectable text 732. A magnifying glass 742 is displayed above the cursor 732. A magnified version of the text surrounding the cursor 732 is displayed within the magnifying glass 736, and the location of the cursor 732 within the selectable text is represented by a copy 742 of the cursor 736 in the magnifying glass.

[0276] Figure 7AO After finger 734 makes contact with the touch screen, finger 734 is shown moving laterally while maintaining contact with touch screen 112. The movement of the contact causes cursor 738 to move within selectable text 732, and the movement of cursor 738 from its previous position (e.g., the cursor position when finger 734 was touch-pressed on touch screen 112) to its current position causes a portion of the text between the cursor's previous position and the current position to be selected (e.g., as shown by selection 740). As the movement of cursor 738 continues as the contact between finger 734 and touch screen 112 moves, selection 740 expands or contracts in accordance with the movement of cursor 738. As shown in FIG. Figure 7AO As shown, selection 740 is also represented within magnifying glass 736 (eg, as selection 744).

[0277] Figure 7AP 740 by moving the contact between the finger 734 and the touch screen 112 and adjusting the end boundary of the selection 740 by moving the cursor 738, the lift-off of the finger 734 is detected. After lifting off from the touch screen 112, the finger 734 is again hovering over the touch screen 112. When the finger 734 is hovering over the touch screen 112, the cursor 738 is disengaged from the selection 740 and moves within the selectable text 732 with the lateral movement of the finger 734 while the finger 734 remains over the touch screen 112 within the hovering range. Figure 7AP As shown, in some embodiments, after the lift-off of contact between the finger 734 and the touch screen 112 is detected, the selection 740 is maintained in the text 732. The magnifying glass continues to move with the cursor 732 and displays a magnified version of the text around the cursor 738.

[0278] Figure 7AQ It is shown that while finger 734 is within the hovering range over touch screen 112, cursor 738 has moved to a new position within text 732 in accordance with the lateral movement of finger 734. Magnifying glass 736 also moves with moving cursor 736 and displays a magnified version of the text surrounding cursor 738. Selection 740 remains at its original position.

[0279] Figure 7AR to Figure 7AS It is shown that when the touch of the finger 734 is detected again at the new position, the selection 740 ( Figure 7ARThe cursor 738 is displayed at a new position (e.g., position 701) that corresponds to the touch position of the finger 734 on the touch screen 112 (e.g., (x, y) position 504), as shown in FIG. Figure 7AR As shown. Figure 7AS , lateral movement of the finger 734 across the touch screen 112 is detected, and the cursor 738 is moved from its previous position (e.g., Figure 7AR The movement of cursor 738 from the previous position of cursor 738 when the second touch of middle finger 734 was pressed to its current position causes a portion of the text between the previous position and the current position of cursor 738 to be selected (e.g., as shown by selection 746). Magnifying glass 736 moves with cursor 738 and displays the current text selection (e.g., represented by magnified copy 748 of selection 746).

[0280] Figure 7AT It is shown that the lift-off of finger 734 is detected, and in response to the lift-off of finger 734 from touch screen 112, cursor 738 disengages from selection 746. As cursor 738 moves away from selection 746 in accordance with the movement of finger 734, selection 746 is maintained.

[0281] Figure 7AU It is shown that when finger 734 is lifted outside of the hover range (e.g., beyond a threshold hover distance above touch screen 112), magnifying glass 736 is replaced by menu 750 (e.g., a "Cut / Copy / Find" menu), and selection 746 changes appearance (e.g., is replaced with selection object 752 having adjustable boundaries). Selection object 752 is optionally moved as a whole to another location by contact (e.g., another contact by finger 734) or resized by a contact (e.g., another contact by finger 734) dragging an adjustable boundary of selection object 752. In some embodiments, when selection object 752 is displayed, device 100 optionally does not display cursor 738 when finger 734 re-enters the hover range. In some embodiments, a tap input by contact (e.g., another contact by finger 734) dismisses menu 750 and cancels the selection represented by selection object 752, and when subsequent hover input is detected, the selection object 752 is again displayed with the cursor 738. Figures 7AK to 7AT Cursor 738 is displayed and positioned in the manner shown.

[0282] Figures 8A to 8V 802 or content display user interface 830) according to some embodiments. In particular, Figures 8A to 8VIllustration of an object moving across a user interface based on lateral movement of an input object while the input object hovers over a touch-sensitive surface, canceling the object's movement when the input object is lifted outside of the hovering proximity range without first contacting the touch-sensitive surface, and confirming the object's movement when the input object contacts the touch-sensitive surface after a lateral movement.

[0283] Fig. 8A A user interface (e.g., home screen user interface 802) is shown that includes many user interface objects corresponding to different applications installed on electronic device 100 (e.g., application launch icon 806 corresponding to a "calculator" application, and folder icon 808 for a folder including one or more application launch icons, application launch icon 810 corresponding to a phone application, application launch icon 812 corresponding to a mail application, application launch icon 814 corresponding to an Internet browser application, etc.). The user interface objects are arranged in the user interface according to a predetermined layout (e.g., according to a four-column grid). Some objects (e.g., application launch icons 810, 812, and 814) are included in a container object (e.g., taskbar 804).

[0284] exist Figure 8B 8, the home screen reconfiguration mode has been triggered and the application launch icon 806 is currently selected, for example, in response to a long press input through contact 814 with the touch-sensitive surface. In the home screen reconfiguration mode, the icons in the home screen user interface can be rearranged by moving one or more icons.

[0285] FIG. 8C to FIG. 8F The selected icon 806 is shown being dragged across the user interface in accordance with lateral movement of the input object (e.g., hand or finger 734) while the input object is hovering over the touch-sensitive surface 112. In some embodiments, a visual indication of the hovering input object (e.g., indication 816) is displayed on the touch screen. In some embodiments, indication 816 is not visible on the touch screen. Figure 8C In the example, when icon 806 is moved out of its original position in the home screen user interface 802, other icons (e.g., icon 808) are rearranged in the home screen user interface 802 according to predefined user interface layout rules. Fig.8D , when icon 806 is dragged near taskbar 804 , other icons (eg, icons 810 , 812 , and 814 ) within taskbar 804 are rearranged to make room for icon 806 .

[0286] FIG. 8C to FIG. 8FIt is also shown that during the lateral movement of the finger 734, when the finger 734 is hovering over the touch screen 112, the appearance of 806 changes dynamically according to the hovering distance 514 of the finger 734. For example, when the finger 734 is hovering higher above the touch screen 112 (e.g., as shown in FIG. FIG. 8C to FIG. 8D ), icon 806 appears larger than when finger 734 is hovering lower on the touch screen (e.g., as shown in FIG. FIG. 8E to FIG. 8F ).

[0287] Figures 8G to 8H 806 across the touch screen in accordance with the lateral movement of the hovering finger 734, the finger 734 contacts the touch screen at a location corresponding to an acceptable drop location for the icon 806 on the touch-sensitive surface (eg, Figure 8G Thus, icon 806 is dropped at an acceptable placement location corresponding to the location of contact 818 on touch screen 112 and remains in that new location after liftoff of contact 818 (e.g., as shown in FIG. 1 ). Figure 8H ). The movement of icon 806 and rearrangement of home screen user interface 802 is confirmed and completed, and the position of the icons on the home screen user interface remains in the new configuration after finger 734 is lifted out of the hover proximity range.

[0288] and Figures 8G to 8H The opposite of the scenario shown, FIG. 8I to FIG. 8J FIG. 8 shows that after icon 806 is moved across the touch screen in accordance with the lateral movement of hovering finger 734, finger 734 is lifted out of the hovering proximity range (eg, Fig.8I As shown, FIG. 8C to FIG. 8F ). As a result, icon 806 flies back to its original position in home screen user interface 806, other icons (e.g., icons 808, 810, 812, and 814) have moved in response to the movement of icon 806; and the movement of icon 806 and the rearrangement of the home screen user interface (e.g., as FIG. 8I to FIG. 8J In other words, after a preview of the rearrangement of the home screen user interface has been presented to the user, when the hover input ends without finger 734 first making contact with touch screen 112, the rearrangement of the home screen user interface is resumed.

[0289] Figures 8K to 8M The icon 806 is shown to be continuously held by the contact 820 from its original position (in Figure 8K ) to the same drop location as before (for example, Fig.8F and Figure 8MAs icon 806 is dragged, other icons in home screen user interface 112 are also rearranged in response to the movement of icon 806 (e.g., Figure 8L and Figure 8M shown).

[0290] exist Figures 8K to 8M Afterwards, in Figures 8N to 8O , finger 734 is lifted off of touch screen 112 and out of the hover proximity range above the touch screen, and icon 806 remains in its new position in taskbar 804 and the home screen user interface remains in the new configuration.

[0291] Figure 8P A content display user interface 830 (e.g., a user interface of a text editor application or a web browser application) is shown. The content display user interface 830 displays selectable text. Within the content display user interface 830, a portion of the text is currently selected, as indicated by a text selection box 843 that includes a portion of the selected text. The text selection box 843 includes a start edge (e.g., a selection handle 836) and an end edge (e.g., a selection handle 838).

[0292] Figure 8Q One of the two selection handles (e.g., selection handle 838) is shown to be selected for subsequent movement (e.g., by contact 842 of finger 734 with the touch screen). In some embodiments, when finger 734 hovers over selection handle 838 (e.g., when the (x, y) position 504 of finger 734 corresponds to the position of selection handle 838), selection handle 838 is selected by another required input such as a predefined hover gesture (e.g., an air finger swing gesture). The selection of selection handle 838 is visually indicated by the enlargement of the end portion of selection handle 838 relative to the corresponding end portion of selection handle 836. In addition, a magnification object (e.g., a magnifying glass 840) is displayed near the selected selection handle 838 to display a portion of the text near the selected selection handle 838. In some embodiments, a cursor 844 is displayed within the text shown in the magnifying glass 840 to indicate the boundaries of the text selection box.

[0293] Figure 8R It is shown that when the finger 734 hovers over the touch screen 112 (e.g., an indication 846 of the hovering finger 734 is shown to indicate the relative positions of the hovering finger 734 and the selected selection handle 38), in accordance with lateral movement of the finger 734, the selected selection handle 838 moves across the touch screen relative to the unselected selection handles 836, thereby changing the size of the text selection 834 (e.g., expanding the text selection 834).

[0294] FIG. 8S to FIG. 8T The selected selection handle 838 is shown after it moves with the hovering finger 734 (as shown in FIG. Figure 8Q to Figure 8R ), finger 734 makes contact with touch screen 112, and as a result, after finger 734 is lifted out of the hover proximity range above touch screen 112, repositioning of selection handle 838 and resizing of text selection object 834 are confirmed and remain effective (e.g., as shown in FIG. Figure 8T shown).

[0295] Figure 8U to Figure 8V Shown, with FIG. 8S to FIG. 8T In contrast to the scenario shown in FIG. 8 , after the selected selection handle 838 moves with the hovering finger 734 (as shown in FIG. Figure 8Q to Figure 8R ), the finger 734 is lifted outside of the hover proximity range without first making contact with the touch screen 112, thereby canceling the repositioning of the selection handle 838 and the resizing of the text selection object 834 when the finger 734 is lifted outside of the hover proximity range above the touch screen 112.

[0296] 9A to 9O A user interaction performed by an input object for displaying a transient visual effect (e.g., a visual cue of a leading image 906 or a quick action menu 914, a mini-app object 912) is shown, and the display of the transient visual effect (e.g., a transient user interface object) is maintained by keeping the input object within a hovering proximity range above the touch screen 112. In addition, the transient visual effect displayed when the input object is hovering in toward the touch screen (e.g., when the input object is entering the hovering proximity range from outside the hovering proximity range and moving within the hovering proximity range before contacting the touch screen) is different from the transient visual effect displayed when the input object is hovering out away from the touch screen (e.g., when the input object is on its way out of the hovering proximity range after contacting the touch screen). In some embodiments, an operation (e.g., launching an application, or activating a menu option) is activated by the input object by contacting and lifting off the touch-sensitive surface. In some embodiments, a quick lift-off of contact by the input object is used to perform an operation without displaying a transient visual effect during the brief period of time that the input object moves through the hovering proximity range.

[0297] Fig. 9A A user interface (eg, home screen user interface 902) is shown that includes a number of objects (eg, application launch icons, including application launch icon 904 for an instant messaging application).

[0298] Fig. 9BIt is shown that when an input object (eg, finger 734 ) hovers over application icon 904 , a quick action menu 914 and a leading image (eg, preview panel 906 ) of mini-app object 912 are displayed. Fig. 9B The relative positions of the input object indication 908 and the icon 904 on the touch screen are shown in FIG. In some embodiments, the size and appearance of the leading image changes dynamically with the hovering distance 514 of the hovering finger 734. Fig. 9B Also shown is that when finger 734 is detected hovering over touch screen 112, portions of home screen user interface 902 outside of icon 904 are dimmed and blurred as a cue indicating that icon 904 is an intensity-responsive icon that will provide an object-specific response (e.g., displaying a quick action menu and mini-app object corresponding to an instant messaging application) in response to a light press input (e.g., as opposed to a tap input). In some embodiments, the amount of dimming and blurring applied to home screen user interface 902 changes dynamically with the hovering distance 514 of hovering finger 734. Although not shown, the display of the leading image and the cue visual effect (e.g., blurring and dimming of home screen user interface 902) are transient and cease to be displayed when finger 734 is lifted out of the hovering proximity range above the touch screen before finger 734 makes any contact with the touch-sensitive surface.

[0299] FIG. 9C to FIG. 9D As finger 734 contacts touch screen 112 (e.g., contacts 910) and increases the intensity of contact (e.g., contacts 910), the leading image and prompt visual effects are further enhanced (e.g., preview disk 906 is enlarged and off-center, and the blurring and dimming of home screen user interface 902 increases). In some embodiments, as finger 734 moves toward, contacts, and presses against the touch screen, the visual effects shown are continuous and non-discrete. Although not shown here, when contact intensity 910 exceeds light press intensity threshold IT L When finger 734 is previously lifted off the touch screen, the device treats the input received by finger 734 as a tap input and launches an application in response to detecting the lift-off of contact 910 .

[0300] Fig.9E When the characteristic intensity of the contact of the finger 734 (eg, contact 910) exceeds the light press intensity threshold IT LWhen the user clicks the icon (as shown by the intensity meter 901), the leading image (e.g., preview panel 906) transforms into a mini-app object 912 and a quick action menu 914 (e.g., a list of selectable options) corresponding to the instant messaging application. When the mini-app object 912 and quick action menu 914 are first displayed, the home screen user interface behind the mini-app object 912 and quick action menu 914 is blurred and dimmed to its maximum extent.

[0301] Fig.9F It is shown that when the finger 734 is lifted off the touch screen 112 but remains within the hovering proximity range above the touch screen, the mini-app object 912 and the quick action menu 914 remain displayed above the blurred and dimmed home screen user interface 902. As the hovering distance of the finger 734 changes, the appearance of the mini-app object 912 and the quick action menu 914 changes to indicate that the display of the mini-app object 912 and the quick action menu 914 is transient and will stop displaying once the finger 734 is lifted out of the hovering proximity range above the touch screen 112. For example, as the finger 734 moves away from the touch screen, the mini-app object 912 and the quick action menu 914 become increasingly translucent. In some embodiments, the mini-app object 912 and the quick action menu 914 are lifted off the z-layer of the home screen user interface along with the hovering finger 734, and optionally vibrate as the finger 734 moves laterally while the finger hovers over the touch screen 112.

[0302] FIG. 9G to FIG. 9H 914 and 915. FIG. 2 shows that finger 734 moves downward and makes contact with touch screen 112 (e.g., as shown by contact 918 on app launch icon 904) before finger 734 is lifted out of hover proximity and while mini app objects 912 and quick action menu 914 are still visible above home screen user interface 902 (e.g., as shown by contact 918 on app launch icon 904). Figure 9G and in response to detecting that the finger 734 is in contact with the touch screen 112 and when the finger 734 is lifted off the touch screen 112 (e.g., in some embodiments, a quick lift with minimal hovering is required), the device launches an instant messaging application (e.g., Figure 9H). In some embodiments, the instant messaging application is launched to display a user interface corresponding to the item at the location of the contact (e.g., contact 918). For example, contact 918 is on the application launch icon 904, and when the contact is lifted (or when it is detected that the finger 734 has exited the hovering proximity range within a threshold amount of time after the contact 918 is lifted), the application is launched to display the conversation list interface 920 of the instant messaging application. If the contact is on one of the avatars in the mini-application object 912, the device optionally launches the instant messaging application to display the conversation interface corresponding to the avatar. If the contact is on one of the selectable options in the quick operation menu 914, the device launches the user interface corresponding to the selectable option. Figure 9G It is shown that when finger 734 contacts the touch screen, mini-app object 912 and quick action menu 914 restore their original appearance. In some embodiments (not shown), the user can move the contact (e.g., contact 918) to a desired location on the touch screen (e.g., above a selectable option in the quick action menu or above an avatar in the mini-app object), and then lift off from that location to cause the application to be launched to display the corresponding user interface. In some embodiments, the device requires finger 734 to exit the hovering proximity range within a threshold amount of time after lifting off the contact in order to launch the application; otherwise, transient user interface objects (e.g., quick action menu and mini-app objects) will be displayed when the finger re-enters the hovering proximity range and remains within the hovering proximity range above the touch screen.

[0303] Fig.9I 914. FIG. 914 shows that if finger 734 moves further away from the touch screen but remains within the hovering proximity range above the touch screen, instead of approaching and contacting the touch screen, mini-app object 912 and quick action menu 914 become increasingly translucent (e.g., more transparent) as finger 734 moves away from the touch screen. Fig.9F In addition, mini-app object 912 and quick action menu 914 are lifted off the home screen user interface along with hovering finger 734, leaving a longer shallow area on the home screen user interface. Fig.9I Also shown is that as the mini-app object 912 becomes more translucent, the amount of blurring and dimming applied to the home screen user interface 902 also decreases, indicating to the user that if the finger 734 continues to move away from the touch screen and exits the hovering proximity range above the touch screen, the home screen user interface 902 will be fully restored.

[0304] Figure 9JIt is shown that the transient user interface objects displayed during the hover-in process (e.g., when the input object moves toward the touch screen and moves into the hover proximity range) and during the hover-out process (e.g., when the input object moves away from the touch screen and moves out of the hover proximity range) are different, and the appearance of the transient user interface objects can change in different ways in response to the changing hover proximity parameters of the input object. Figure 9J In the top row, when finger 734 just enters the hover proximity range above the touch screen and does not make any contact with the touch screen, a preview disk is displayed behind the application launch icon (e.g., as shown in (A)); when the finger approaches the touch screen, the preview disk becomes larger (e.g., as shown in (B)); and when the finger presses against the touch screen, the preview disk expands and moves, and when the contact intensity exceeds the light press threshold, it finally transforms into a mini application object and a quick action menu (as shown in (C)). Figure 9J In the bottom row of , after a finger is in contact with the touch screen and after the mini-app objects and quick action menu are displayed, moving the finger away from the touch screen causes the mini-app objects and quick action menu to become increasingly translucent, indicating that they will no longer be displayed once the finger is lifted out of the hovering proximity range above the touch screen (e.g., as shown in (D) and (E)); and moving the finger toward the touch screen causes the mini-app objects and quick action menu to become increasingly opaque, indicating that they will fully restore their initial appearance when the finger is again in contact with the touch screen (as shown in (F)).

[0305] Figure 9K from Fig.9E Continuing, after the mini-app object 912 and quick action menu 914 are displayed in response to detecting contact intensity 910 exceeding a light press intensity threshold (as shown by intensity meter 901), the contact intensity decreases, but the mini-app object 912 and quick action menu 914 retain their original appearance as long as the contact is maintained.

[0306] Figure 9L and Figure 9M A quick lift-off (e.g., the speed of finger 734 exceeds a first threshold speed and / or finger 734 exits the hover proximity range within a threshold amount of time) is shown, which stops displaying transient user interface objects (e.g., mini-app object 912 and quick action menu 914) even when finger 734 is still moving through the hover proximity range over touch screen 112 (e.g., FIG. 14A ). Figure 9L shown).

[0307] FIG. 9N to FIG. 9O from Fig.9E or Fig.9I Continuing, after the mini-app object 912 and the quick action menu 914 are displayed, the finger 934 does not make contact with the touch screen 112 again. FIG. 9N to FIG. 9O 14. FIG. 14 shows that when finger 934 is slowly lifted out of the hovering proximity range (e.g., the speed of finger 734 is lower than the second threshold speed and / or finger 734 lingers within the hovering proximity range for more than a threshold amount of time), transient user interface objects (e.g., mini-app object 912 and quick action menu 914) are still displayed for a moment (e.g., the mini-app object 912 and quick action menu 914) even after finger 734 exits the hovering proximity range over touch screen 112. Figure 9N shown). Fig.9O The transient user interface object is shown to stay only for a moment (e.g., 200ms) and stops being displayed after that moment. In some embodiments (not shown), if finger 734 reenters the hovering proximity range, the device keeps displaying the transient user interface object and interprets the exit of finger 734 as accidental and ignores it.

[0308] In some embodiments, Figure 9K from Figure 9G Continuing, wherein after mini-app object 912 and quick action menu 914 are displayed when finger 934 is lifted out of the hovering proximity range sufficiently slowly (e.g., the speed of finger 734 is less than the second threshold speed and / or finger 734 hovers within the hovering proximity range for more than a threshold amount of time), finger 934 again contacts touch screen 112. In this case, the application launch operation is not activated upon lifting off the contact, and after finger 734 is lifted off touch screen 112, transient user interface objects (e.g., mini-app object 912 and quick action menu 914) remain displayed.

[0309] FIG. 10A to FIG. 10Y Different device responses to input objects entering and / or leaving different levels of hover proximity range are shown according to some embodiments.

[0310] FIG. 10A to FIG. 10E It is shown that when finger 734 enters the first hovering proximity range from outside the first hovering proximity range, the toolbar is displayed, and when finger 734 enters the second hovering proximity range from outside the second hovering proximity range, the controls in the toolbar are activated or highlighted, wherein the first hovering proximity range is a larger range that includes the second hovering proximity range. For example, the first hovering proximity range corresponds to a larger threshold hovering distance above the touch screen (e.g., threshold hovering distance 1), and the second hovering proximity range corresponds to a smaller threshold hovering distance above the touch screen (e.g., threshold hovering distance 2).

[0311] Fig. 10A A user interface (e.g., browser user interface 1002) is shown. The browser user interface 1002 displays the content of a web page. The browser user interface 1002 is displayed in full screen mode without displaying any menu bar or toolbar of the browser application.

[0312] Fig. 10B It is shown that when finger 734 enters the first level hover proximity range from outside the first level hover proximity range, when the (x, y) position 504 of finger 734 is within a predefined lower edge portion of the touch screen 112 (e.g., as indicated by indication 1004 of finger 734 relative to the touch screen 112), the displayed content in the browser user interface 1002 as a whole moves upward away from the lower edge of the touch screen, and a toolbar (e.g., toolbar 1006) is displayed in the space vacated by the displaced content. In some embodiments (not shown), when finger 734 then moves out of the first level hover proximity range without first entering the second level hover proximity range, the displayed content moves downward again toward the lower edge of the touch screen and the toolbar stops being displayed. Note that in some embodiments, the indication of finger 734 is not visible on the touch screen.

[0313] Fig. 10C It is shown that when finger 734 continues to move toward the touch screen while remaining near the bottom edge of the touch screen (as shown by indication 1004 of finger 734 relative to touch screen 112) and when finger 734 enters the second level hover proximity range from outside the second level hover proximity range, the control enable representation (e.g., enable representation 1008 for sharing displayed content) closest to finger 734 in toolbar 1006 is highlighted and enlarged. In some embodiments, if finger 734 does not continue to move toward the touch screen, but leaves the second level hover proximity range by moving away from the touch screen, the highlighting of control enable representation 1008 is removed and control enable representation 1008 restores its original appearance. Although finger 734 remains within the first level hover proximity range, toolbar 1006 is still displayed with all of its enable representations presenting their original appearance. In some embodiments (not shown), if finger 734 moves laterally to a hover position above another control enable representation in toolbar 1006 while finger 734 remains within the second-level hover proximity range, the control enable representation closest to finger 734 is highlighted and enlarged rather than enable representation 1008.

[0314] FIG. 10D to FIG. 10E It is shown that when finger 734 contacts the touch screen at a touch location corresponding to a control-enable representation in toolbar 1006 (e.g., indicated by contact 1010 on highlighted control-enable representation 1008), an operation corresponding to the selected control-enable representation 1008 is performed (e.g., displaying a share card 1012 showing various sharing options). In some embodiments (not shown), after toolbar 1006 is displayed, moving finger 734 toward the touch screen does not highlight any control-enable representation, and a control-enable representation is activated only when finger 734 contacts the touch screen at a touch location corresponding to a control-enable representation.

[0315] FIG. 10F to FIG. 10M A control panel including one or more control indicators is shown according to some embodiments when the finger 734 enters a first hovering proximity range from outside the first hovering proximity range, and controls in a toolbar are highlighted when the finger 734 enters a second hovering proximity range from outside the second hovering proximity range.

[0316] Fig.10F The media player application is shown operating in a full screen play mode. The user interface 1020 of the media player application displays the video in a playback mode.

[0317] Figure 10G It is shown that when the finger 734 moves toward the touch screen and enters the first level hover proximity range, a control panel (e.g., control panel 1023) having three playback control enable indications (e.g., rewind enable indication 1024, pause enable indication 1026, and fast forward enable indication 1028) is displayed on the media player user interface 1020 while the video continues to play. The position of the control panel 1023 is selected based on the (x, y) position 504 of the finger 734. For example, the center of the control panel 1023 is slightly above the (x, y) position 504 of the finger 734 (e.g., the (x, y) position of the finger 734 is indicated by the indication 1030 relative to the touch screen 112). It should be noted that in some embodiments, the indication of the finger 734 is not visible on the touch screen. Figure 10G Also shown is a swipe 1022 displayed at the top of the user interface 1020 to indicate the current progress of the video playback. The position of the swipe 1022 is predefined and is independent of the (x, y) position 504 of the finger 734.

[0318] Fig. 10H It is shown that when the finger 734 moves laterally while hovering in the first horizontal hover proximity range (but not entering the second level hover proximity range), the control panel 1023 is repositioned on the touch screen according to the (x, y) position 504 of the finger 734.

[0319] Fig.10I It is shown that when finger 734 moves closer to the touch screen and enters the second level hover proximity range, the control enable representation closest to finger 734 (e.g., rewind enable representation 1024) is highlighted relative to other control enable representations in control panel 1023.

[0320] exist Fig.10J , when finger 734 moves laterally while remaining within the second level hover proximity range, control panel 1023 remains stationary and, instead, a different control enable representation (e.g., pause enable representation 1026) that is closest to finger 734 is highlighted.

[0321] Figure 10K It is shown that when finger 734 contacts (e.g., indicated by contact 1032) a control enable indication (e.g., pause enable indication 1026) in control panel 1023, the control enable indication is activated (e.g., pause enable indication 1026 is switched to play enable indication 1026'). In some embodiments, the control enable indication is activated upon contact. In some embodiments, the control enable indication is activated when the contact is lifted.

[0322] Fig.10L It is shown that when the finger 734 is lifted from the touch screen after contacting the pause control 1026 and returns to the first hover proximity range after exiting the second hover proximity range, the highlighting of the control indication stops being displayed and the control panel 1023 remains displayed at a position corresponding to the (x, y) position 504 of the finger 734 (e.g., as shown by the relative position of the control panel 1023 and the indication 1034 of the finger 734).

[0323] exist Figure 10M , after the finger 734 exits the first level hover proximity range, the control panel 1023 and the swipe 1022 stop being displayed, and the video playback in the user interface 1020 remains in a paused state.

[0324] Figures 10N to 10R It is shown that according to some embodiments, when finger 734 hovers over an application icon or a folder icon within a first-level hover proximity range, information about one or more applications in a folder (e.g., missed notifications) is displayed, and when finger 734 enters a second hover proximity range from outside the second hover proximity range, more details of the information (e.g., details of the missed notifications) are displayed.

[0325] Fig.10N A home screen user interface 1042 is shown that includes an application launcher icon and a folder icon 1044. A badge (e.g., an indicator "4") on the folder icon 1044 indicates that there are multiple unread notifications (e.g., four unread notifications) for the application within the folder represented by the folder icon 1044.

[0326] Fig.10OIt is shown that when the finger 734 moves toward the touch screen and enters the first level hover proximity range near the location of the folder icon 1044 (e.g., as indicated by the relative position of the folder icon 1044 and the indication 1046 of the (x, y) position 504 of the finger 734), the notification list 1048 is displayed near the folder icon 1044. The notification list 1048 includes a count of unread notifications for each application within the folder that has unread notifications. Optionally, the portion of the home screen user interface 1042 outside the folder icon 1044 is dimmed and blurred below the notification list 1048. In this example, three of the five applications in the folder have unread notifications, for example, the App Store application has one unread notification, the Messages application has two unread notifications, and the Mail application has one unread notification. In some embodiments (not shown), if the finger 734 now leaves the first level hover proximity range by moving away from the touch screen, the notification list 1048 will stop being displayed, and the notifications remain unread.

[0327] Figure 10P It is shown that when the finger 734 moves close to the touch screen and enters the second-level hover proximity range near the location of the folder icon 1044 (or near any part of the notification list 1048), more details of the unread notifications are displayed. For example, the content of each unread notification is displayed in the extended notification list 1050. In some embodiments, if the finger 734 is lifted off the touch screen and exits the second-level hover proximity range, the extended notification list 150 remains displayed as long as the finger 734 remains in the first-level hover proximity range. In some embodiments, if the finger 734 is lifted off the touch screen and leaves the second-level hover proximity range, the extended notification list 1050 stops being displayed and the notification 1048 is redisplayed as long as the finger 734 remains in the first-level hover proximity range. In some embodiments, once the finger 734 is lifted outside the first-level hover proximity range, the notification list is not displayed, and the status of the previously unread notification is changed to "read" by the device.

[0328] Figure 10QIt is shown that if finger 734 does not exit the first level hover proximity range, but contacts the touch screen (e.g., contact 1052 is detected on folder icon 1044), the expanded notification list 1050 remains displayed while contact 1052 is maintained on the touch screen. In some embodiments, if a lift-off of contact 1052 is detected while contact 1052 is on folder icon 1044, the display of home screen user interface 1042 and expanded notification list 1052 is replaced or partially replaced by the contents of the folder represented by folder icon 1044. In some embodiments, if the finger hovers over an application launch icon corresponding to an application with unread notifications instead of a folder icon, the lift-off of the finger after finger 734 contacts the application launch icon causes the application to be launched, and causes the display of the application's user interface to replace the display of the home screen user interface.

[0329] FIG. 10R to FIG. 10S It is shown that while finger 734 maintains contact with the touch screen (e.g., contact 1052), finger 734 moves across the touch screen to a new position above the unread notifications of the messaging application; and in response to detecting the lift-off of contact 1052 above the unread notifications of the messaging application, the device replaces the display of home screen user interface 1042 and expanded notification list 1050 with user interface 1062 of the messaging application corresponding to the conversation associated with the missed notification. Although not shown, in some embodiments, the read / unread status of all previously unread notifications shown in notification list 1050 is changed by the device to "read". In some embodiments, the status of unread notifications of other applications (e.g., App Store and Mail) within the folder remains unread.

[0330] Figures 10T to 10Y It is shown that according to some embodiments, a cursor placement mode is activated when a finger 734 hovers over selectable content within a first level hover proximity range, and a selection resize mode is activated when the finger 734 enters a second hover proximity range from outside the second hover proximity range.

[0331] exist Figure 10T , user interface 1072 displays content (eg, selectable text).

[0332] Figure 10U, when the finger 734 moves toward the touch screen and enters the first level hover proximity range, a cursor 1074 is displayed within the displayed text at a location corresponding to the (x, y) position 504 of the finger 734 relative to the touch screen 112 (e.g., as shown by indication 1076 (not visible)). In addition, a magnifying glass (e.g., magnifying glass 1078) is displayed. An enlarged version of a portion of the text near the cursor 1074 is shown in the magnifying glass 1078. An enlarged version 1080 of the cursor 1074 is also shown in the magnifying glass 1078.

[0333] Figure 10V , when the finger 734 moves laterally within the first horizontal hovering proximity range, the cursor 1076 and the magnifying glass 1078 move across the touch screen according to the (x, y) position 504 of the finger 734. In some embodiments (not shown), if the finger 734 is lifted outside the first level hovering proximity range, the cursor 1076 and the magnifying glass 1078 are stopped from being displayed. In some embodiments, when there is already a selection in the content, the cursor placement mode can be activated. In some embodiments, the cursor 1074 remains displayed (e.g., for at least a threshold amount of time), and the magnifying glass is stopped from being displayed when the finger 734 is lifted outside the first level hovering proximity range without first entering the second level hovering proximity range. For example, the user can start typing or pasting other content at the cursor position thereafter. If no user input is detected within a threshold amount of time, the cursor 1074 can be stopped from being displayed.

[0334] Figure 10W As shown, when finger 734 continues to move toward the touch screen and enters the second level hover proximity range, the appearance of cursor 1074 changes to indicate that the selection resize mode is activated. For example, cursor 1074 transforms into two overlapping selection handles 1084 and 1086. The appearance of magnifying glass 1978 is changed (e.g., to magnifying glass 1088) to indicate that the selection resize mode is activated.

[0335] Figure 10X to Figure 10YIt is shown that once the selection resize mode is activated, subsequent lateral movement of finger 723 within the second level hover proximity causes text (e.g., in selection object 1082) to be selected, with one selection handle (e.g., selection handle 1086) remaining in its initial position and another selection handle (e.g., selection handle 1084) moving according to the lateral movement of finger 723. Magnifying glass 1088 moves with the selection handle (e.g., selection handle 1084) repositioned by finger 734. In some embodiments (not shown), after the text selection has been extended, if finger 743 is lifted outside the second hover proximity and then leaves the first hover proximity without first contacting the touch screen, the selection is canceled; and if finger 734 first contacts the touch screen, the selection is maintained after finger 734 is lifted outside the second hover proximity and then leaves the first hover proximity. In some embodiments, the device requires finger 734 to hover within the second level hover proximity for at least a threshold amount of time before the cursor placement mode ends and the selection resize mode is activated. In some embodiments, before the cursor placement mode ends and the selection resize mode is activated, the device requires that the finger 734 hovers over the cursor 1074 for at least a threshold amount of time within the second level hover proximity range. In some embodiments, once the selection object 1082 is extended, the finger 734 can move back to the first level hover proximity range without interfering with the selection, and re-enter the second level hover proximity range to reselect one of the two selection handles (e.g., by hovering near the selection handle, or hovering and remaining above the selection handle for a threshold amount of time), and move the selected selection handle to adjust the size of the selection object. In some embodiments, when the finger 734 moves back to the first level hover proximity range, the magnifying glass 1088 stops being displayed, and when the finger 734 re-enters the second level hover proximity range and selects one of the selection handles, the magnifying glass 1088 is redisplayed near the selected selection handle. In some embodiments, when the finger 734 moves out of the first level hover proximity range again, the selection object 1082 is removed (e.g., deselected) unless the finger 734 first contacts the touch screen. In some embodiments, when finger 734 moves back outside of the first level hover proximity range, selection object 1082 remains displayed, and when finger 734 is then moved back into the first level hover proximity range, cursor placement mode is re-entered in which selection object 1082 remains on the touch screen.

[0336] FIG. 11A to FIG. 11JIt is shown that an input object is hovered over a first object for at least a threshold amount of time to enter a mode for performing a corresponding operation relative to the first object (e.g., an information display mode for displaying a tooltip for an object or a missed notification, such as a control enable indication or an application launch icon), and once in the mode, when the input object is moved laterally over another object (e.g., an object of the same type), the corresponding operation is performed relative to the object without requiring the input object to hover over the object for at least a threshold amount of time (e.g., immediately performing the corresponding operation). In some embodiments, the device requires the input object to meet requirements other than the hover hold time threshold to enter the mode. For example, a predefined air gesture is required to enter the mode, and once the mode is entered, the device does not need to perform an air gesture again for the next object to perform the operation relative to the next object. In some embodiments, the device requires the input object to enter a second-level hover proximity range (e.g., close to the touch screen) to enter the mode, and once the mode is entered, the device only requires the input object to be in a first-level hover proximity range in order to perform the operation associated with the mode.

[0337] exist Fig.11A , home screen user interface 1102 is displayed. Within home screen user interface 1102, a number of application launch icons are displayed, including some application launch icons with unread notifications or messages. For example, there are four unread notifications or messages for the phone application (e.g., as shown by a badge on application launch icon 1106); there are five unread notifications or messages for the mail application (e.g., as shown by a badge on application launch icon 1104), and there are two unread notifications or messages for the instant messaging application.

[0338] exist Fig. 11B , a new message for the mail application has arrived at the device, and a notification is generated (e.g., as shown by notification banner 1108) and displayed above home screen user interface 1102. If no user interaction with notification banner 1108 is detected within a threshold amount of time after the initial display of notification banner 1108, notification banner 1108 ceases to be displayed (e.g., the notification may be stored in a notification center for the user to view later).

[0339] exist FIG. 11C to FIG. 11D In the example, finger 734 has moved toward the touch screen and entered the hovering proximity range above the touch screen (for example, finger 734 at time t 0 1104). Then, finger 734 is held over application launch icon 1104 for the mail application (e.g., FIG. 11C to FIG. 11D, the indication 1110 of the finger 734 is above the badge portion of the application launch icon 1104). When the finger 734 is held above the application launch icon 1104 for less than a threshold amount of time T, no visible change is applied to the application launch icon 1104 (e.g., as shown in FIG. 1 ). Fig. 11C ). When the finger 734 is held over the application launch icon 1104 for at least a threshold amount of time T, the application launch icon 1104 is highlighted and the previously displayed notification banner 1108 is redisplayed on the home screen user interface 1102. In some embodiments (not shown), the notification banner 1108 will remain displayed for a period of time without the finger 734 being continuously held over the application launch icon 1104. If the finger 734 is lifted out of the hovering proximity range and the notification banner 1108 is tapped while the notification banner 1108 is still displayed, the mail application will be launched. If the finger 734 is lifted out of the hovering proximity range and no user interaction with the notification banner 1108 is detected within a threshold amount of time, the notification banner 1108 is stopped from being displayed. As long as the finger 734 is hovering over the application launch icon 1104, the notification banner 1108 will remain displayed. In some embodiments, the portion of the home screen user interface 1102 outside the icon 1104 is blurred and darkened below the notification banner 1108.

[0340] FIG. 11E to FIG. 11F 1108, the device enters an information display mode that displays information for an additional object (e.g., a missed notification) without requiring finger 734 to be held over the additional object for the same amount of time as the threshold amount of time held over application launch icon 1104. In some embodiments, the additional object is the same type as the first object (e.g., application launch icon 1104) that first activated the information display mode, and the information displayed for the additional object is the same type as the information (e.g., notification banner 1108) displayed for the first object (e.g., application launch icon 1104). Fig.11E As shown, while notification banner 1108 remains displayed above home screen user interface 1102, finger 734 begins to move laterally toward application launch icon 1106 while remaining within hover proximity above the touch screen. Fig.11F , once finger 734 moves to a hovering position above application launch icon 1106 (as indicated by the position of finger 734's indicator 1110), application launch icon 1106 becomes highlighted and displays a notification banner 1112 for the phone application. At the same time, application launch icon 1104 stops being highlighted and notification banner 1108 stops being displayed.

[0341] Fig.11G An embodiment is shown in which when finger 734 moves laterally to application launch icon 1106 after entering information display mode, notification banner 1112 for application launch icon 1106 is displayed while notification banner 1108 for application launch icon 1104 remains displayed over home screen user interface 1102. In some embodiments, when finger 734 is lifted into hover proximity range over the touch screen and information display mode is exited, both notification banner 1108 and notification banner 1112 cease to be displayed.

[0342] Fig.11H from Fig.11F or Fig.11G continue. Fig.11H It is shown that when finger 734 is lifted outside of the hovering proximity range above the touch screen, (e.g., after a predefined period of time) notification banner 1112 (and notification banner 1108) cease to be displayed and the information display mode is exited (e.g., immediately when finger 734 leaves the hovering proximity range).

[0343] FIG. 11I to FIG. 11J 1106 (e.g., as indicated by an indicator 1114 on the application launch icon 1106) for at least a threshold amount of time T. Once the finger 734 re-enters the hovering proximity range (e.g., the finger 734 re-enters the hovering proximity range above the application launch icon 1106 at time t3, as indicated by an indicator 1114 on the application launch icon 1106), the notification banner 1112 is displayed again. Fig.11I ) and hovers over the application launch icon 1106 for at least a threshold amount of time, then the information display mode is re-entered and the notification banner 1112 is redisplayed (e.g., at time t 4 , redisplay notification banner 1112).

[0344] FIG. 12A to FIG. 12RThe use of a hover input to distinguish between user interface objects that have a differentiated object-specific response to intensity-based input (e.g., light press, deep press input, e.g., as opposed to non-intensity-based input, such as a tap input, a touch hold input) and user interface objects that do not have a differentiated object-specific response to intensity-based input is shown. An intensity threshold that is higher than a contact detection intensity threshold is used as part of the criteria for detecting and identifying intensity-based input. Non-intensity-based inputs, such as a tap input or a touch hold input (e.g., a long press input) do not require the contact to have an intensity that satisfies another intensity threshold that is higher than the contact detection intensity threshold in order to detect and identify the input. A time threshold between touch press and lift-off is sometimes used as part of the criteria for detecting and identifying non-intensity-based input. An object that has a differentiated response to intensity-based input has a first response to a tap input and a different response to a light press input having an intensity exceeding a light press intensity threshold, and optionally has another different response to a deep press input having an intensity exceeding a deep press intensity threshold that is higher than the light press intensity threshold. For example, an object that does not have a differentiated response to intensity-based input has the same response to a tap input and a light press input.

[0345] exist Fig. 12A , home screen user interface 1202 is displayed. Home screen user interface 1202 includes a number of application launch icons corresponding to different applications installed on the device. Some application launch icons (e.g., application launch icons 1204 and 1212) have corresponding application-specific quick action menus that are activated when pressed with a force exceeding a light press intensity threshold IT. L These quick action menus are displayed when an application launch icon is activated by a touch of a certain intensity. Some other application launch icons (e.g., application launch icons 1210 and 1216) do not have corresponding application-specific quick action menus, and when the application launch icon is touched by an intensity exceeding the light press intensity threshold IT L When the touch is activated, no such menu or other associated application-specific user interface objects are displayed. A tap input on any application launch icon will cause the device to launch the application corresponding to the tapped application launch icon.

[0346] FIG. 12B to FIG. 12C It is shown that when finger 734 enters the hover proximity range above the touch screen and finger 734 hovers over application launch icon 1204 (as shown by indication 1208 of finger 734), preview disk 1206 merges from the following application launch icon 1204 to indicate to the user that application launch icon 1204 has a differentiated object-specific / application-specific response to intensity-based input. FIG. 12B to FIG. 12C, when finger 734 hovers over application launch icon 1204, portions of home screen user interface 1202 outside of application launch icon 1204 are dimmed and blurred. In some embodiments, the amount of dimming and blurring applied to the appearance of home screen user interface 1202 and preview disk 1206 changes dynamically based on the current value of a hover proximity parameter (e.g., hover distance 514) of finger 734. For example, as finger 734 hovers near the touch screen, preview disk 1206 increases in size and home screen user interface 1202 becomes increasingly dim and blurred. If finger 734 is lifted outside of the hover proximity range or moved over another application launch icon that does not have a differentiated object-specific response to intensity-based input, the visual effect (e.g., dimming and blurring of preview disk and home screen user interface) is removed.

[0347] FIG. 12D to FIG. 12E 1202. It is shown that when the finger 734 moves laterally to another hover position over the application launch icon 1210 that does not differentiate object-specific responses to intensity-based input, the appearance of the application launch icon 1210 is changed to indicate that the finger 734 is now hovering over the application launch icon 1210 (as shown by the indication 1208 of the finger 734 and the highlighting of the application launch icon 1210), but no preview disk is merged from behind the application launch icon 1210, and the dimming and blurring of the home screen user interface 1202 is not applied. In some embodiments, when the current value of the hover proximity parameter (e.g., the hover distance 514) of the finger 734 changes (e.g., the application launch icon 1210 is in Fig.12D and Fig.12E ), the highlighting of the application launch icon 1210 remains unchanged.

[0348] Fig.12F 1204. The home screen user interface 1202 is similar to the home screen user interface 1202. The home screen user interface 1202 is similar to the home screen user interface 1204. The home screen user interface 1202 is similar to the home screen user interface 1204. The home screen user interface 1202 is similar to the home screen user interface 1204. The home screen user interface 1202 is similar to the home screen user interface 1204. The home screen user interface 1202 is similar to the home screen user interface 1204. Fig. 12B Darken and blur in the same way as shown in .

[0349] Figure 12G1208, the application launch icon 1216 is highlighted. The highlighting applied to the application launch icon 1216 is different from the highlighting applied to the application launch icon 1216. Fig.12D The highlighting of the application launch icon 1210 in is consistent, similar, and optionally identical.

[0350] FIG. 12H to FIG. 12I from Fig.12E After finger 734 hovers over application launch icon 1210, finger 734 makes contact with application launch icon 1210 (e.g., as shown by contact 1220). When contact intensity 1220 increases and exceeds light press intensity threshold IT L When the light press input is detected (as shown by the intensity meter 1201), a preset visual effect is shown to indicate that a light press input has been detected, but the application launch icon 1210 has no object-specific response to the light press input. Fig.12H As shown, when a light press input is detected, a hint 1222 of the preview disk appears from behind the application launch icon 1210 and a tactile output 1224 (e.g., a "failure" tactile sensation) is generated. During the preset visual effect, the hint 1222 of the preview disk quickly fades out. Fig.12I The preset visual effect is shown to be completed, and finger 734 is lifted off the touch screen. When finger 734 hovers over application launch icon 1210 (as shown by indicator 1226), the same highlight is applied to application launch icon 1210 to indicate that it has no differentiated object-specific response to intensity-based input and only has a preset generic response (which has been set in the example above). Fig.12H ).

[0351] exist FIG. 12J to FIG. 12L , finger 734 is moved horizontally again to a hovering position above application launch icon 1204. Redisplay Fig. 12C The same visual effect as shown in Figure 12K , finger 734 makes contact with application launch icon 1204 (e.g., as shown by contact 1226). As contact intensity 1226 increases, preview disk 1206 increases in size and moves relative to application launch icon 1204, and user interface 1202 further darkens and blurs (e.g., as shown by contact 1226). Figure 12K Once contact intensity 1226 exceeds the light press intensity threshold, preview panel 1206 transforms into a mini-app object 1228 and a quick action menu 1230 corresponding to the messaging app (as shown in FIG. Figure 12L1202 ). The amount of blurring and dimming applied to user interface 1202 is at its maximum extent. When a light press input is detected and the quick action menu and mini-app objects are displayed, a tactile output (e.g., tactile output 1232) is generated.

[0352] FIG. 12M to FIG. 12O It is shown that when finger 734 is lifted from the touch screen after mini-app object 1228 and quick action menu 1230 are displayed in response to a light press input made by contact 1226, the appearance of mini-app object 1228 and quick action menu 1230 changes according to the current value of the hover proximity parameter of finger 734, for example, becoming increasingly translucent as the hover distance 514 above the touch screen increases. In addition, the amount of blurring and dimming of home screen user interface 1202 also decreases as the hover distance 514 above the touch screen increases. FIG. 12M to FIG. 12O It is also shown that after finger 734 is lifted from the touch screen, finger 734 moves laterally while remaining within the hovering proximity range above the touch screen (as shown by indication 1238), and the mini-app object 1228 and the quick action menu 1230 remain displayed as long as finger 734 is still detected within the hovering proximity range above the touch screen.

[0353] Figure 12P from Fig.12O continue. Figure 12P As shown, if finger 734 is lifted outside of the hover proximity range without first making contact with the touch screen again, mini-app objects 1228 and quick action menu 1230 cease to be displayed and home screen user interface 1202 resumes its original appearance.

[0354] Figures 12Q to 12R from Fig.12O Continue. Figure 12P , finger 734 contacts the touch screen at a position corresponding to one of the selectable options in quick action menu 1230 (as indicated by contact 1234). When contact 1234 is lifted off, the selectable option touched by contact 1234 is activated, and the instant messaging application corresponding to application launch icon 1204 is launched, and a user interface 1236 corresponding to the selectable option is displayed (as indicated by contact 1234). Figure 12R shown).

[0355] FIG. 13A to FIG. 13N Shown is a visual indication of which of multiple selection handles of a content selection object is selected for movement when an input object hovers over the selection handle according to some embodiments.

[0356] exist Fig.13AIn the example embodiment, a content display user interface 1302 (e.g., a user interface of a text editor application or a user interface of a web browser application, etc.) displays selectable content (e.g., text). A portion of the displayed content has been selected. The selection is contained in a selection object (e.g., a selection object 1304 having two selection handles, including a start selection handle 1306 and an end selection handle 1308). Optionally, the selection handle 1308 is displayed relative to the content of the video. Figures 10T to 10Y or other conventional text selection methods to perform text selection.

[0357] FIG. 13B to FIG. 13C It is shown that when finger 734 moves toward the touch screen and enters the hover proximity range, and if the (x, y) position 504 of finger 734 is above one of the selection handles (e.g., selection handle 1308) (e.g., as shown by indication 1314 of finger 734) for at least a threshold amount of time T, the selection handle (e.g., selection handle 1308) is highlighted relative to another selection handle (e.g., selection handle 1306). For example, the end of selection handle 1308 is enlarged relative to the corresponding end of selection handle 1306 to indicate that selection handle 1308 has been selected for movement. In some embodiments, magnifying glass 1310 is displayed near the selected selection handle 1308. In some embodiments, magnifying glass 1310 is displayed only after finger 734 hovers above selection handle 1308 for at least a threshold amount of time T. The magnifying glass includes an enlarged version of a portion of the text surrounding the selection handle 1308. The selection handle 1308 is represented as a cursor 1312 within the enlarged text in the magnifying glass 1310.

[0358] FIG. 13D to FIG. 13G Repositioning the selected selection handle 1308 by movement of the continuously maintained contact is shown. In some embodiments, Fig.13D from Fig. 13C Continue. In some embodiments, Fig.13D from Fig. 13B When finger 734 moves closer to the touch screen and contacts the touch screen at a location corresponding to a selected selection handle 1308 (e.g., as indicated by intensity meter 1301) (e.g., as indicated by contact 1316), the appearance of selection handle 1308 optionally changes to indicate that contact has been made with selection handle 1308 (e.g., as indicated by intensity meter 1301). Fig.13D As contact 1316 moves across the touch screen, contact 1316 drags selection handle 1308 to expand the selection contained in selection object 1304. Magnifying glass 1310 moves with selection handle 1308, and the text shown in magnifying glass 1310 is continuously updated according to the position of selection handle 1308 within the text (as shown in FIG. 13A ). Fig.13E ). Fig.13F, finger 734 is lifted off the touch screen, and selection handle 1308 is selected while finger 734 continues to hover over selection handle 1308 (e.g., in a manner similar to Fig. 12C 1306 remains highlighted). If finger 734 is moved laterally to another hover position (not shown) above selection handle 1306, selection handle 1308 will return to its original appearance and selection handle 1306 will be highlighted relative to selection handle 1308 to indicate that selection handle 1306 is now selected for movement. Additionally, magnifying glass 1310 will be moved (not shown) to be proximate to selection handle 1306 and display text surrounding selection handle 1306. Figure 13G , once finger 734 is lifted out of the hover proximity range above the touch screen, the previously highlighted selection handle (e.g., selection handle 1308) restores its original appearance but remains in the new position, and the resizing of the selection object is completed.

[0359] FIG. 13H to FIG. 13J 13 shows a situation where finger 734 is not held over the selection handle for more than a threshold amount of time T before finger 734 makes contact with the touch screen. As the contact moves across the touch screen, the scrolling content display user interface 1302. Fig.13H 13, finger 734 is in a hover position (as indicated by indication 1318 of finger 734) above selection handle 1308 (e.g., when finger 734 is on its way to contact with the touch screen). The appearance of the selection object does not change until finger 734 hovers over selection handle 1308 for at least a threshold amount of time T. After finger 734 makes contact with the touch screen (e.g., as indicated by contact 1320), the entire content display user interface 1302 scrolls as contact 1320 moves across the touch screen (e.g., as indicated by contact 1320). FIG. 13I to FIG. 13J ). Although content selection object 1304 is displayed on the content display user interface, scrolling of user interface 1302 by contact 1320 does not require the presence of the content selection object. Additionally, scrolling of the user interface does not require the contact to be located on or near the selection handle, and can be performed by contact on any portion of the displayed content.

[0360] Figure 13K from Fig.13D continue. Figure 13K Selection handle 1308 is shown moving in accordance with lateral movement of finger 734 while finger 734 remains within the hover proximity above the touch screen.

[0361] Figure 13L exist Figure 13K Thereafter, and as shown, if finger 734 is lifted outside of the hover proximity range without first making contact with the touch screen again, selection handle 1308 is restored to its original position and the resizing of selection object 1304 is canceled.

[0362] FIG. 13M to FIG. 13N exist Figure 13K Thereafter, and showing that if finger 734 is lifted outside of the hover proximity range after making contact with the touch screen (e.g., as shown by contact 1322), selection handle 1308 remains in the new position in the text. Figure 8P and Figure 8V Describes additional details of this behavior.

[0363] FIG. 14A to FIG. 14M Operation of a reaction zone for maintaining input focus on an object, wherein the reaction zone has different lateral extents at different hovering distances above a touch screen, is illustrated according to some embodiments.

[0364] Fig.14A A user interface (e.g., map user interface 1402) is shown that includes multiple user interface objects (e.g., location pin 1404 and location pin 1406) that are selectable when finger 734 hovers over these user interface objects. The (x, y) position 1408 of location pin 1404 is indicated on touch screen 112.

[0365] FIG. 14B to FIG. 14C 7 shows that when the finger 734 moves toward the touch screen and enters the hover proximity range over an area that does not have any selectable objects (e.g., Fig. 14C ), an empty information pop-up window (e.g., pop-up window 1412) is displayed at a position corresponding to the (x, y) position 504 of the finger 734.

[0366] Fig.14D It is shown that when the finger 734 moves to a hover position directly above the pin 1404 (e.g., the (x, y) position 504 of the finger 734 is within a narrow lateral range around the position 1408), the input focus is moved to the location pin 1404 and an information pop-up window 1414 corresponding to the location pin 1404 is displayed.

[0367] FIG. 14E to FIG. 14F A reaction area for positioning pin 1404 is shown, which is wider as it is closer to the touch screen and narrower as it is farther away from the touch screen (e.g., as shown by boundary line 1418 of the reaction area, where the reaction area is located within a cone defined by boundary line 1418 (only half of the cone is shown in the figure)). Fig.14EIt is shown that once the input focus is shifted to the positioning pin 1404 (after the finger 734 has met the strict requirements for moving the input focus onto the positioning pin 1404), the input focus will not deviate from the positioning pin 1404 when the finger 734 is no longer directly above the positioning pin 1404 (for example, the (x, y) position 504 of the finger 734 is outside the narrow lateral range around the position 1408). FIG. 14E to FIG. 14F 1404, as long as the finger 734 remains within the reaction area defined by the boundary line 1418, the input focus will remain on the positioning pin 1404, and the information pop-up window 1414 will remain displayed. Note that although the boundary line 1418 is shown as straight, it does not need to be straight in some embodiments. In some embodiments, the lateral range of the reaction area is calculated on the fly based on various factors, including the hovering proximity parameters of the finger 734 and optionally the speed and direction of the finger 734 at this time, so that the reaction area is not a predefined area with a fixed shape, but is dynamically determined during the hovering movement of the finger 734. In addition, in some embodiments, the lateral range of the reaction area may increase with the increase of the hovering distance for the first range of hovering distances, and may decrease with the increase of the hovering distance for the second range of hovering distances (e.g., higher than the first range or lower than the first range).

[0368] Figure 14G Finger 734 is shown moving outside of the reaction area defined by border 1418 , input focus is taken away from positioning pin 1404 , and information pop-up window 1414 is replaced by an empty information pop-up window 1410 .

[0369] FIG. 14H to FIG. 14I It is shown that when the finger 734 is outside the reaction area of ​​the positioning pin 1404, and then hovers directly over the positioning pin 1404 again, the input focus is placed on the positioning pin 1404 and the information pop-up window 1414 is displayed again.

[0370] Fig.14J It shows that when the finger 734 is (with Fig.14F When the finger 734 moves laterally from position 1408 at a higher hovering distance than the case shown in FIG. 1404 , the finger 734 moves laterally from position 1408 at a higher hovering distance than the case shown in FIG. FIG. 14F to FIG. 14G , input focus is taken away from the locate pin 1404 because the threshold lateral range for moving input focus away from the locate pin 1404 is smaller at higher hovering distances above the touch screen.

[0371] FIG14 shows that when finger 734 hovers over positioning pin 1404 again, input focus is placed on positioning pin 1404 and information pop-up window 1414 is displayed again. If finger 734 moves toward the touch screen at a speed greater than the threshold speed, input focus remains on positioning pin 1404 for at least the threshold amount of time, even if finger 734 is outside the reaction area of ​​positioning pin 1404 during its travel to the touch screen. For example, if Figure 14L As shown, when finger 734 is in contact with the touch screen for a threshold amount of time (e.g., as shown by contact 1420), input focus remains on location pin 1404 even if finger 734 has been in contact with the touch screen outside of the reaction area of ​​location pin 734. Figure 14M It is shown that when lift-off of contact 1420 is detected, additional information related to positioning pin 1404 is displayed, for example, in information panel 1422.

[0372] FIG. 15A to FIG. 15C 1 is a flowchart illustrating a method 1500 for interacting with a user interface object through proximity-based and contact-based input according to some embodiments. The method 1500 is performed on an electronic device having a display and a touch-sensitive surface (e.g., device 300, Figure 3 ; or a portable multifunction device 100, Figure 1A ). In some embodiments, the electronic device includes one or more first sensors to detect the proximity of an input object (e.g., a finger or stylus) above the touch-sensitive surface (e.g., a proximity sensor (such as an infrared sensor), a capacitive sensor in the touch-sensitive surface, or a camera next to the touch-sensitive surface), and one or more second sensors to detect the intensity of contact of the input object with the touch-sensitive surface. In some embodiments, the one or more second sensors are different from the one or more first sensors. In some embodiments, the one or more second sensors are the same as the one or more first sensors. In some embodiments, the touch-sensitive surface and the display are integrated into a touch-sensitive display. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 1500 are optionally combined, and / or the order of some operations is optionally changed.

[0373] As described below, method 1500 provides an intuitive way to interact with user interface objects through proximity-based and contact-based inputs. In particular, by dynamically changing the visual characteristics of a selected user interface object according to the current hover proximity parameters of the input object when the input object drags the user interface object, the device provides visual feedback to the user about the user input, so that the user can adjust the input (e.g., the hover proximity parameters) in a timely manner to avoid input errors (e.g., accidentally dropping the object by moving too close or too far from the touch-sensitive surface) and (e.g., by placing the object in a desired location in a timely manner or continuing to move the object to a predetermined destination) to more effectively complete the intended operation. The method reduces the amount, degree and / or nature of input from the user when interacting with the user interface object, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, enabling users to reposition objects more quickly and efficiently saves power and increases the data interval between battery charges.

[0374] The device displays (1502) a user interface object (e.g., user interface object 704) at a first position on a display (e.g., Fig. 7A shown).

[0375] The device detects ( 1504 ) selection of a user interface object at a first location on the display (eg, selection of the user interface object in response to a hover input, a tap input, or a press input directed to the user interface object).

[0376] While the user interface object is selected (1506), the device detects lateral movement of the input object (e.g., a finger or stylus) over the touch-sensitive surface when the input object satisfies a hover criterion (e.g., the hover criterion requires that a hover proximity parameter (e.g., a parameter based on a hover distance) associated with the input object be less than a threshold proximity value (e.g., a threshold hover distance) in order to satisfy the hover criterion); and in response to detecting lateral movement of the input object over the touch-sensitive surface when the input object satisfies the hover criterion, the device moves the selected user interface object in accordance with the lateral movement of the input object over the touch-sensitive surface (e.g., from a first position on the display to a second position); and while moving the selected user interface object in accordance with the lateral movement of the input object over the touch-sensitive surface, the device dynamically changes a first visual characteristic (e.g., size, color saturation, transparency, perceived depth relative to the display surface, etc.) of the selected user interface object in accordance with the current hover proximity parameter of the input object. For example, as Figure 7L to Figure 7R As shown, after object 704 is lifted up with stylus 203 , the size of object 704 changes dynamically according to the hovering distance 514 of stylus 203 during lateral movement of stylus 203 while stylus 203 remains within a threshold hovering distance above touch-sensitive surface 651 .

[0377] In some embodiments, the hover criteria include (1508) a criterion that is met when a hover proximity parameter of the input object is less than a threshold proximity value. For example, the detected distance of the fingertip above the touch-sensitive surface is less than a threshold distance. As used in the specification and claims, the term "hover proximity parameter" refers to the (non-zero) distance of the input object above the touch-sensitive surface or a substitute (proxy) for the distance of the input object above the touch-sensitive surface. In some embodiments, the change in capacitance detected on the touch-sensitive surface due to the change in the distance of the input object above the touch-sensitive surface is a proxy for the distance of the input object above the touch-sensitive surface. In some embodiments, the hover criteria include a criterion that the input object is not touching the touch-sensitive surface.

[0378] In some embodiments, when an input object is detected, the device displays (1510) a position indicator (e.g., an image of a cursor, fingertip, or hand, an image representing a shadow cast by the input object, an icon representing a focus selector, an icon representing a selection tool, etc.) (e.g., indicator 706) at a position on the display corresponding to the current position of the input object on the touch-sensitive surface (e.g., the (x, y) position 504 of the stylus 203). In some embodiments, the position on the display corresponding to the current position of the input object is located directly below the input object. In some embodiments, the position on the display corresponding to the current position of the input object is not necessarily located directly below the input object, and may be laterally offset from the position directly below the input object (e.g., position 701 deviates from the (x, y) position 504 of the stylus 203). For example, the position on the display corresponding to the current position of the input object may be a projection of the tip of the input object on the display surface based on the current azimuth and tilt angle of the input object. In another example, a finger hovering over the touch screen causes the position indicator (also serving as a focus selector) to be displayed at a position on the touch screen that is offset a certain distance upward so that the finger does not obstruct the user's view of the focus selector. By offsetting the position of the position indicator from the (x,y) position of the fingertip, the user is able to better control the movement and position of the position indicator and manipulate the object with better precision and fewer errors.

[0379] In some embodiments, the device dynamically changes (e.g., increases or decreases) the size or transparency of the position indicator based on the current hover proximity parameters of the input object (e.g., when the (non-zero) hover distance of the input object detected above the touch-sensitive surface changes (e.g., increases or decreases, or vice versa)) (dynamically) changes (1512) visual characteristics of the position indicator (e.g., size, color saturation, transparency, and / or 3D visual effects, such as shadow shape and / or length). For example, Figure 7L to Figure 7RAs shown, the device dynamically changes the size of the indicator 706 according to the hovering distance 514 during the movement of the stylus 203. In some embodiments, when the input object is completely within the hovering distance from the touch-sensitive surface, the position indicator has a "normal" appearance (e.g., a neutral color or normal opacity); and when the input object is very close to the touch-sensitive surface and is about to contact the touch-sensitive surface, the position indicator presents a first "warning" appearance (e.g., an orange alert color and / or completely opaque); and when the input object is very close to the proximity threshold and is about to terminate the hover interaction, the position indicator presents a second "warning" appearance (e.g., a red alert color and / or nearly completely transparent). This is Figure 7Q In some embodiments, by (dynamically) changing the visual characteristics of the position indicator according to the current hover proximity parameters of the input object (e.g., before the user interface object is selected, when the user interface object is dragged by the input object, and / or after the object is dropped), the device provides the user with visual feedback about the current state of the user input (e.g., the function currently being executed or the function that can be activated by the current input), so that the user can adjust the input in time (e.g., adjust the hover proximity parameters) to avoid input errors (e.g., accidentally picking up or dropping an object by moving too close or too far from the touch-sensitive surface) and complete the intended operation more efficiently (e.g., by picking up the desired object in time, or dropping the object at the desired location, or continuing to move the object to the intended destination).

[0380] In some embodiments, detecting selection of a user interface object includes (1514) detecting the input object making and breaking contact with the touch-sensitive surface at a location corresponding to a first location of the user interface object on the display (e.g., detecting a tap gesture or a long press gesture by the input object at a location corresponding to the user interface object). In some embodiments, the user interface object is selected when the input object is lifted off; and when the user interface object is selected, the position indicator of the input object also changes appearance (e.g., the hand indicator changes from an open palm to a clenched fist). For example, this Figures 7J to 7M Shown in.

[0381] In some embodiments, detecting selection of a user interface object includes (1516) detecting a press input of an input object at a position corresponding to a first position of the user interface object on the display, wherein detecting the press input includes detecting that a characteristic intensity of contact of the input object with the touch-sensitive surface is greater than a first threshold intensity (e.g., a light press or deep press intensity threshold). In some embodiments, the user interface object is selected when a lift-off is detected immediately after the input object satisfies the first threshold intensity or when the input object satisfies the first threshold intensity; and when the user interface object is selected, the position indicator of the input object also changes appearance (e.g., a hand indicator changes from an open palm to a clenched fist). In some embodiments, when a press input is required to select an object, an operation associated with the object is optionally triggered by an input of a contact that does not meet the first intensity threshold. For example, when the object is a weather item, a tap input triggers display of a user interface including more information about the weather item, and a press input that meets the first threshold intensity selects the weather item. In another example, in the case where the object is a calendar item, a tap input triggers display of a user interface including more information about the calendar, and a press input that meets the first threshold intensity selects the calendar item. By requiring the characteristic intensity of contact to increase above a first threshold intensity in order to select a user interface object, the device reduces user errors (e.g., accidental selection of an object while hovering over a touch-sensitive surface can be avoided) and improves the operability of the device (e.g., precise control of input objects over the touch-sensitive surface is not required).

[0382] In some embodiments, detecting selection of the user interface object includes (1518) detecting a long press input of the input object at a location corresponding to the first location of the user interface object on the display, wherein detecting the long press input includes detecting that a characteristic intensity of contact of the input object with the touch-sensitive surface remains above a first threshold intensity for at least a threshold amount of time. Figures 7K to 7L . In some embodiments, when a long press input is required to select an object, an input through a contact that does not meet a first intensity threshold or a time threshold optionally triggers an operation associated with the object or triggers the display of additional information related to the object. For example, when the object is an application launch icon, a tap input triggers launching an application associated with the application launch icon, a press input causes a quick action menu with selectable options to activate a corresponding operation associated with the application, and a long press input causes selection of the application launch icon. By requiring the characteristic intensity of the contact to increase to above a first threshold intensity and be maintained for at least a threshold amount of time in order to select a user interface object, the device reduces user errors (e.g., accidentally selecting an object when hovering over a touch-sensitive surface can be avoided) and improves the operability of the device (e.g., precise control of the input object over the touch-sensitive surface is not mandatory).

[0383] In some embodiments, before detecting selection of a user interface object at a first location (e.g., selecting a user interface object as a target of a “hover drag” gesture before a tap gesture, a long press input, or a press input with respect to the input object) and when the input object satisfies a hover criterion (1520): the device detects an input object at a location on the touch-sensitive surface that corresponds to the first location of the user interface object on the display; and when the input object is detected at a location on the touch-sensitive surface that corresponds to the first location of the user interface object on the display, the device (dynamically) changes the appearance of the user interface object at the first location based on a current hover proximity parameter of the input object (e.g., dynamically changing the size and / or color of unselected user interface objects as the (non-zero) hover distance of the detected input object above the touch-sensitive surface changes) (e.g., dynamically changing the size and / or color of unselected user interface objects as the (non-zero) hover distance of the detected input object above the touch-sensitive surface changes) (e.g., FIG. 7C to FIG. 7E while (dynamically) changing the appearance of the user interface object at the first location, the device detects lateral movement of the input object over the touch-sensitive surface (e.g., lateral movement over an unselected user interface object beyond a proximate area (e.g., a threshold lateral range)); and, in response to detecting lateral movement of the input object over the touch-sensitive surface: the device ceases (dynamically) changing the appearance of the user interface object at the first location; and, the device maintains the user interface object at the first location (e.g., as Fig.7I In some embodiments, when an input object changes its lateral position while hovering over an unselected user interface object, the image of the unselected user interface object tilts (e.g., dynamically stretches while remaining anchored in its original position) with small lateral movements of the input object (e.g., as shown in FIG. 1 ). FIG. 7F to FIG. 7I In some embodiments, by providing visual feedback to the user about the interaction with the object (e.g., lateral movement of the input object close to the object) before the object is selected (e.g., before the input object meets the selection criteria), the device helps the user to adjust the input (e.g., hover proximity parameters) in a timely manner to avoid input errors (e.g., accidentally selecting an object by moving too close to the touch-sensitive surface, or accidentally selecting the wrong object when moving too close to the wrong object among several closely positioned objects) and to complete the intended operation more efficiently (e.g., by adjusting the movement direction toward the desired target and adjusting the hover proximity parameters at the desired location).

[0384] In some embodiments, in response to detecting a selection of a user interface object at a first location on the display, the device provides (1522) visual feedback to indicate that the user interface object is selected (e.g., displaying an animation showing the user interface object being "picked up" by a hovering input object, showing the user interface object flashing to indicate the activation state of the user interface object, or casting a shadow under the selected user interface object to indicate that the user interface object is now hovering above its original plane) (e.g., a position indicator of the input object changes its appearance to indicate that the object is selected (e.g., the position indicator changes its appearance from an open palm to a clenched fist)). This Figure 7L Shown in.

[0385] In some embodiments, dynamically changing a first visual characteristic of a user interface object based on a current hover proximity parameter of an input object includes (1524) dynamically changing a size of the user interface object based on the current hover proximity parameter of the input object (e.g., dynamically changing a size of a selected user interface object as a (non-zero) hover distance of the input object detected above the touch-sensitive surface changes). In some embodiments, the size of the user interface object increases as the distance between the input object and the display (or touch-sensitive surface) increases (e.g., the user interface object is lifted with a finger or stylus). For example, this Figure 7L to Figure 7R In some embodiments, when the distance between the input object and the display increases (e.g., the user interface object gradually moves away from the finger), the size of the user interface object decreases. In some embodiments, by (dynamically) changing the size of the user interface object according to the current hover proximity parameter of the input object when the input object drags the user interface object, the device provides the user with visual feedback about the current state of the user input, so that the user can adjust the input in time (e.g., adjust the hover proximity parameter) to avoid input errors (e.g., accidentally dropping the object by moving too close to the touch-sensitive surface) and complete the intended operation more efficiently (e.g., by placing the object in the desired location in a timely manner or continuing to move the object to a predetermined destination).

[0386] It should be understood that Figures 15A to 15C The specific order in which the operations have been described in the description is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 1600, 1700, 1800, 1900, 2000, 2100, 2200, 2300, and 2400) are also applicable in a similar manner to the details of other processes described above with respect to FIG. 15A to FIG. 15CMethod 1500 as described. For example, the contacts, gestures, input objects, user interface objects, tactile outputs, audio outputs, intensity thresholds, position indicators, focus selectors, and animations described above with reference to method 1500 optionally have one or more of the features of the contacts, gestures, input objects, user interface objects, tactile outputs, audio outputs, intensity thresholds, position indicators, focus selectors, and animations described herein with reference to other methods described herein (e.g., methods 1600, 1700, 1800, 1900, 2000, 2100, 2200, 2300, and 2400). For the sake of brevity, these details are not repeated here.

[0387] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (for example, as described above with respect to Figure 1A and Figure 3 described) or application-specific chips.

[0388] References FIG. 15A to FIG. 15C The operation is optionally performed by Figure 1A to Figure 1B 1504, 1506, and 1520 are implemented by the components depicted in . For example, the detection operations and feedback operations 1504, 1506, and 1520 are optionally implemented by the event classifier 170, the event identifier 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects contact on the touch-sensitive display 112, and the event distributor module 174 transmits the event information to the application 136-1. The corresponding event identifier 180 of the application 136-1 compares the event information with the corresponding event definition 186, and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know that based on Figure 1A to Figure 1B How the components depicted in the figure can implement other processes.

[0389] FIG. 16A to FIG. 16E 1 is a flowchart illustrating a method 1600 for interacting with a user interface object through proximity-based and contact-based input according to some embodiments. The method 1600 is performed on an electronic device (e.g., device 300, Figure 3 ; or a portable multifunction device 100, Figure 1A ). In some embodiments, the device includes one or more first sensors to detect the proximity of an input object above the touch-sensitive surface (e.g., a proximity sensor (such as an infrared sensor), a capacitive sensor in the touch-sensitive surface, or a camera next to the touch-sensitive surface), and one or more second sensors to detect the intensity of contact of the input object with the touch-sensitive surface. In some embodiments, the one or more second sensors are different from the one or more first sensors. In some embodiments, the one or more second sensors are the same as the one or more first sensors. In some embodiments, the touch-sensitive surface and the display are integrated into a touch-sensitive display. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 1600 are optionally combined, and / or the order of some operations is optionally changed.

[0390] As described below, method 1600 provides an intuitive way to interact with user interface objects through proximity-based input and contact-based input. In some embodiments, by providing distinguishable visual feedback about whether the object responds to the hovering portion or the pressed portion of the input, the device provides a more responsive user interface and allows the user to adjust his / her input in a timely manner to achieve the expected task (e.g., to continue to press down toward the touch-sensitive surface to contact with the object) and / or avoid input errors (e.g., to stop pressing harder to avoid reaching the intensity threshold that triggers unexpected actions). A more responsive user interface also helps to avoid causing user confusion during input, because each part of the input has a direct and immediate impact on the visual appearance of the user interface. The method reduces the number, degree and / or nature of the input from the user when interacting with the user interface object, thereby creating a more efficient human-computer interface. For battery-driven electronic devices, enabling users to interact with the user interface faster and more effectively (e.g., by reducing user errors with improved visual feedback) saves power and increases the time between battery charges.

[0391] The device displays (1602) a user interface object (e.g., user interface object 704) at a first location on a display, e.g., as Figure 7U shown.

[0392] While displaying a user interface object at a first position on the display, the device (1604) detects a first portion of input made by an input object (e.g., a fingertip or stylus), wherein detecting the first portion of the input includes detecting the input object at a location above the touch-sensitive surface that corresponds to the first location of the user interface object on the display when the input object satisfies a hover criterion. For example, while displaying the user interface object, the device detects a finger approaching the user interface object from above the touch-sensitive surface without contacting the touch-sensitive surface while a hover proximity parameter of the input object is less than a threshold distance above the touch-sensitive surface (e.g., when the input object remains within a threshold distance above the touch-sensitive surface without contacting the touch-sensitive surface). For example, this Figure 7V Shown in.

[0393] In response to detecting the first portion of the input, the device dynamically changes (1606) the appearance of the user interface object in a first manner based on the current hover proximity parameters of the input object (e.g., the appearance of the user interface object dynamically changes as the hover distance of the input object changes). For example, in some embodiments, when the hover criteria are met and before the input object contacts the touch-sensitive surface, the size of the user interface object increases as the input object approaches the touch-sensitive surface, and the size of the user interface object decreases as the input object moves away from the touch-sensitive surface. For example, this Figure 7V to 7X In another example, in some embodiments, when the hover criteria are met and before the input object contacts the touch-sensitive surface, the color of the user interface object darkens as the input object approaches the touch-sensitive surface, and the color of the user interface object lightens as the input object retreats from the touch-sensitive surface.

[0394] After dynamically changing the appearance of the user interface object in a first manner based on the current hover proximity parameter of the input object, the device detects (1608) a second portion of the input made by the input object, wherein detecting the second portion of the input includes detecting contact of the input object with the touch-sensitive surface at an initial contact location that corresponds to the first location on the display. Figure 7Y Shown in.

[0395] In response to detecting the second portion of the input, the device dynamically changes (1610) the appearance of the user interface object in a second manner based on the current intensity of contact made by the input object on the touch-sensitive surface (e.g., when the input object contacts the touch-sensitive surface, the appearance of the user interface object dynamically changes based on the changing force or pressure of the input object on the touch-sensitive surface). In some embodiments, the second manner of changing the appearance involves changing the same attribute of the user interface object as the attribute changed in the first manner (e.g., size or color), but based on a change in an input parameter or metric that is different from the input parameter or metric used in the first manner (e.g., the value of the attribute (e.g., size) increases as a hover proximity parameter in the first portion of the input increases, and the value of the attribute (e.g., size) increases as the contact intensity in the second portion of the input increases). In some embodiments, the first manner and the second manner of changing the appearance involve changing two different attributes of the user interface object (e.g., changing the size with the hover proximity parameter in the first portion of the input, and changing the color with the contact intensity in the second portion of the input). Figure 7Z to Figure 7AE , the device dynamically changes the size of object 704 as the intensity of contact between stylus 203 and touch-sensitive surface 651 changes.

[0396] In some embodiments, the hover criteria requires (1612) that a hover proximity parameter of an input object is less than a threshold proximity value in order to meet the hover criteria. For example, the detected distance of a fingertip above the touch-sensitive surface is less than a threshold distance. As used in the specification and claims, the term "hover proximity parameter" refers to the (non-zero) distance of an input object above the touch-sensitive surface or a substitute (proxy) for the distance of an input object above the touch-sensitive surface. In some embodiments, the change in capacitance detected on the touch-sensitive surface due to a change in the distance of the input object above the touch-sensitive surface is a proxy for the distance of the input object above the touch-sensitive surface. In some embodiments, the hover criteria includes a criterion that the input object is not touching the touch-sensitive surface.

[0397] In some embodiments, when an input object is detected, the device displays (1614) a position indicator (e.g., an image of a cursor, a fingertip, or a hand, an image representing a shadow cast by the input object, an icon representing a focus selector, an icon representing a selection tool, etc.) at a location on the display that corresponds to the current location of the input object above or on the touch-sensitive surface (e.g., Figure 7VIndicator 706 in ). In some embodiments, the position on the display corresponding to the current position of the input object is located directly below the input object. In some embodiments, the position on the display corresponding to the current position of the input object is not necessarily located directly below the input object, and may be laterally offset from the position directly below the input object. For example, the position on the display corresponding to the current position of the input object may be a projection of the tip of the input object on the display surface according to the current azimuth and tilt angle of the input object.

[0398] In some embodiments, the device dynamically changes (e.g., increases or decreases) the size or transparency of the position indicator based on the current hover proximity parameters of the input object (e.g., when the (non-zero) hover distance of the input object detected above the touch-sensitive surface changes (e.g., increases or decreases, or vice versa)) (dynamically) changes (1616) visual characteristics of the position indicator (e.g., size, color saturation, transparency, and / or 3D visual effects, such as shadow shape and / or length). For example, this Figure 7V to Figure 7X In some embodiments, when the input object is completely within the hovering distance from the touch-sensitive surface, the position indicator has a "normal" appearance (e.g., a neutral color or normal opacity); and when the input object is very close to the touch-sensitive surface and is about to contact the touch-sensitive surface, the position indicator presents a first "warning" appearance, for example, an orange alert color and / or full opacity; and when the input object is very close to the proximity threshold and is about to terminate the hover interaction, the position indicator presents a second "warning" appearance, for example, a red alert color and / or close to full transparency (e.g., red alert color and / or close to full transparency). Figure 7AI shown).

[0399] In some embodiments, the first manner for dynamically changing the appearance of the user interface object is different from (1618) the second manner for dynamically changing the appearance. For example, before the input object initially contacts the touch-sensitive surface, the user interface object is in a first state (e.g., a hovering state), and it visually responds to the changed hovering distance in a first manne...

Claims

1. A method comprising: At electronic devices with touch-sensitive surfaces and displays: displaying a position indicator within text on the display when an input object over the touch-sensitive surface satisfies hover criteria, wherein the text is located at a first position on the display; while displaying the position indicator within the text on the display, detecting a first movement of the input object including a component of movement parallel to the touch-sensitive surface, wherein the input object continues to satisfy the hover criteria during the first movement; In response to detecting the first movement of the input object over the touch-sensitive surface: moving the position indicator within the displayed text according to the first movement of the input object; as well as maintaining display of the text at the first position on the display during movement of the position indicator within the displayed text; after moving the position indicator within the displayed text in accordance with the first movement of the input object, detecting a second movement of the input object including a component of movement parallel to the touch-sensitive surface; as well as In response to detecting the second movement of the input object across the touch-sensitive surface: Based on determining that the second movement of the input object is detected while the input object is in contact with the touch-sensitive surface, text is selected based on the second movement of the input object across the touch-sensitive surface while the input object remains in contact with the touch-sensitive surface.

2. The method according to claim 1, further comprising: detecting contact of the input object with the touch-sensitive surface at a location on the touch-sensitive surface followed by the second movement of the input object across the touch-sensitive surface; as well as In response to detecting that the input object makes contact with the touch-sensitive surface at the location on the touch-sensitive surface, and then the input object makes the second movement across the touch-sensitive surface: initiating text selection within the text at a location on the display corresponding to the location on the touch-sensitive surface, and The selected text is extended according to the second movement of the input object across the touch-sensitive surface.

3. The method according to claim 1, further comprising: In response to detecting the second movement of the input object across the touch-sensitive surface: Based on determining that the second movement of the input object is detected while the input object is hovering over the touch-sensitive surface without contacting the touch-sensitive surface, the position indicator is moved based on the second movement of the input object without selecting text. The method of claim 1 , wherein the hover criteria comprises a criterion that is satisfied when a hover proximity parameter of the input object is less than a threshold proximity value.

5. The method according to claim 4, wherein: In response to detecting that the hover proximity parameter of the input object over a location on the touch-sensitive surface is less than the threshold proximity value, the location indicator is initially displayed within the text.

6. The method of claim 1 , wherein displaying the position indicator within the text comprises displaying the position indicator within the text at a position based on one or more of: a projected position of the tip of the input object on the touch-sensitive surface; an orientation of the input object relative to the touch-sensitive surface; a tilt of the input object relative to the touch-sensitive surface; and The distance of the input object relative to the touch-sensitive surface.

7. The method according to claim 1, further comprising: When the hover criteria are met, a magnifying glass object is displayed over a portion of the displayed text, wherein the magnifying glass object displays a magnified image of the position indicator and text adjacent to the position indicator.

8. The method according to claim 7, further comprising: In response to detecting the first movement of the input object over the touch-sensitive surface: moving the magnifier object according to the first movement of the input object over the touch-sensitive surface; as well as The magnifying glass object is updated to display an updated magnified image of the position indicator and text adjacent to the position indicator.

9. The method according to claim 1, further comprising: detecting liftoff of the input object from the touch-sensitive surface while selecting the first portion of the text on the display in accordance with the second movement of the input object; as well as, In response to detecting the liftoff of the input object, maintaining the first portion of the text selected on the display.

10. The method according to claim 9, further comprising: In response to detecting the liftoff of the input object, a menu of options for the selected first portion of the text is displayed.

11. The method according to claim 9, further comprising: After detecting the lift-off of the input object: determining whether the input object satisfies the hovering criteria; Based on determining that the input object satisfies the hovering criterion, maintaining display of the position indicator on the display; as well as Based on determining that the input object ceases to satisfy the hovering criteria, ceasing to display the position indicator on the display.

12. The method according to claim 9, further comprising: After detecting the lift-off of the input object: detecting a third movement of the input object over the touch-sensitive surface when the input object satisfies the hovering criteria; as well as, In response to detecting the third movement of the input object over the touch-sensitive surface, moving the position indicator in accordance with the third movement of the input object while maintaining selection of the first portion of the text on the display.

13. The method according to claim 12, further comprising: detecting contact of the input object with the touch-sensitive surface at the location on the touch-sensitive surface after moving the position indicator in accordance with the third movement of the input object over the touch-sensitive surface; as well as, In response to detecting that the input object makes contact with the touch-sensitive surface at the location on the touch-sensitive surface: deselecting the first portion of the text; and Text selection is resumed at the location on the touch-sensitive surface.

14. An electronic device comprising: monitor; Touch-sensitive surfaces; one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the one or more programs comprising instructions for executing any of the methods according to claims 1 to 13 or causing any of the methods according to claims 1 to 13 to be executed.

15. A computer-readable storage medium comprising executable instructions, which, when executed by one or more processors of an electronic device having a display and a touch-sensitive surface, cause the electronic device to perform any of the methods according to claims 1 to 13 or cause any of the methods according to claims 1 to 13 to be performed.

Images