Implementation of biometric authentication
By detecting the error status of the biometric sensor in electronic devices and displaying the error cause, the problem of users having difficulty identifying the error cause in the prior art is solved, and faster and more effective biometric authentication is achieved, which improves the User experience and device efficiency.
Patent Information
- Application Number
- CN202110011700.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-03-29
- Filing Date
- 2019-06-01
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-06-01
AI Technical Summary
In the prior art, when realizing biometric authentication, it is difficult to identify the cause of error when a user registration or authentication fails, resulting in wasting user time and device energy, especially in battery-driven devices.
Provide a faster and more efficient method and interface for biometric authentication, which reduces user input and device power consumption, improves user experience and Equipment efficiency.
The biometric authentication process is simplified, the user's cognitive burden is reduced, the device's battery life and user time is saved, and the device's usage efficiency and user satisfaction are improved.
Smart Images

Figure CN112699828B_ABST
Abstract
Description
[0001] This application is a divisional application of an invention patent application with an international application date of June 1, 2019, a national application number of 201980036743.5 (international application number PCT / US2019 / 035092), and an invention name of “Implementation of Biometric Authentication”.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application claims priority to U.S. Application No. 16 / 369,355, filed on March 29, 2019, entitled “IMPLEMENTATION OF BIOMETRIC AUTHENTICATION”; U.S. Application No. 62 / 752,234, filed on October 29, 2018, entitled “IMPLEMENTATION OF BIOMETRIC AUTHENTICATION” and U.S. Application No. 62 / 679,955, filed on June 3, 2018, entitled “IMPLEMENTATION OF BIOMETRIC AUTHENTICATION”. Technical Field
[0004] The present disclosure relates generally to biometric authentication, and more particularly, to interfaces and techniques for enrollment and authentication of biometric features. Background Art
[0005] Using electronic devices to perform biometric authentication, such as face, iris, or fingerprint, is a convenient and effective way to authenticate users of electronic devices. Biometric authentication allows devices to quickly and easily verify the identity of any number of users. Summary of the invention
[0006] However, some techniques for implementing biometric authentication using electronic devices are often cumbersome. When a user fails to register a biometric feature for biometric authentication or fails to perform biometric authentication, the user is often unaware of the root cause of the failure. Therefore, the user may be prevented from using biometric authentication altogether. In addition, when a user performs additional attempts to register a biometric feature or authenticate with biometrics after a failure, the user often does so without understanding how to correct the root cause of the failure. In view of the foregoing deficiencies, the prior art requires more time than necessary, which wastes the user's time and device energy. This latter consideration is particularly significant in the operation of battery-powered devices.
[0007] Therefore, the present technology provides a faster and more efficient method and interface for implementing biometric authentication for electronic devices. Such methods and interfaces optionally supplement or replace other methods for implementing biometric authentication. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-driven computing devices, such methods and interfaces save power and increase the time interval between battery charges. Such methods and interfaces also reduce the number of unnecessary, irrelevant or repeated inputs required at computing devices (such as smart phones and smart watches).
[0008] According to some examples, a method is described, the method comprising: at an electronic device having a display and one or more input devices: receiving a request to perform an operation requiring authentication via the one or more input devices; and in response to the request to perform the operation requiring authentication: based on determining that the authentication is successful, performing the operation; and based on determining that the authentication is unsuccessful and a set of error condition criteria are met: displaying an indication of the error condition on the display, wherein the indication includes information about the cause of the error condition; and abandoning performance of the operation.
[0009] According to some examples, a non-transitory computer-readable storage medium is described, which includes one or more programs configured to be executed by one or more processors of an electronic device having a display and one or more input devices, the one or more programs including instructions for performing the following operations: receiving a request to perform an operation requiring authentication via the one or more input devices; and in response to the request to perform the operation requiring authentication: based on determining that the authentication is successful, performing the operation; and based on determining that the authentication is unsuccessful and a set of error condition criteria are met: displaying an indication of the error condition on the display, wherein the indication includes information about the cause of the error condition; and abandoning performance of the operation.
[0010] According to some examples, a transient computer-readable storage medium is described, which includes one or more programs configured to be executed by one or more processors of an electronic device having a display and one or more input devices, the one or more programs including instructions for performing the following operations: receiving a request to perform an operation requiring authentication via the one or more input devices; and in response to the request to perform the operation requiring authentication: based on determining that the authentication is successful, performing the operation; and based on determining that the authentication is unsuccessful and a set of error condition criteria are met: displaying an indication of the error condition on the display, wherein the indication includes information about the cause of the error condition; and abandoning performance of the operation.
[0011] According to some examples, an electronic device is described, which includes: one or more input devices; a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving a request to perform an operation requiring authentication via the one or more input devices; and in response to the request to perform the operation requiring authentication: based on determining that the authentication is successful, performing the operation; and based on determining that the authentication is unsuccessful and a set of error condition criteria are met: displaying an indication of the error condition on the display, wherein the indication includes information about the cause of the error condition; and abandoning performance of the operation.
[0012] According to some examples, an electronic device is described, which includes: one or more input devices; a display; a device for receiving a request to perform an operation requiring authentication via the one or more input devices; and a device for: in response to the request to perform the operation requiring authentication, based on determining that the authentication is successful, performing the operation; and based on determining that the authentication is unsuccessful and a set of error condition criteria are met: displaying an indication of the error condition on the display, wherein the indication includes information about the cause of the error condition; and abandoning performance of the operation.
[0013] According to some examples, a method is described, the method comprising: at an electronic device having a display and a biometric sensor located at a first portion of the electronic device: detecting the presence of an error condition that prevents the biometric sensor from obtaining biometric information about a device user; in response to detecting the presence of the error condition, displaying an error indication on the display, wherein the error indication is displayed at a location proximate to the first portion of the electronic device, comprising: based on determining that a user interface of the electronic device is in a first orientation relative to the biometric sensor, displaying the error indication at a first location in the user interface proximate to the first portion of the electronic device; and based on determining that the user interface of the electronic device is in a second orientation relative to the biometric sensor, displaying the error indication at a second location in the user interface proximate to the first portion of the electronic device, the first orientation being different from the second orientation.
[0014] According to some examples, a non-transitory computer-readable storage medium is described, which includes one or more programs configured to be executed by one or more processors of an electronic device having a display and a biometric sensor located at a first portion of the electronic device, the one or more programs including instructions for the following operations: detecting the presence of an error condition that prevents the biometric sensor from obtaining biometric information about a device user; in response to detecting the presence of the error condition, displaying an error indication on the display, wherein the error indication is displayed at a location proximate to the first portion of the electronic device, including: based on determining that the user interface of the electronic device is in a first orientation relative to the biometric sensor, displaying the error indication at a first location in the user interface proximate to the first portion of the electronic device; and based on determining that the user interface of the electronic device is in a second orientation relative to the biometric sensor, displaying the error indication at a second location in the user interface proximate to the first portion of the electronic device, the first orientation being different from the second orientation.
[0015] According to some examples, a transient computer-readable storage medium is described, which includes one or more programs configured to be executed by one or more processors of an electronic device having a display and a biometric sensor located at a first portion of the electronic device, the one or more programs including instructions for the following operations: detecting the presence of an error condition that prevents the biometric sensor from obtaining biometric information about a device user; in response to detecting the presence of the error condition, displaying an error indication on the display, wherein the error indication is displayed at a location proximate to the first portion of the electronic device, including: based on determining that the user interface of the electronic device is in a first orientation relative to the biometric sensor, displaying the error indication at a first location in the user interface proximate to the first portion of the electronic device; and based on determining that the user interface of the electronic device is in a second orientation relative to the biometric sensor, displaying the error indication at a second location in the user interface proximate to the first portion of the electronic device, the first orientation being different from the second orientation.
[0016] According to some examples, an electronic device is described, comprising: a biometric sensor located at a first portion of the electronic device; a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: detecting the presence of an error condition that prevents the biometric sensor from obtaining biometric information about a device user; in response to detecting the presence of the error condition, displaying an error indication on the display, wherein the error indication is displayed at a position near the first portion of the electronic device, comprising: displaying the error indication at a first position in the user interface near the first portion of the electronic device based on determining that the user interface of the electronic device is in a first orientation relative to the biometric sensor; and displaying the error indication at a second position in the user interface near the first portion of the electronic device based on determining that the user interface of the electronic device is in a second orientation relative to the biometric sensor, the first orientation being different from the second orientation.
[0017] According to some examples, an electronic device is described, which includes: a biometric sensor located at a first part of the electronic device; a display; a device for detecting the existence of an error condition that prevents the biometric sensor from obtaining biometric information about a device user; and a device for displaying an error indication on the display in response to detecting the existence of the error condition, wherein the error indication is displayed at a location near the first part of the electronic device, including: displaying the error indication at a first location in the user interface near the first part of the electronic device based on determining that the user interface of the electronic device is in a first orientation relative to the biometric sensor; and displaying the error indication at a second location in the user interface near the first part of the electronic device based on determining that the user interface of the electronic device is in a second orientation relative to the biometric sensor, the first orientation being different from the second orientation.
[0018] According to some examples, a method is described, the method including: at an electronic device having a display and one or more biometric sensors: displaying on the display a biometric registration user interface for initiating biometric registration using the one or more biometric sensors; while displaying the biometric registration user interface, receiving input corresponding to a request to initiate biometric registration; and in response to receiving the input: based on determining that the orientation of the electronic device satisfies a set of registration criteria, initiating a process for registering biometric features using the one or more biometric sensors; and based on determining that the orientation of the electronic device does not satisfy the set of registration criteria, outputting one or more prompts to change the orientation of the electronic device to a different orientation that satisfies the set of registration criteria.
[0019] According to some examples, a non-transitory computer-readable storage medium is described, which includes one or more programs configured to be executed by one or more processors of an electronic device having a display and one or more biometric sensors, the one or more programs including instructions for performing the following operations: displaying on the display a biometric registration user interface for initiating biometric registration using one or more biometric sensors; while displaying the biometric registration user interface, receiving input corresponding to a request to initiate biometric registration; and in response to receiving the input: based on determining that the orientation of the electronic device satisfies a set of registration criteria, initiating a process for registering biometric features using one or more biometric sensors; and based on determining that the orientation of the electronic device does not satisfy a set of registration criteria, outputting one or more prompts to change the orientation of the electronic device to a different orientation that satisfies a set of registration criteria.
[0020] According to some examples, a transient computer-readable storage medium is described, which includes one or more programs configured to be executed by one or more processors of an electronic device having a display and one or more biometric sensors, the one or more programs including instructions for performing the following operations: displaying a biometric registration user interface on the display for initiating biometric registration using one or more biometric sensors; while displaying the biometric registration user interface, receiving input corresponding to a request to initiate biometric registration; and in response to receiving the input: initiating a process for registering biometric features using one or more biometric sensors based on determining that the orientation of the electronic device satisfies a set of registration criteria; and outputting one or more prompts to change the orientation of the electronic device to a different orientation that satisfies the set of registration criteria based on determining that the orientation of the electronic device does not satisfy the set of registration criteria.
[0021] According to some examples, an electronic device is described, the electronic device comprising: one or more biometric sensors; a display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: displaying on the display a biometric registration user interface for initiating biometric registration using the one or more biometric sensors; while displaying the biometric registration user interface, receiving input corresponding to a request to initiate biometric registration; and in response to receiving the input: initiating a process for registering biometric features using the one or more biometric sensors based on determining that the orientation of the electronic device satisfies a set of registration criteria; and outputting one or more prompts to change the orientation of the electronic device to a different orientation that satisfies the set of registration criteria based on determining that the orientation of the electronic device does not satisfy the set of registration criteria.
[0022] According to some examples, an electronic device is described, which includes: one or more biometric sensors; a display; a device for displaying on the display a biometric registration user interface for initiating biometric registration using the one or more biometric sensors; a device for receiving input corresponding to a request to initiate biometric registration while displaying the biometric registration user interface; and a device for, in response to receiving the input: initiating a process for registering biometric features using the one or more biometric sensors based on determining that the orientation of the electronic device satisfies a set of registration criteria; and outputting one or more prompts to change the orientation of the electronic device to a different orientation that satisfies the set of registration criteria based on determining that the orientation of the electronic device does not satisfy the set of registration criteria.
[0023] According to some examples, a method is described, the method including: at an electronic device having a biometric sensor and a touch-sensitive display: detecting an occurrence of an error condition for detecting biometric information at the biometric sensor; in response to detecting the occurrence of the error condition, displaying an indication of a location of the biometric sensor on the electronic device on the touch-sensitive display; while displaying the indication of the location of the biometric sensor on the electronic device, detecting a request to unlock the electronic device using the biometric sensor; and in response to detecting the request to unlock the electronic device using the biometric sensor: based on determining that the error condition still occurs at a corresponding time occurring after detecting the request to unlock the electronic device: stopping displaying the indication of the location of the biometric sensor; and displaying a touch-based user interface for entering touch-based authentication information; and based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
[0024] According to some examples, a non-transitory computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of an electronic device having a biometric sensor and a touch-sensitive display, the one or more programs including instructions for performing the following operations: detecting the occurrence of an error condition for detecting biometric information at a biometric sensor; in response to detecting the occurrence of the error condition, displaying an indication of the location of the biometric sensor on the electronic device on the touch-sensitive display; while displaying the indication of the location of the biometric sensor on the electronic device, detecting a request to unlock the electronic device using the biometric sensor; and in response to detecting the request to unlock the electronic device using the biometric sensor: based on determining that the error condition still occurs at a corresponding time that occurs after detecting the request to unlock the electronic device: stopping displaying the indication of the location of the biometric sensor; and displaying a touch-based user interface for entering touch-based authentication information; and based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
[0025] According to some examples, a transient computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of an electronic device having a biometric sensor and a touch-sensitive display, the one or more programs including instructions for performing the following operations: detecting the occurrence of an error condition for detecting biometric information at a biometric sensor; in response to detecting the occurrence of the error condition, displaying an indication of the location of the biometric sensor on the electronic device on the touch-sensitive display; while displaying the indication of the location of the biometric sensor on the electronic device, detecting a request to unlock the electronic device using the biometric sensor; and in response to detecting the request to unlock the electronic device using the biometric sensor: based on determining that the error condition still occurs at a corresponding time that occurs after detecting the request to unlock the electronic device: stopping displaying the indication of the location of the biometric sensor; and displaying a touch-based user interface for entering touch-based authentication information; and based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
[0026] According to some examples, an electronic device is described, which includes: a biometric sensor; a touch-sensitive display; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting the occurrence of an error condition for detecting biometric information at a biometric sensor; in response to detecting the occurrence of the error condition, displaying an indication of the location of the biometric sensor on the electronic device on the touch-sensitive display; while displaying the indication of the location of the biometric sensor on the electronic device, detecting a request to unlock the electronic device using the biometric sensor; and in response to detecting the request to unlock the electronic device using the biometric sensor: based on determining that the error condition still occurs at a corresponding time that occurs after detecting the request to unlock the electronic device: stopping displaying the indication of the location of the biometric sensor; and displaying a touch-based user interface for entering touch-based authentication information; and based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
[0027] According to some examples, an electronic device is described, which includes: a biometric sensor; a touch-sensitive display; a device for detecting the occurrence of an error condition for detecting biometric information at the biometric sensor; a device for displaying an indication of the location of the biometric sensor on the electronic device on the touch-sensitive display in response to detecting the occurrence of the error condition; a device for detecting a request to unlock the electronic device using the biometric sensor while displaying the indication of the location of the biometric sensor on the electronic device; and a device for performing the following operations in response to detecting the request to unlock the electronic device using the biometric sensor: based on determining that the error condition still occurs at a corresponding time occurring after the request to unlock the electronic device is detected: stopping displaying the indication of the location of the biometric sensor; and displaying a touch-based user interface for entering touch-based authentication information; and based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
[0028] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.
[0029] Thus, a faster, more efficient method and interface is provided for devices to implement biometric authentication, thereby improving the effectiveness, efficiency, and user satisfaction of such devices.Such methods and interfaces optionally supplement or replace other methods for implementing biometric authentication. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] For a better understanding of the various described examples, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.
[0031] Figure 1A is a block diagram showing a portable multifunction device with a touch-sensitive display according to some embodiments.
[0032] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.
[0033] Figure 1C is a block diagram illustrating example components for generating tactile output according to some embodiments.
[0034] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.
[0035] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.
[0036] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to some embodiments.
[0037] Figure 4B An exemplary user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.
[0038] FIG. 4C to FIG. 4H Exemplary tactile output patterns with specific waveforms according to some embodiments are shown.
[0039] Figure 5A A personal electronic device according to some embodiments is shown.
[0040] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.
[0041] FIG. 5C to FIG. 5D Example components of a personal electronic device with a touch-sensitive display and an intensity sensor are shown according to some embodiments.
[0042] FIG. 5E to FIG. 5H Exemplary components and user interfaces of a personal electronic device according to some embodiments are shown.
[0043] Figure 6 Example devices connected via one or more communication channels are shown in accordance with some embodiments.
[0044] 7A to 7AD An exemplary user interface is shown for providing an indication of error conditions during biometric authentication according to some examples.
[0045] FIG. 8A to FIG. 8B is a flowchart illustrating a method for providing an indication of an error condition during biometric authentication according to some examples.
[0046] FIG. 9A to FIG. 9U An exemplary user interface for providing indications about a biometric sensor during biometric authentication according to some examples is shown.
[0047] FIG. 10A to FIG. 10C is a flowchart illustrating a method for providing indications about a biometric sensor during biometric authentication according to some examples.
[0048] FIG. 11A to FIG. 11SAn exemplary user interface for orienting a device to register a biometric feature is shown according to some examples.
[0049] FIG. 12A to FIG. 12C is a flow chart illustrating a method for orienting a device to register a biometric feature according to some examples.
[0050] FIG. 13A to FIG. 13Z An exemplary user interface is shown for providing an indication of the position of a biometric sensor to correct a detected error condition according to some examples.
[0051] FIG. 14A to FIG. 14B is a flow chart illustrating a method for providing an indication of the position of a biometric sensor to correct a detected error condition according to some examples. DETAILED DESCRIPTION
[0052] The following description sets forth exemplary methods, parameters, etc. However, it should be appreciated that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.
[0053] Electronic devices need to provide effective methods and interfaces for implementing biometric authentication of biometric features. For example, electronic devices need to provide convenient and effective methods for registering one or more parts of biometric features. As another example, electronic devices need to provide fast and intuitive techniques for selectively accessing secure data based on biometric authentication. As another example, electronic devices need to provide fast and intuitive techniques for enabling device functionality based on biometric authentication. Such techniques can reduce the cognitive burden on users who use devices to register biometric features and / or authenticate with biometrics, thereby improving overall productivity. In addition, such techniques can reduce processor power and battery power that would otherwise be wasted on redundant user input.
[0054] under Figures 1A to 1C , Figure 2 , Figure 3 , FIG. 4A to FIG. 4B and FIG. 5A to FIG. 5H A description of an exemplary device for performing techniques for implementing biometric authentication is provided. Figure 6 Example devices connected via one or more communication channels are shown in accordance with some embodiments. 7A to 7AD An exemplary user interface for providing an indication of error conditions during biometric authentication is shown. FIG. 8A to FIG. 8B is a flow chart illustrating a method for providing an indication of an error condition during biometric authentication. 7A to 7AD The user interface in is used to illustrate the process described below, including FIG. 8A to FIG. 8B process. FIG. 9A to FIG. 9UAn exemplary user interface for providing indications about a biometric feature sensor during biometric authentication is shown. FIG. 10A to FIG. 10C is a flow chart illustrating a method for providing indications regarding a biometric feature sensor during biometric authentication. FIG. 9A to FIG. 9U The user interface in is used to illustrate the process described below, including FIG. 10A to FIG. 10C process. FIG. 11A to FIG. 11S An exemplary user interface for orienting a device to enroll a biometric feature is shown. FIG. 12A to FIG. 12C is a flow chart illustrating a method for orienting a device to enroll a biometric feature. FIG. 11A to FIG. 11S The user interface in is used to illustrate the process described below, including FIG. 12A to FIG. 12C process. FIG. 13A to FIG. 13Z An exemplary user interface is shown for providing an indication of the position of a biometric sensor to correct a detected error condition. FIG. 14A to FIG. 14B is a flow chart illustrating a method for providing an indication of the position of a biometric sensor to correct a detected error condition. FIG. 13A to FIG. 13Z The user interface in is used to illustrate the process described below, including FIG. 14A to FIG. 14B process.
[0055] Although the following description uses the terms "first", "second", etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element. For example, a first touch can be named a second touch and similarly a second touch can be named a first touch without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.
[0056] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.
[0057] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]," or "in response to detecting [a stated condition or event]," depending on the context.
[0058] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).
[0059] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.
[0060] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.
[0061] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.
[0062] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A is a block diagram showing a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.
[0063] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values that includes at least four different values and more typically includes hundreds of different values (e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some specific implementations, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (for example, the intensity threshold is described in units corresponding to the substitute measurement). In some specific implementations, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (for example, the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functions that are not otherwise accessible to the user on a smaller device with limited actual area in some cases, and the smaller device is used to display an enable indication (for example, on a touch-sensitive display) and / or receive user input (for example, via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control, such as a knob or button).
[0064] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, the movement of the touch-sensitive surface may be optionally interpreted or sensed by the user as the "roughness" of the touch-sensitive surface. Although such interpretation of touch by the user will be limited by the user's individualized sensory perception, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user. Providing tactile feedback to the user using tactile output enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0065] In some embodiments, the tactile output pattern specifies characteristics of the tactile output, such as the amplitude of the tactile output, the shape of the motion waveform of the tactile output, the frequency of the tactile output, and / or the duration of the tactile output.
[0066] When a device (e.g., one or more tactile output generators that generate tactile outputs via moving a movable mass) generates tactile outputs with different tactile output patterns, the tactile outputs may cause different tactile sensations to a user holding or touching the device. Although the user's senses are based on the user's perception of the tactile output, most users will be able to recognize changes in the waveform, frequency, and amplitude of the tactile output generated by the device. Therefore, the waveform, frequency, and amplitude may be adjusted to indicate to the user that different operations have been performed. In this way, a tactile output having a tactile output pattern designed, selected, and / or arranged to simulate the characteristics (e.g., size, material, weight, stiffness, smoothness, etc.) of an object in a given environment (e.g., a user interface including graphical features and objects, a simulated physical environment with virtual boundaries and virtual objects, a real physical environment with physical boundaries and physical objects, and / or a combination of any of the above); behavior (e.g., oscillation, displacement, acceleration, rotation, extension, etc.); and / or interaction (e.g., collision, adhesion, repulsion, attraction, friction, etc.) will provide helpful feedback to the user in some cases, which reduces input errors and improves the efficiency of the user's operation of the device. In addition, tactile output is optionally generated to correspond to feedback that is independent of simulated physical properties (such as input threshold or object selection). Such tactile output will provide helpful feedback to the user in some cases, which reduces input errors and improves the efficiency of the user's operation of the device.
[0067] In some embodiments, the tactile output with a suitable tactile output mode serves as a prompt for an event of interest in the user interface or behind the screen in the device. Examples of events of interest include activation of affordances (such as real or virtual buttons or toggle switches) provided on the device or in the user interface, the success or failure of the requested operation, reaching or crossing a boundary in the user interface, entering a new state, switching input focus between objects, activating a new mode, reaching or crossing an input threshold, detecting or identifying a type of input or gesture, etc. In some embodiments, a tactile output is provided to serve as a warning or prompt for an upcoming event or result that will occur unless a change in direction or interruption input is detected in time. Tactile output is also used in other contexts to enrich user experience, improve accessibility of devices for users with visual or motor difficulties or other accessibility needs, and / or improve the efficiency and functionality of user interfaces and / or devices. Optionally, the tactile output is compared with audio input and / or visual user interface changes, which further enhances the user's experience when the user interacts with the user interface and / or device, and is conducive to better transmission of information about the state of the user interface and / or device, and this reduces input errors and improves the efficiency of user operations on the device.
[0068] FIG. 4C to FIG. 4EA set of sample tactile output patterns are provided that can be used, individually or in combination, as is or through one or more transformations (e.g., modulation, amplification, truncation, etc.) to generate suitable tactile feedback in various scenarios for various purposes (such as those described above and those described with respect to the user interfaces and methods discussed herein). This example of a control panel for tactile output shows how a set of three waveforms and eight frequencies can be used to generate an array of tactile output patterns. In addition to the tactile output patterns shown in the figure, each of these tactile output patterns is optionally adjusted in amplitude by changing the gain value of the tactile output pattern, as shown in the figure, for example, for FIG. 4F to FIG. 4H 80Hz, FullTap 200Hz, MiniTap 80Hz, MiniTap200Hz, MicroTap 80Hz, and MicroTap 200Hz, each of which is shown with a gain of 1.0, 0.75, 0.5, and 0.25. FIG. 4F to FIG. 4H As shown, changing the gain of a tactile output pattern changes the amplitude of the pattern without changing the frequency of the pattern or changing the shape of the waveform. In some embodiments, changing the frequency of the tactile output pattern also results in a lower amplitude because some tactile output generators are limited by how much force can be applied to the movable mass, so the higher frequency movement of the mass is constrained to a lower amplitude to ensure that the acceleration required to form the waveform does not require forces outside the operating force range of the tactile output generator (e.g., the peak amplitude of FullTap at 230 Hz, 270 Hz, and 300 Hz is lower than the amplitude of FullTap at 80 Hz, 100 Hz, 125 Hz, and 200 Hz).
[0069] FIG. 4C to FIG. 4H A tactile output pattern having a specific waveform is shown. The waveform of the tactile output pattern represents a tactile output pattern relative to a neutral position (e.g., x zero ) versus time, through which the movable mass passes to generate a tactile output having the tactile output pattern. For example, Figure 4C The first set of tactile output patterns shown (eg, the “FullTap” tactile output pattern) each have a waveform that includes oscillations having two complete cycles (eg, oscillations that start and end at a neutral position and pass through the neutral position three times). Figure 4D The second set of tactile output patterns shown (eg, “MiniTap” tactile output patterns) each have a waveform that includes oscillations having one complete cycle (eg, oscillations that begin and end at a neutral position and pass through the neutral position once). Figure 4EThe third set of tactile output patterns shown (e.g., the "MicroTap" tactile output patterns) each have a waveform that includes an oscillation that includes half a complete cycle (e.g., an oscillation that starts and ends at a neutral position and does not pass through the neutral position). The waveforms of the tactile output patterns also include a start buffer and an end buffer that represent the gradual acceleration and deceleration of the movable mass at the beginning and end of the tactile output. FIG. 4C to FIG. 4H The exemplary waveforms shown include x and y representing the maximum and minimum extents of movement of the movable mass. min and x max For larger electronic devices with larger movable masses, the minimum and maximum degrees of movement of the mass may be larger or smaller. FIG. 4C to FIG. 4H The examples shown describe movement of a mass in 1 dimension, but similar principles may also apply to movement of a movable mass in two or three dimensions.
[0070] like FIG. 4C to FIG. 4E As shown, each tactile output pattern also has a corresponding characteristic frequency that affects the "pitch" of the tactile sensation felt by the user from the tactile output having the characteristic frequency. For continuous tactile output, the characteristic frequency represents the number of cycles (e.g., cycles per second) that the movable mass of the tactile output generator completes in a given time period. For discrete tactile output, a discrete output signal is generated (e.g., having 0.5, 1, or 2 cycles), and the characteristic frequency value specifies how fast the movable mass needs to move to generate a tactile output having the characteristic frequency. FIG. 4C to FIG. 4H As shown, for each type of tactile output (e.g., defined by a corresponding waveform, such as FullTap, MiniTap, or MicroTap), higher frequency values correspond to faster movement of the movable mass and, therefore, generally speaking, to shorter tactile output completion times (e.g., including the time required to complete the number of cycles of the discrete tactile output plus the start and end buffer times). For example, a FullTap with a characteristic frequency of 80 Hz takes longer to complete than a FullTap with a characteristic frequency of 100 Hz (e.g., in Figure 4C, 35.4ms vs. 28.3ms). In addition, for a given frequency, a tactile output having more cycles in its waveform at the corresponding frequency takes longer to complete than a tactile output having fewer cycles in its waveform at the same corresponding frequency. For example, a FullTap at 150Hz takes longer to complete than a MiniTap at 150Hz (e.g., 19.4ms vs. 12.8ms), and a MiniTap at 150Hz takes longer to complete than a MicroTap at 150Hz (e.g., 12.8ms vs. 9.4ms). However, for tactile output modes with different frequencies, this rule may not apply (e.g., a tactile output with more cycles but a higher frequency may take a shorter amount of time to complete than a tactile output with fewer cycles but a lower frequency, and vice versa). For example, at 300Hz, FullTap takes the same amount of time as MiniTap (e.g., 9.9ms).
[0071] like FIG. 4C to FIG. 4E As shown, the tactile output pattern also has a characteristic amplitude, which affects the amount of energy contained in the tactile signal or the "intensity" of the tactile sensation that the user can feel through the tactile output with the characteristic amplitude. In some embodiments, the characteristic amplitude of the tactile output pattern refers to the absolute value or normalized value representing the maximum displacement of the movable mass relative to the neutral position when generating the tactile output. In some embodiments, the characteristic amplitude of the tactile output pattern can be adjusted according to various conditions (e.g., customized based on user interface context and behavior) and / or preconfigured metrics (e.g., input-based metrics, and / or user interface-based metrics), such as by a fixed or dynamically determined gain factor (e.g., a value between 0 and 1). In some embodiments, an input-based metric (e.g., an intensity change metric or an input speed metric) measures the characteristics of the input during the input that triggers the generation of the tactile output (e.g., the rate of change of the characteristic intensity of the contact in the press input or the rate of movement of the contact on the touch-sensitive surface). In some embodiments, a user interface-based metric (e.g., a cross-border speed metric) measures a characteristic of a user interface element (e.g., the speed at which the element moves across a hidden or visible boundary in the user interface) during a user interface change that triggers generation of a tactile output. In some embodiments, a characteristic amplitude of a tactile output pattern may be "envelope" modulated, and peaks of adjacent cycles may have different amplitudes, wherein one of the waveforms shown above is further modified by multiplying an envelope parameter that changes over time (e.g., from 0 to 1) to gradually adjust the amplitude of a portion of the tactile output over time when generating a tactile output.
[0072] Although in FIG. 4C to FIG. 4ESpecific frequencies, amplitudes, and waveforms are shown in sample tactile output patterns for illustrative purposes, but tactile output patterns with other frequencies, amplitudes, and waveforms may be used for similar purposes. For example, a waveform with between 0.5 and 4 cycles may be used. Other frequencies in the range of 60 Hz to 400 Hz may also be used. Table 1 provides examples of specific tactile feedback behaviors, configurations, and examples of their use.
[0073] It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the EMBODIMENTS 2000 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0074] Memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.
[0075] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.
[0076] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuits for detecting near field communication (NFC) fields, such as by detecting through a short-range communication radio component. Wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communications protocol including communications protocols that have not been developed as of the filing date of this document.
[0077] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).
[0078] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, a depth camera controller 169, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive / send electrical signals from other input control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 is optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an increase / decrease button for volume control of the speaker 111 and / or the microphone 113. One or more buttons optionally include a push button (e.g., Figure 2 206).
[0079] A quick press of the push button optionally disengages the lock of the touch screen 112 or optionally begins the process of unlocking the device using gestures on the touch screen, as described in U.S. Patent Application 11 / 322,549, filed on December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image" (i.e., U.S. Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.
[0080] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.
[0081] The touch screen 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.
[0082] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® from Apple Inc. (Cupertino, California). and iPod The technology used in.
[0083] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive trackpad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, while the touch-sensitive trackpad does not provide visual output.
[0084] In some embodiments, the touch-sensitive display of the touch screen 112 is as described in the following patent applications: (1) U.S. patent application No. 11 / 381,313, entitled “Multipoint Touch Surface Controller,” filed on May 2, 2006; (2) U.S. patent application No. 10 / 840,862, entitled “Multipoint Touchscreen,” filed on May 6, 2004; (3) U.S. patent application No. 10 / 903,964, entitled “Gestures For Touch Sensitive Input Devices,” filed on July 30, 2004; (4) U.S. patent application No. 11 / 048,264, entitled “Gestures For Touch Sensitive Input Devices,” filed on January 31, 2005; (5) U.S. patent application No. 11 / 086,874, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices,” filed on January 18, 2005 and (9) U.S. patent application No. 11 / 367,749, filed on March 3, 2006, entitled “Multi-Functional Hand-Held Device.” All of these applications are hereby incorporated by reference in their entirety.
[0085] The touch screen 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.
[0086] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.
[0087] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.
[0088] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures a static image or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to the touch screen display 112 on the front of the device, so that the touch screen display can be used as a viewfinder for static images and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device so that the user's image is optionally obtained for video conferencing while the user views other video conference participants on the touch screen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that a single optical sensor 164 is used with the touch screen display for both video conferencing and static image and / or video image acquisition.
[0089] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to a force sensor controller 159 in the I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite to touch screen display 112 located on the front of device 100.
[0090] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent applications: No. 11 / 241,839, entitled "Proximity Detector In Handheld Device"; No. 11 / 240,788, entitled "Proximity Detector In Handheld Device"; No. 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and No. 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.
[0091] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices, such as a speaker or other audio component; and / or an electromechanical device that converts energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in parallel or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.
[0092] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 coupled to the peripheral device interface 118 is shown. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 is optionally implemented as described in the following U.S. patent publications: U.S. Patent Publication No. 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication No. 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on a touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.
[0093] In some embodiments, the device 100 also includes one or more fingerprint sensors (or communicates with them). The one or more fingerprint sensors are coupled to the peripheral device interface 118. Alternatively, the one or more fingerprint sensors are optionally coupled to the input controller 160 in the I / O system 106. However, in a common embodiment, a secure dedicated computing hardware (e.g., one or more processors, memory and / or communication bus) with additional security features is used to perform fingerprint recognition operations in order to enhance the security of the fingerprint information determined by the one or more fingerprint sensors. As used herein, a fingerprint sensor is a sensor that is capable of distinguishing fingerprint features (sometimes called "detail features") of ridges and valleys of the skin, such as those found on human fingers and toes. The fingerprint sensor can use any of a variety of technologies to distinguish fingerprint features, including but not limited to: optical fingerprint imaging, ultrasonic fingerprint imaging, active capacitive fingerprint imaging, and passive capacitive fingerprint imaging. In some embodiments, in addition to distinguishing fingerprint features in a fingerprint, the one or more fingerprint sensors are also capable of tracking the movement of fingerprint features over time, thereby determining / characterizing the movement of the fingerprint over time on the one or more fingerprint sensors. Although the one or more fingerprint sensors may be separate from the touch-sensitive surface (e.g., touch-sensitive display system 112), it should be understood that in some embodiments, the touch-sensitive surface (e.g., touch-sensitive display system 112) has a spatial resolution high enough to detect fingerprint features formed by individual fingerprint ridges and is used as a fingerprint sensor instead of or in addition to the one or more fingerprint sensors. In some embodiments, the device 100 includes a set of one or more orientation sensors for determining the orientation of a finger or hand on or adjacent to the device (e.g., the orientation of a finger over one or more fingerprint sensors). Additionally, in some embodiments, in addition to or in place of a fingerprint sensor, the set of one or more orientation sensors is used to detect rotation of a contact that is interacting with the device (e.g., in one or more of the methods described below, instead of using a fingerprint sensor to detect rotation of a fingerprint / contact, the set of one or more orientation sensors is used to detect rotation of a contact that includes a fingerprint, where features of the fingerprint are detected or not).
[0094] In some embodiments, the features of the fingerprint and the comparison between the features of the detected fingerprint and the features of the stored fingerprint are performed by secure dedicated computing hardware (e.g., one or more processors, memory, and / or communication buses) separate from the processor 120 to improve the security of the fingerprint data generated, stored, and processed by the one or more fingerprint sensors. In some embodiments, the features of the fingerprint and the comparison between the features of the detected fingerprint and the features of the registered fingerprint are performed by the processor 120 using a fingerprint analysis module.
[0095] In some embodiments, during the registration process, the device (e.g., a fingerprint analysis module or an independent security module in communication with the one or more fingerprint sensors) collects biometric information about one or more fingerprints of the user (e.g., identifying the relative positions of multiple nodes in the fingerprint of the user). After the registration process has been completed, the biometric information is stored at the device (e.g., in a secure fingerprint module) for later use in authenticating the detected fingerprint. In some embodiments, the biometric information stored at the device excludes the image of the fingerprint and also excludes information from which the image of the fingerprint can be reconstructed, so that the image of the fingerprint does not become available unintentionally when the security of the device is compromised. In some embodiments, during the authentication process, the device (e.g., a fingerprint analysis module or an independent security module in communication with the one or more fingerprint sensors) determines whether the finger input detected by the one or more fingerprint sensors includes a fingerprint that matches a fingerprint previously registered by collecting biometric information about the fingerprint detected on the one or more fingerprint sensors (e.g., identifying the relative positions of multiple nodes in the fingerprint detected on the one or more fingerprint sensors) and compares the biometric information corresponding to the detected fingerprint with the biometric information corresponding to the registered fingerprint. In some embodiments, comparing the biometric information corresponding to the detected fingerprint with the biometric information corresponding to the registered fingerprint includes comparing the types and positions of nodes in the biometric information corresponding to the detected fingerprint with the types and positions of nodes in the biometric information corresponding to the registered fingerprint. However, the determination as to whether the finger input includes a fingerprint that matches a previously registered fingerprint registered using the device is optionally performed using any of a variety of well-known fingerprint authentication techniques to determine whether the detected fingerprint matches the registered fingerprint.
[0096] Device 100 optionally also includes one or more depth camera sensors 175 . Figure 1AA depth camera sensor coupled to a depth camera controller 169 in the I / O subsystem 106 is shown. The depth camera sensor 175 receives data from the environment to create a three-dimensional model of an object (e.g., a face) within the scene from a viewpoint (e.g., a depth camera sensor). In some embodiments, together with the imaging module 143 (also referred to as a camera module), the depth camera sensor 175 is optionally used to determine a depth map of different portions of the image captured by the imaging module 143. In some embodiments, the depth camera sensor is located at the front of the device 100, so that the user's image with depth information is optionally obtained for video conferencing while the user views other video conference participants on the touch screen display, and a selfie with depth map data is captured. In some embodiments, the depth camera sensor 175 is located at the rear of the device, or the rear and front of the device 100. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that the depth camera sensor 175 is used with the touch screen display for both video conferencing and static image and / or video image acquisition.
[0097] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; a sensor state, including information obtained from the device's various sensors and input control devices 116; and position information about the device's position and / or posture.
[0098] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitating communication between various hardware components and software components.
[0099] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to (trademark of Apple Inc.) devices.
[0100] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on a touch pad.
[0101] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).
[0102] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.
[0103] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.
[0104] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.
[0105] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .
[0106] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).
[0107] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services, such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).
[0108] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:
[0109] Contacts module 137 (sometimes called address book or contact list);
[0110] Telephone module 138;
[0111] Video conferencing module 139;
[0112] Email client module 140;
[0113] Instant messaging (IM) module 141;
[0114] Fitness support module 142;
[0115] A camera module 143 for still images and / or video images;
[0116] Image management module 144;
[0117] Video player module;
[0118] Music player module;
[0119] Browser module 147;
[0120] Calendar module 148;
[0121] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;
[0122] A widget creator module 150 for forming a user-created widget 149-6;
[0123] Search module 151;
[0124] Video and music player module 152, which merges the video player module and the music player module;
[0125] Notepad module 153;
[0126] Map module 154; and / or
[0127] Online video module 155.
[0128] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.
[0129] In conjunction with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and the like.
[0130] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a sequence of characters corresponding to a phone number, access one or more phone numbers in contact module 137, modify an entered phone number, dial a corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.
[0131] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, the text input module 134, the contact module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.
[0132] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.
[0133] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).
[0134] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154 and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing and transmitting fitness data.
[0135] In conjunction with the touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132 and image management module 144, the camera module 143 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.
[0136] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.
[0137] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with user instructions, including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
[0138] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.
[0139] In conjunction with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).
[0140] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop widget creator module 150 is optionally used by a user to create a desktop widget (e.g., converting a user-specified portion of a web page into a desktop widget).
[0141] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.
[0142] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).
[0143] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.
[0144] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the GPS module 135 and the browser module 147, the map module 154 is optionally used to receive, display, modify and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.
[0145] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for performing the following operations: allowing a user to access, browse, receive (e.g., by streaming and / or downloading), playback (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a specific online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a specific online video. Additional descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.
[0146] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.
[0147] In some embodiments, the device 100 is a device on which operation of a predefined set of functions is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.
[0148] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using a trackpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.
[0149] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).
[0150] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event distributor module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.
[0151] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.
[0152] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.
[0153] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).
[0154] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.
[0155] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.
[0156] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.
[0157] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.
[0158] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.
[0159] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receivers 182.
[0160] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.
[0161] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in the application views 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.
[0162] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).
[0163] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).
[0164] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.
[0165] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and the object that triggered the hit test.
[0166] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.
[0167] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.
[0168] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.
[0169] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.
[0170] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.
[0171] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.
[0172] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.
[0173] Figure 1C is a block diagram showing a tactile output module according to some embodiments. In some embodiments, the I / O subsystem 106 (e.g., the tactile feedback controller 161 ( Figure 1A ) and / or other input controller 160 ( Figure 1A ))include Figure 1C At least some of the example components shown. In some embodiments, the peripheral device interface 118 includes Figure 1C At least some of the example components shown.
[0174] In some embodiments, the tactile output module includes a tactile feedback module 133. In some embodiments, the tactile feedback module 133 aggregates and combines tactile outputs from software applications on the electronic device for user interface feedback (e.g., feedback in response to user input corresponding to the displayed user interface and prompts and other notifications indicating the execution of operations or the occurrence of events in the user interface of the electronic device). The tactile feedback module 133 includes one or more of a waveform module 123 (for providing a waveform for generating tactile output), a mixer 125 (for mixing waveforms, such as waveforms in different channels), a compressor 127 (for reducing or compressing the dynamic range of the waveform), a low-pass filter 129 (for filtering out high-frequency signal components in the waveform), and a thermal controller 131 (for adjusting the waveform according to thermal conditions). In some embodiments, the tactile feedback module 133 is included in a tactile feedback controller 161 ( Figure 1AIn some embodiments, a separate unit of the tactile feedback module 133 (or a separate specific implementation of the tactile feedback module 133) is also included in the audio controller (e.g., the audio circuit 110, Figure 1A ) and used to generate an audio signal. In some embodiments, a single tactile feedback module 133 is used to generate an audio signal and to generate a waveform for a tactile output.
[0175] In some embodiments, the tactile feedback module 133 also includes a trigger module 121 (e.g., a software application, operating system, or other software module that determines that a tactile output is to be generated and triggers a process for generating the corresponding tactile output). In some embodiments, the trigger module 121 generates a trigger signal for triggering (e.g., by the waveform module 123) the generation of a waveform. For example, the trigger module 121 generates the trigger signal based on a preset timing standard. In some embodiments, the trigger module 121 receives a trigger signal from outside the tactile feedback module 133 (e.g., in some embodiments, the tactile feedback module 133 receives the trigger signal from a hardware input processing module 146 located outside the tactile feedback module 133) and relays the trigger signal to other components within the tactile feedback module 133 (e.g., the waveform module 123) or a software application that triggers an operation (e.g., using the trigger module 121) based on the activation of a user interface element (e.g., an application icon or an enable representation within an application) or a hardware input device (e.g., a main button or an intensity-sensitive input interface, such as an intensity-sensitive touch screen). In some embodiments, the trigger module 121 also (e.g., from a hardware input processing module 146 that receives the trigger signal from the hardware input processing module 146 that is located outside the tactile feedback module 133). Figure 1A and Figure 3 In some embodiments, the trigger module 121 responds to the tactile feedback module 133 (or the trigger module 121 in the tactile feedback module 133) (for example, from Figure 1A and Figure 3 The tactile feedback module 133 in receives the tactile feedback instruction and generates a trigger signal.
[0176] Waveform module 123 receives the trigger signal as input (e.g., from trigger module 121) and provides a waveform for generating one or more tactile outputs in response to receiving the trigger signal (e.g., a waveform selected from a predefined set of waveforms assigned for use by waveform module 123, such as those described below with reference to FIG. 4C to FIG. 4D waveform described in more detail).
[0177] The mixer 125 receives waveforms as input (e.g., from the waveform module 123) and mixes the waveforms together. For example, when the mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform in a second channel that at least partially overlaps the first waveform), the mixer 125 outputs a combined waveform corresponding to the sum of the two or more waveforms. In some embodiments, the mixer 125 also modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the rest of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or reducing the scale of the other waveforms in the waveforms). In some cases, the mixer 125 selects one or more waveforms to remove from the combined waveform (e.g., when waveforms from more than three sources have been requested to be output simultaneously by the tactile output generator 167, the waveform from the oldest source is discarded).
[0178] Mixer 127 receives waveforms (e.g., combined waveforms from mixer 125) as input and modifies these waveforms. In some embodiments, compressor 127 reduces these waveforms (e.g., based on tactile output generator 167 ( Figure 1A ) or 357( Figure 3 ) such that the tactile output corresponding to these waveforms is reduced. In some embodiments, compressor 127 limits the waveforms, such as by imposing a predefined maximum amplitude on the waveforms. For example, compressor 127 reduces the amplitude of the waveform portion that exceeds the predefined amplitude threshold, while maintaining the amplitude of the waveform portion that does not exceed the predefined amplitude threshold. In some embodiments, compressor 127 reduces the dynamic range of the waveform. In some embodiments, compressor 127 dynamically reduces the dynamic range of the waveform so that the combined waveform remains within the performance specifications of tactile output generator 167 (e.g., force and / or movable mass displacement limits).
[0179] Low-pass filter 129 receives a waveform (e.g., a compressed waveform from compressor 127) as input and filters (e.g., smoothes) the waveform (e.g., removes or reduces high-frequency signal components in the waveform). For example, in some cases, compressor 127 includes extraneous signals (e.g., high-frequency signal components) in the compressed waveform that interfere with tactile output generation and / or exceed the performance specifications of tactile output generator 167 when generating tactile output based on the compressed waveform. Low-pass filter 129 reduces or removes such extraneous signals in the waveform.
[0180] Thermal controller 131 receives a waveform (e.g., a filtered waveform from low pass filter 129) as input and adjusts the waveform based on a thermal condition of device 100 (e.g., based on an internal temperature detected within device 100, such as the temperature of haptic feedback controller 161, and / or an external temperature detected by device 100). For example, in some cases, the output of haptic feedback controller 161 varies based on temperature (e.g., in response to receiving the same waveform, haptic feedback controller 161 generates a first tactile output when haptic feedback controller 161 is at a first temperature and generates a second tactile output when haptic feedback controller 161 is at a second temperature different from the first temperature). For example, the magnitude (or amplitude) of the tactile output may vary based on temperature. To reduce the effects of temperature changes, the waveform is modified (e.g., the amplitude of the waveform is increased or decreased based on temperature).
[0181] In some embodiments, the tactile feedback module 133 (e.g., trigger module 121) is coupled to the hardware input processing module 146. In some embodiments, Figure 1A The other input controllers 160 in the embodiment include a hardware input processing module 146. In some embodiments, the hardware input processing module 146 receives input from a hardware input device 145 (e.g., Figure 1A In some embodiments, the hardware input device 145 is any input device described herein, such as a touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 )、Touchpad 355( Figure 3 ), other input or control devices 116 ( Figure 1A ) or a strength-sensitive home button. In some embodiments, the hardware input device 145 consists of a strength-sensitive home button rather than a touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 ) or touchpad 355( Figure 3 ). In some embodiments, in response to input from a hardware input device 145 (e.g., an intensity-sensitive home button or a touch screen), the hardware input processing module 146 provides one or more trigger signals to the tactile feedback module 133 to indicate that a user input that meets predefined input criteria has been detected, such as an input corresponding to a home button "click" (e.g., a "press click" or a "release click"). In some embodiments, the tactile feedback module 133 provides a waveform corresponding to a home button "click" in response to an input corresponding to a home button "click", thereby simulating the tactile feedback of pressing a physical home button.
[0182] In some embodiments, the tactile output module includes a tactile feedback controller 161 (e.g., Figure 1A In some embodiments, the tactile feedback controller 161 is coupled to a plurality of tactile output generators, and selects one or more tactile output generators from the plurality of tactile output generators and sends a waveform to the selected one or more tactile output generators for generating a tactile output. In some embodiments, the tactile feedback controller 161 coordinates a tactile output request corresponding to activation of the hardware input device 145 and a tactile output request corresponding to a software event (e.g., a tactile output request from the tactile feedback module 133), and modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the remaining waveforms of the two or more waveforms (e.g., by increasing the scale of a particular waveform and / or reducing the scale of the remaining waveforms of the waveforms to prioritize the tactile output corresponding to the activation of the hardware input device 145 over the tactile output corresponding to the software event).
[0183] In some embodiments, such as Figure 1C As shown, the output of the haptic feedback controller 161 is coupled to the audio circuit of the device 100 (eg, Figure 1A 100) and provides the audio signal to the audio circuit of the device 100. In some embodiments, the tactile feedback controller 161 provides both a waveform for generating a tactile output and an audio signal for providing an audio output together with the generated tactile output. In some embodiments, the tactile feedback controller 161 modifies the audio signal and / or the waveform (for generating the tactile output) so that the audio output and the tactile output are synchronized (for example, by delaying the audio signal and / or the waveform). In some embodiments, the tactile feedback controller 161 includes a digital-to-analog converter for converting a digital waveform into an analog signal, which is received by the amplifier 163 and / or the tactile output generator 167.
[0184] In some embodiments, the tactile output module includes an amplifier 163. In some embodiments, the amplifier 163 receives a waveform (e.g., from the tactile feedback controller 161), amplifies the waveform, and then sends the amplified waveform to the tactile output generator 167 (e.g., the tactile output generator 167 ( Figure 1A ) or 357( Figure 3)). For example, amplifier 163 amplifies the received waveform to a signal level that meets the physical specifications of tactile output generator 167 (e.g., amplifies to the voltage and / or current required by tactile output generator 167 to generate a tactile output so that the signal sent to tactile output generator 167 generates a tactile output corresponding to the waveform received from tactile feedback controller 161) and sends the amplified waveform to tactile output generator 167. In response, tactile output generator 167 generates a tactile output (e.g., by shifting the movable mass back and forth in one or more dimensions relative to the neutral position of the movable mass).
[0185] In some embodiments, the tactile output module includes a sensor 169 coupled to the tactile output generator 167. The sensor 169 detects the state or state change (e.g., mechanical position, physical displacement, and / or movement) of the tactile output generator 167 or one or more components of the tactile output generator 167 (e.g., one or more moving components, such as a membrane, for generating a tactile output). In some embodiments, the sensor 169 is a magnetic field sensor (e.g., a Hall effect sensor) or other displacement and / or motion sensor. In some embodiments, the sensor 169 provides information (e.g., the position, displacement, and / or movement of one or more components in the tactile output generator 167) to the tactile feedback controller 161, and based on the information about the state of the tactile output generator 167 provided by the sensor 169, the tactile feedback controller 161 adjusts the waveform output from the tactile feedback controller 161 (e.g., the waveform sent to the tactile output generator 167, optionally via the amplifier 163).
[0186] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining the event to be recognized.
[0187] Figure 2A portable multifunction device 100 with a touch screen 112 according to some embodiments is shown. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more of these graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics will occur when the user interrupts contact with the one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic will not select the graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.
[0188] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.
[0189] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the strength of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.
[0190] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 167 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or contact intensity sensors (similar to the above referenced Figure 1A The memory 370 may include one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.
[0191] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the previously mentioned memory device. Each module in the above-mentioned modules corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned modules or programs (e.g., instruction sets) do not have to be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.
[0192] Attention is now turned to an embodiment of a user interface that is optionally implemented on, for example, portable multifunction device 100.
[0193] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 according to some embodiments is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:
[0194] One or more signal strength indicators 402 of one or more wireless communications such as cellular signals and Wi-Fi signals;
[0195] Time 404;
[0196] Bluetooth indicator 405;
[0197] Battery status indicator 406;
[0198] A tray 408 with icons for commonly used applications, such as:
[0199] o an icon 416 of the phone module 138 labeled “Phone”, the icon 416 optionally including an indicator 414 of the number of missed calls or voice messages;
[0200] o an icon 418 of the email client module 140 labeled “Mail”, the icon 418 optionally including an indicator 410 of the number of unread emails;
[0201] o An icon 420 labeled "Browser" of the browser module 147; and
[0202] o an icon 422 labeled “iPod” of a video and music player module 152 (also referred to as an iPod (trademark of Apple Inc.) module 152); and
[0203] Icons for other apps, such as:
[0204] o Icon 424 labeled “Message” of IM module 141;
[0205] o Icon 426 labeled “Calendar” of calendar module 148;
[0206] o Icon 428 labeled “Photos” of the image management module 144;
[0207] o An icon 430 labeled “Camera” of the camera module 143;
[0208] o Icon 432 labeled “Online Video” of the online video module 155;
[0209] o Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;
[0210] o an icon 436 labeled “Map” of the map module 154;
[0211] o Icon 438 labeled “Weather” of the Weather widget 149-1;
[0212] o Icon 440 labeled as “Clock” of the alarm clock widget 149-4;
[0213] o an icon 442 labeled “Fitness Support” of the fitness support module 142;
[0214] o an icon 444 labeled "Notepad" of the Notepad module 153; and
[0215] o An icon 446 of a settings application or module labeled “Settings” that provides access to settings for the device 100 and its various applications 136 .
[0216] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.
[0217] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touchpad 355) Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.
[0218] Although some of the examples below are given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the principal axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B 460 corresponds to 468 and 462 corresponds to 470) in contact with touch-sensitive surface 451 (e.g., Figure 4B Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and the display of a multi-function device ( Figure 4B When the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is separated, the device is used to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.
[0219] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.
[0220] Figure 5AAn exemplary personal electronic device 500 is shown. Device 500 includes a body 502. In some embodiments, device 500 may include a body 502 relative to devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in the foregoing. In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. As an alternative or in addition to touch screen 504, device 500 has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more strength sensors for detecting the intensity of contact (e.g., touch) applied. One or more strength sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of touch. The user interface of device 500 can respond to touch based on the intensity of the touch, which means that touches of different intensities can invoke different user interface operations on device 500.
[0221] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, filed on May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed on November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.
[0222] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.
[0223] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, the device 500 may include a reference Figure 1A , Figure 1B and Figure 3 Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. I / O portion 514 may be connected to a display 504, which may have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, I / O portion 514 may be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, input mechanism 508 is optionally a button.
[0224] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as GPS sensor 532, accelerometer 534, orientation sensor 540 (e.g., compass), gyroscope 536, motion sensor 538, and / or combinations thereof, all of which are operably connected to I / O portion 514.
[0225] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which when executed by one or more computer processors 516 may, for example, cause the computer processors to perform the techniques described below, including methods 800, 1000, 1200, and 1400 (Figures 8, 10, 12, and 14). A computer-readable storage medium may be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with an instruction execution system, apparatus, and device. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory such as flash memory, solid-state drives, and the like. The personal electronic device 500 is not limited to Figure 5B components and configurations, but may include other components or additional components in a variety of configurations.
[0226] As used herein, the term "indicative representation" refers to an optional feature of the apparatus 100, 300, and / or 500 ( Figure 1A , Figure 3 and FIG. 5A to FIG. 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.
[0227] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that a touch-screen display (e.g., Figure 1A A touch-sensitive display system 112 or Figure 4AIn some implementations of the touch screen 112 in FIG. 1 , a contact detected on the touch screen acts as a “focus selector” such that when an input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted in accordance with the detected input. In some implementations, the focus moves from one area of the user interface to another area of the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).
[0228] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.
[0229] Figure 5C Detecting multiple contacts 552A-552E on touch-sensitive display screen 504 using multiple intensity sensors 524A-524D is shown. Figure 5C Also included is an intensity graph that shows the current intensity measurements of intensity sensors 524A-524D relative to intensity units. In this embodiment, the intensity measurements of intensity sensors 524A and 524D are each 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are each 7 intensity units. In some implementations, the cumulative intensity is the sum of the intensity measurements of multiple intensity sensors 524A-524D, which in this embodiment is 32 intensity units. In some embodiments, each contact is assigned a corresponding intensity, which is a portion of the cumulative intensity. Figure 5DThe cumulative strength assigned to contacts 552A-552E based on their distance from the center of force 554 is shown. In this embodiment, each of contacts 552A, 552B, and 552E is assigned a contact strength of 8 strength units of the cumulative strength, and each of contacts 552C and 552D is assigned a contact strength of 4 strength units of the cumulative strength. More generally, in some embodiments, each contact j is assigned a corresponding strength Ij, which is a portion of the cumulative strength A, according to a predefined mathematical function Ij=A·(Dj / ΣDi), where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i=1 to the last) from the center of force. The reference may be performed using an electronic device similar to or equivalent to device 100, 300, or 500. Figure 5C-5D In some embodiments, the characteristic intensity of the contact is based on one or more intensities of the contact. In some embodiments, the intensity sensor is used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included in the Figure 5C-5D To assist the readers.
[0230] In some embodiments, a portion of a gesture is identified for determining characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position and reaches an end position, where the contact strength increases. In this example, the characteristic strength of the contact at the end position is optionally based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm is optionally applied to the strength of the swipe contact before determining the characteristic strength of the contact. For example, a smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or depressions in the strength of the swipe contact to achieve the purpose of determining the characteristic strength.
[0231] The intensity of contact on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, when a contact having a characteristic intensity lower than the light press intensity threshold (e.g., and higher than a nominal contact detection intensity threshold, contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.
[0232] An increase in contact feature intensity from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact feature intensity from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact feature intensity from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting contact on the touch surface. A decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.
[0233] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the strength of the corresponding contact increases to above the press input strength threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and the strength of the contact subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., an "up stroke" of the corresponding press input).
[0234] Figure 5E-Figure 5HDetection of a gesture is shown, the gesture comprising contact 562 with an intensity from below Figure 5E The light press intensity threshold in L ”) increases to a strength higher than Figure 5H The deep compression intensity threshold in D ”). On the displayed user interface 570 including application icons 572A-572D displayed in the predefined area 574, when the cursor 576 is displayed above the application icon 572B corresponding to application 2, a gesture performed using contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch-sensitive display 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines whether the intensity of the contact 562 is within the deep press intensity threshold (for example, “IT D ”) reaches a peak above. Contact 562 is maintained on touch-sensitive surface 560. In response to detecting the gesture, and according to the intensity increasing to a deep press intensity threshold (e.g., “IT D ”) above contact 562, displays scaled representations 578A-578C (e.g., thumbnails) of documents recently opened for application 2, such as Figure 5F-5H In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in the Figure 5E-Figure 5H To assist the readers.
[0235] In some embodiments, the display of representations 578A-578C includes animation. For example, representation 578A is initially displayed near application icon 572B, such as Fig. 5F As the animation progresses, representation 578A moves upward and representation 578B is displayed near application icon 572B, as shown. Figure 5G Then, representation 578A moves upward, 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown in FIG. Figure 5H 578A-578C form an array above icon 572B. In some embodiments, the animation progresses according to the intensity of contact 562, such as Figure 5F-5G , where representations 578A-578C appear and increase as the strength of contact 562 approaches a deep press strength threshold (eg, “IT D ”) increases and moves upward. In some embodiments, the intensity according to which the animation progresses is the characteristic intensity of the contact. The reference may be performed using an electronic device similar to or equivalent to device 100, 300, or 500. Figure 5E-Figure 5H The operation described.
[0236] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below the hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upward stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).
[0237] For ease of explanation, optionally, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is triggered in response to detecting any of the following: contact intensity increasing to above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing to below the press input intensity threshold, and / or contact intensity decreasing to below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.
[0238] As used herein, an "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched (e.g., become open) on the device. In some embodiments, a downloaded application becomes an installed application using an installer that extracts program portions from a downloaded software package and integrates the extracted portions with the operating system of a computer system.
[0239] As used herein, the term "open application" or "executing application" refers to a software application that has retained state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or executing application is optionally any of the following types of applications:
[0240] The active application currently displayed on the display of the device on which the application is being used;
[0241] Background applications (or background processes) that are not currently displayed but whose processes or processes are being processed by one or more processors; and
[0242] - A suspended or dormant application that is not running but has state information stored in memory (volatile and non-volatile, respectively) and can be used to resume execution of the application.
[0243] As used herein, the term "closed application" refers to a software application that does not have retained state information (e.g., the state information of a closed application is not stored in the memory of the device). Therefore, closing an application includes stopping and / or removing the application process of the application and removing the state information of the application from the memory of the device. Generally speaking, when in a first application, opening a second application does not close the first application. When the second application is displayed and the first application stops being displayed, the first application becomes a background application.
[0244] Attention is now turned to embodiments of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.
[0245] Figure 6 Exemplary devices connected via one or more communication channels to participate in a transaction according to some embodiments are shown. One or more exemplary electronic devices (e.g., devices 100, 300, and 500) are configured to optionally detect input (e.g., specific user input, NFC field), and optionally transmit payment information (e.g., using NFC). The one or more electronic devices optionally include NFC hardware and are configured to support NFC.
[0246] The electronic device (e.g., devices 100, 300, and 500) is optionally configured to store payment account information associated with each of one or more payment accounts. The payment account information includes, for example, one or more of the following: a person's name or company name, a billing address, a login name, a password, an account number, an expiration date, a security code, a phone number, a bank associated with the payment account (e.g., an issuing bank), and a card network identifier. In some embodiments, the payment account information includes an image, such as a photo of a payment card (e.g., a photo taken by the device and / or received by the device). In some embodiments, the electronic device receives user input including at least some payment account information (e.g., receiving a credit card number, a debit card number, an account number or a shopping card number and an expiration date entered by the user). In some embodiments, the electronic device detects at least some payment account information from an image (e.g., of a payment card captured by a camera sensor of the device). In some embodiments, the electronic device receives at least some payment account information from another device (e.g., another user device or a server). In some embodiments, the electronic device receives payment account information from a server associated with another service (e.g., an application for renting or selling audio files and / or video files) for which the user or user device's account has previously made a purchase or for which payment account data has been identified.
[0247] In some embodiments, a payment account is added to an electronic device (e.g., devices 100, 300, and 500) so that the payment account information is securely stored on the electronic device. In some embodiments, after a user initiates such a process, the electronic device transmits the payment account information to a transaction coordination server, which then communicates with a server operated by the account's payment network (e.g., a payment server) to ensure the validity of the information. The electronic device is optionally configured to receive a script from the server that allows the electronic device to program the account's payment information onto a secure element.
[0248] In some embodiments, communication between devices 100, 300, and 500 facilitates transactions (e.g., general transactions or specific transactions). For example, a first electronic device (e.g., 100) may be used as a configuration device or a management device, and may send new or updated payment account data (e.g., information about a new account, updated information about an existing account, and / or a warning about an existing account) to a second electronic device (e.g., 500). As another example, a first electronic device (e.g., 100) may send data to a second electronic device, wherein the data reflects information about a payment transaction facilitated at the first electronic device. The information optionally includes one or more of the following: the payment amount, the account used, the time of purchase, and whether to change the default account. The second device (e.g., 500) optionally uses such information to update the default payment account (e.g., based on a learning algorithm or explicit user input).
[0249] Electronic devices (e.g., 100, 300, 500) are configured to communicate with each other via any of a variety of networks. For example, the devices communicate using a Bluetooth connection 608 (e.g., including a traditional Bluetooth connection or a Bluetooth low energy connection) or using a WiFi network 606. Communications between user devices are optionally adjusted to reduce the possibility of improper sharing of information between devices. For example, communication regarding payment information requires that the communicating devices be paired (e.g., associated with each other via explicit user interaction) or associated with the same user account.
[0250] In some embodiments, an electronic device (e.g., 100, 300, 500) is used to communicate with a point of sale (POS) payment terminal 600, which optionally supports NFC. The communication is optionally carried out using various communication channels and / or technologies. In some examples, an electronic device (e.g., 100, 300, 500) uses an NFC channel 610 to communicate with a payment terminal 600. In some embodiments, the payment terminal 600 uses a peer-to-peer NFC mode to communicate with an electronic device (e.g., 100, 300, 500). An electronic device (e.g., 100, 300, 500) is optionally configured to transmit a signal to the payment terminal 600, the signal including payment information of a payment account (e.g., a default account or an account selected for a specific transaction).
[0251] In some embodiments, continuing the transaction includes transmitting a signal including payment information of an account (such as a payment account). In some embodiments, continuing the transaction includes reconfiguring the electronic device (e.g., 100, 300, 500) to respond as a contactless payment card (such as an NFC-enabled contactless payment card) and then transmitting the credentials of the account via NFC to, for example, a payment terminal 600. In some embodiments, after transmitting the credentials of the account via NFC, the electronic device is reconfigured not to respond as a contactless payment card (e.g., authorization is required before being reconfigured again to respond as a contactless payment card via NFC).
[0252] In some embodiments, the generation and / or transmission of the signal is controlled by a security element in an electronic device (e.g., 100, 300, 500). Before issuing payment information, the security element optionally requires specific user input. For example, the security element optionally requires: detecting the electronic device being worn, detecting button presses, detecting password input, detecting touch, detecting one or more option selections (e.g., option selections received when interacting with an application), detecting fingerprint signatures, detecting voice or voice commands, and / or detecting gestures or movements (e.g., rotation or acceleration). In some embodiments, if a communication channel (e.g., NFC communication channel) with another device (e.g., payment terminal 600) is established within a defined time period from when the input is detected, the security element issues payment information so that it is transmitted to another device (e.g., payment terminal 600). In some embodiments, the security element is a hardware component that controls the release of security information. In some embodiments, the security element is a software component that controls the release of security information.
[0253] In some embodiments, the protocol associated with transaction participation depends on, for example, the device type. For example, the conditions for generating and / or transmitting payment information may be different for a wearable device (e.g., device 500) and a phone (e.g., device 100). For example, a generation condition and / or a transmission condition for a wearable device includes detecting that a button has been pressed (e.g., after security verification), while a corresponding condition for a phone does not require a button press, but requires detection of a specific interaction with an application. In some embodiments, conditions for transmitting and / or publishing payment information include receiving specific input on each of a plurality of devices. For example, the release of payment information optionally requires detection of a fingerprint and / or password at a device (e.g., device 100), as well as detection of a mechanical input (e.g., a button press) on another device (e.g., device 500).
[0254] The payment terminal 600 optionally uses the payment information to generate a signal for transmission to the payment server 604 to determine whether the payment is authorized. The payment server 604 optionally includes any device or system configured to receive the payment information associated with the payment account and determine whether the proposed purchase is authorized. In some embodiments, the payment server 604 includes a server of the issuing bank. The payment terminal 600 communicates directly with the payment server 604 or indirectly via one or more other devices or systems (e.g., a server of the acquiring bank and / or a server of the card network).
[0255] Payment server 604 optionally uses at least some of the payment information to identify the user account from a database of user accounts (e.g., 602). For example, each user account includes payment information. The account is optionally located by locating an account with specific payment information that matches the information from the POS communication. In some embodiments, payment is rejected when the provided payment information is inconsistent (e.g., the expiration date does not correspond to the credit card number, the debit card number, or the shopping card number does not correspond) or when no account includes payment information that matches the information from the POS communication.
[0256] In some embodiments, the user account data further identifies one or more constraints (e.g., credit limit); current or previous balances; previous transaction dates, locations, and / or amounts; account status (e.g., active or frozen), and / or authorization instructions. In some embodiments, the payment server (e.g., 604) uses such data to determine whether to authorize a payment. For example, the payment server denies payment when the purchase amount added to the current balance would cause the account limit to be exceeded, when the account is frozen, when the previous transaction amount exceeds a threshold, or when the number or frequency of previous transactions exceeds a threshold.
[0257] In some embodiments, the payment server 604 responds to the POS payment terminal 600 with an indication of whether the proposed purchase is authorized or rejected. In some embodiments, the POS payment terminal 600 transmits a signal to the electronic device (e.g., 100, 300, 500) to identify the result. For example, when the purchase is authorized (e.g., via a transaction coordination server that manages a transaction application on a user device), the POS payment terminal 600 sends a receipt to the electronic device (e.g., 100, 300, 500). In some cases, the POS payment terminal 600 presents an output (e.g., a visual output or an audio output) indicating the result. The payment may be transmitted to the merchant as part of the authorization process, or may be transmitted later.
[0258] In some embodiments, the electronic device (e.g., 100, 300, 500) participates in a transaction that is completed without involving the POS payment terminal 600. For example, upon detecting that the mechanical input has been received, the secure element in the electronic device (e.g., 100, 300, 500) releases the payment information to allow an application on the electronic device to access the information (e.g., and transmit the information to a server associated with the application).
[0259] In some embodiments, the electronic device (e.g., 100, 300, 500) is in a locked state or an unlocked state. In the locked state, the electronic device is powered on and is operable, but it is prevented from performing a predefined set of operations in response to user input. The predefined set of operations optionally includes navigating between user interfaces, activating or deactivating a predefined set of functions, and activating or deactivating certain applications. The locked state can be used to prevent unconscious or unauthorized use of some functions of the electronic device, or to activate or deactivate some functions on the electronic device. In the unlocked state, the electronic device 100 is powered on and is operable, and is not prevented from performing at least a portion of a predefined set of operations that cannot be performed when in the locked state.
[0260] When the device is in the locked state, the device is said to be locked. In some embodiments, the device in the locked state optionally responds to a restricted set of user inputs, including inputs corresponding to attempts to transition the device to the unlocked state or inputs corresponding to shutting down the device.
[0261] In some examples, a security element (e.g., 115) is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms so that the device cannot access securely stored data without correct authentication information from the user of the device. Keeping securely stored data in a secure element separate from other storage devices on the device prevents access to securely stored data, even if other storage locations on the device are compromised (e.g., by malicious code or other attempts to compromise information stored on the device). In some embodiments, the security element provides (or publishes) payment information (e.g., account number and / or transaction-specific dynamic security code). In some embodiments, the security element provides (or publishes) payment information in response to receiving authorization, and the authorization is, for example, user authentication (e.g., fingerprint authentication; password authentication; when the device is in an unlocked state, and optionally when the device has been continuously on the user's wrist since the device was unlocked by providing authentication credentials to the device, a double press of the hardware button is detected, wherein the device is determined to be continuously present on the user's wrist by periodically checking that the device is in contact with the user's skin). For example, the device detects a fingerprint at a fingerprint sensor of the device (e.g., a fingerprint sensor integrated into a button). The device determines whether the fingerprint is consistent with the registered fingerprint. Based on the determination that the fingerprint is consistent with the registered fingerprint, the security element provides (or releases) the payment information. Based on the determination that the fingerprint is inconsistent with the registered fingerprint, the security element abandons providing (or releasing) the payment information.
[0262] Attention is now turned to embodiments of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.
[0263] 7A to 7AD An exemplary user interface for providing an indication of an error condition during biometric authentication according to some examples is shown. The user interface in these figures is used to illustrate the process described below, including FIG. 8A to FIG. 8B process.
[0264] Fig. 7A An electronic device 700 (e.g., portable multifunction device 100, device 300, or device 500) is shown. 7A to 7ADIn the illustrated exemplary example, electronic device 700 is a smart phone. In other examples, electronic device 700 may be a different type of electronic device, such as a tablet (e.g., electronic device 900). Electronic device 700 includes display 702, one or more input devices (e.g., a touch screen of display 702, buttons 704, and a microphone), a wireless communication radio component, and biometric sensor 703. Electronic device 700 includes biometric sensor 703. In some examples, biometric sensor 703 includes one or more biometric sensors, which may include a camera, such as a depth camera (infrared camera), a thermal imaging camera, or a combination thereof. In some examples, the biometric sensor 703 includes a biometric sensor (e.g., a facial recognition sensor), such as those described in U.S. Serial No. 14 / 341,860, entitled “Overlapping Pattern Projector,” filed on July 14, 2014, U.S. Publication No. 2016 / 0025993, and U.S. Serial No. 13 / 810,451, entitled “Scanning Projects and Image Capture Modules For 3D Mapping,” which are hereby incorporated by reference in their entirety for any purpose. In some examples, the biometric sensor 703 includes one or more fingerprint sensors (e.g., a fingerprint sensor integrated into a button). In some examples, the electronic device 700 also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the biometric feature (e.g., face) during the capture of biometric data of the biometric feature by the biometric sensor 703. In some examples, the electronic device 700 includes multiple cameras separate from the biometric sensor 703. In some examples, the electronic device 700 includes only one camera separate from the biometric sensor 703.
[0265] exist Fig. 7AAt , the user learns from notification 708 that she has received a message from John Appleseed. The user wishes to view the restricted content of notification 708 (e.g., the message from John Appleseed), but is unable to do so because electronic device 700 is currently in a locked state. Electronic device 700 displays a locked state user interface (UI) with lock icon 706, which provides an indication that electronic device 700 is in a locked state. Viewing the restricted content of notification 708 requires successful authentication (e.g., determining that information (or data) about a biometric feature obtained using biometric sensor 703 corresponds to (or matches) stored authorization credentials). In order to view the restricted content of notification 708, the user lifts (or raises) electronic device 700 (e.g., from a substantially horizontal orientation to an orientation of the device, as in Fig. 7A 708 ). The electronic device 700 detects a change in the orientation of the electronic device 700 and, in response, initiates biometric authentication. In some examples, after initiating biometric authentication, the electronic device 700 determines that the biometric authentication is successful. In some examples, upon determining that the biometric authentication is successful, the electronic device 700 transitions from a locked state to an unlocked state and displays the restricted content of the notification 708.
[0266] After initiating biometric authentication (e.g., before successful authentication), the electronic device 700 determines whether the biometric sensor 703 detects a face. Figure 7B At , upon determining that a face is detected, the electronic device 700 displays an authentication glyph 710, which includes a plurality of spherically rotating rings. The authentication glyph 710 provides an indication that biometric authentication is being performed. In some examples, the electronic device 700 displays an animation of a lock icon 706 that is deformed into the authentication glyph 710. In some examples, upon determining that a face is not detected using the biometric sensor 703, the electronic device 700 remains locked and the authentication glyph 710 is not displayed.
[0267] After detecting the presence of the face, the electronic device 700 determines that the authentication is unsuccessful due to failure to obtain sufficient information about the user's face using the biometric sensor 703. Specifically, if Figure 7BAs shown, the biometric sensor 703 is located outside the acceptable distance range 712 (e.g., above the maximum threshold range), resulting in an inability to obtain sufficient information about the user's face. Upon determining that biometric authentication is unsuccessful because the user's face is outside the acceptable distance range 712, the electronic device 700 maintains the device in a locked state and does not display the restricted content of the notification 708. In some examples, the electronic device 700 maintains the device in a locked state and does not display the restricted content of the notification 708 upon determining that the authentication is unsuccessful and that there is no error condition. In some examples, upon determining that the authentication is unsuccessful (e.g., because the captured biometric information does not match an authorized biometric information profile (e.g., stored authorization credentials)) and that there is no error condition (e.g., there is no state that prevents the capture of sufficient biometric information), the electronic device 700 remains in a locked state and automatically retries the biometric authentication. In some examples, when retrying the biometric authentication, the electronic device 700 continues in the locked state. Figure 7B An authentication glyph 710 is shown in FIG.
[0268] like FIG. 7C to FIG. 7G As shown, when it is determined that the biometric authentication is unsuccessful because the user's face is outside the acceptable distance range 712, the electronic device 700 displays an animation in which the authentication glyph 710 is transformed into an error indication 714A, so that the error indication 714A replaces the display of the authentication glyph 710. Figure 7G At this point, the electronic device 700 displays an error indication 714A, which prompts the user to take action to correct the error condition below the error indication 714A. Specifically, the error indication 714A prompts the user to move her face closer to the biometric sensor 703. The error indication 714A also suggests to the user that the user's face is too far away from the biometric sensor 703, which is the reason for the error indication 714A. As long as the user's face is outside the acceptable distance range 712, the electronic device 700 will continue to determine that there is an error indication 714A. Upon determining that the error indication 714A still exists, the electronic device 700 does not attempt to retry the biometric authentication. It should be noted that the electronic device 700 is in communication with Fig. 7A The error indication 714A is displayed at a position consistent with the position of the lock icon 706 in the display 702. In addition, the electronic device 700 displays the error indication 714A on a portion of the display 702 adjacent to the biometric sensor 703 to suggest to the user that the error indication 714A is associated with (or corresponds to) the biometric sensor 703.
[0269] like Figure 7H As shown, after being prompted to correct the error indication 714A, the user moves her face closer to the biometric sensor 703 so that the user's face is within the acceptable distance range 712. Figure 7HAt, the electronic device determines that the error indication 714A no longer exists. When determining that the error indication 714A no longer exists, the electronic device 700 enables biometric authentication on the device and automatically retries the biometric authentication using the biometric sensor 703.
[0270] In response to automatically retrying biometric authentication, the electronic device 700 displays an error indication 714A with a flashing effect (e.g., animating the error indication so that one or more portions of the error indication move left or right to create an effect that the error indication appears to be flashing) to indicate that the electronic device 700 is attempting to biometrically authenticate the user again. FIG. 7H to FIG. 7L An animation of error indication 714A with a flashing effect is depicted. In some examples, instead of displaying error indication 714A with a flashing effect, electronic device 700 displays (e.g., replaces the display of error indication 714A with) authentication glyph 710 to indicate that electronic device 700 is attempting to biometrically authenticate the user again. Thus, in some examples, electronic device 700 displays an animation of authentication glyph 710 morphing into lock icon 706, rather than an animation of error indication 714A morphing into lock icon 706.
[0271] exist Figure 7L At , after retrying the biometric authentication, the electronic device 700 successfully authenticates the user in a biometric manner. In response to the successful biometric authentication, the electronic device 700 transitions the device from the locked state to the unlocked state. When transitioning from the locked state to the unlocked state, the electronic device 700 displays an animation of the error indication 714A transforming into the lock icon 706, such as Figures 7L to 7N After displaying the animation of error indication 714A transforming into lock icon 706, electronic device 700 displays the animation of lock icon 706 transforming into unlock icon 716, as shown in FIG. FIG. 7N to FIG. 7O As shown. Unlock icon 716 provides an indication that electronic device 700 is in an unlocked state. Fig.7O As shown, in response to successful biometric authentication, the electronic device 700 displays limited content of the notification 708 (eg, “Hey, is our meeting still going on?”).
[0272] exist Figure 7P Instead of the above relative to Figure 7B When determining that the user's face is outside the acceptable distance range 712 as discussed above, the electronic device 700 determines that biometric authentication is not available on the device. When determining that biometric authentication is not available, the electronic device 700 Figure 7PAn error indication 714B is displayed in the device, which provides an indication that biometric authentication is currently not available on the device. Biometric authentication may be unavailable for various reasons, including that biometric authentication has failed more than a predetermined number of times (e.g., 5, 10, 15) since the last successful authentication.
[0273] Since biometric authentication is not available, the user must use an alternative method to authenticate the user. For example, the user can authenticate by entering a password at the electronic device 700. Figure 7P When error indication 714B is displayed in the electronic device 700, input 720 is received at error indication 714B.
[0274] exist Figure 7Q In response to receiving input 720 at error indication 714B, electronic device 700 displays a passcode input UI 722A having multiple input affordances for entering a password (or password).
[0275] In some examples, instead of determining that authentication was successful by retrying biometric authentication, as described above with respect to FIG. 7L to FIG. 7O As discussed, the electronic device 700 determines that the authentication is unsuccessful. In some examples, when it is determined that the authentication is unsuccessful, the electronic device 700 remains locked and displays Figure 7R The lock icon 706 in the embodiment alternates between different positions to simulate a "shaking" effect. The shaking animation provides the user with an indication that the biometric authentication has failed and the electronic device 700 remains in a locked state.
[0276] After determining that the authentication was not successful, the user can perform an action at the electronic device 700 to trigger a retry of the biometric authentication. Figure 7S At 724, the user triggers a retry of the biometric authentication by swiping upward from an area near the bottom edge of the display 702. The electronic device 700 receives the input 724 and, in response, retries the biometric authentication. In some examples, after retrying the biometric authentication, the electronic device 700 determines that the authentication is successful. In some examples, upon determining that the authentication is successful by retrying the biometric authentication, the electronic device 700 transitions from the locked state to the unlocked state.
[0277] exist FIG. 7S to FIG. 7T At this point, the electronic device 700 determines that the biometric authentication is not successfully authenticated by retrying it in response to the input 724. When it is determined that the biometric authentication is not successfully authenticated by retrying it, the electronic device 700 displays Figure 7T The password in UI 722B may be entered and / or remain locked.
[0278] exist Figure 7UAt the password input UI 722B, the electronic device 700 determines that the authentication is successful by retrying the biometric authentication at the password input UI 722B. When the authentication is determined to be successful, the electronic device changes from the locked state to the unlocked state, such as Figure 7U to Figure 7W In some examples, Figure 7U At 722B, the electronic device determines that the biometric authentication is not successfully authenticated by retrying it at the password input UI 722B. In some examples, when making this determination, the electronic device remains locked.
[0279] Figures 7X to 7AD 1 shows various error conditions that the electronic device 700 may detect when attempting to biometrically authenticate a user. Figure 7G When displaying the error indication 714A, the electronic device 700 may display any one of the error indications described below (eg, error indications 714C-I). Figures 7X to 7AD The electronic device 700 is also depicted instructing the user (e.g., via error indication 714C-I) to take action to correct the detected error condition so that the electronic device 700 can retry biometrically authenticating the user.
[0280] exist Figure 7X , the user's face is positioned too close to the biometric sensor 703. Therefore, the electronic device 700 determines that the user's face is positioned outside the acceptable distance range 712 (e.g., below the minimum threshold range). Upon determining that the user's face is positioned outside the acceptable distance range 712, the electronic device 700 displays an error indication 714C that prompts the user to move her face away from the biometric sensor 703. The error indication 714C also provides an indication of the cause of the error condition (e.g., an indication that the user's face is too close to the biometric sensor 703).
[0281] exist Figure 7Y At , the user's hand covers the biometric sensor 703. Therefore, the electronic device 700 determines that the object (e.g., the user's hand) covers the biometric sensor 703, so that the sensor cannot obtain any information about the user's face. Upon determining that the object covers the biometric sensor 703, the electronic device 700 displays an error indication 714D, which prompts the user to move her hand away from the biometric sensor 703. The error indication 714D also provides an indication of the cause of the error condition (e.g., an indication that the biometric sensor 703 is covered).
[0282] exist Figure 7Z, the user is not looking at the electronic device 700. Therefore, the electronic device 700 determines that the user's eyes are not looking at the device. When determining that the user's eyes are not looking at the device, the electronic device 700 displays an error indication 714E, which prompts the user to look at the device to correct the error condition. The error indication 714E also provides an indication of the cause of the error condition (e.g., an indication that the user is not looking at the device).
[0283] exist Figure 7AA , the user's face is within field of view 728, but the user is wearing a hat. Therefore, the electronic device 700 determines that a portion of the user's face is obscured (or blocked). For example, the electronic device 700 uses the biometric sensor 703 to obtain partial information about the user's face, where the partial information is below a threshold amount required for comparison with the stored authorization credentials. Upon determining that a portion of the user's face is obscured, the electronic device 700 displays an error indication 714F, which prompts the user to remove the hat. The error indication 714F also provides an indication of the cause of the error condition (e.g., an indication that a portion of the user's face is obscured).
[0284] exist Figure 7AB , the user's face is outside the field of view 728 of the biometric sensor 703. Therefore, the electronic device 700 determines that the user's face is outside the field of view 728 of the biometric sensor 703. In some examples, the user's face is outside the field of view 728 when more than a threshold portion of the face is outside the field of view. In some examples, the user's face is outside the field of view 728 when no face is detected within the field of view. Upon determining that the user's face is outside the field of view 728, the electronic device 700 displays an error indication 714G, which prompts the user to move her face into the field of view 728. The error indication 714G also provides an indication of the cause of the error condition (e.g., an indication that the user's face is outside the field of view 728).
[0285] exist Figure 7AC , the user's face is within the field of view 728, but is turned away from the biometric sensor 703. Therefore, the electronic device 700 determines that the user's face is turned away from the biometric sensor 703. Upon determining that the user's face is turned away from the biometric sensor 703, the electronic device 700 displays an error indication 714H, which prompts the user to turn her face toward the sensor. The error indication 714H also provides an indication of the cause of the error condition (e.g., an indication that the user's face is turned away from the biometric sensor 703).
[0286] exist Figure 7AD, the user's face is properly positioned within the field of view and acceptable distance range of the biometric sensor 703. However, the lighting conditions of the environment in which the user is located are not suitable for biometric authentication. Specifically, the amount of light is so great that it interferes with biometric authentication. Therefore, the electronic device 700 determines (e.g., via one or more ambient light sensors) that the amount of light exceeds a predefined threshold. Upon determining that the amount of light exceeds the threshold, the electronic device 700 displays an error indication 714I, prompting the user to seek improved lighting conditions using a smaller amount of light. The error indication 714I also provides an indication of the cause of the error condition (e.g., an indication that the light conditions are not suitable for biometric authentication).
[0287] FIG. 8A to FIG. 8B is a flowchart illustrating a method for providing an indication of an error condition during biometric authentication according to some examples. Method 800 is performed at an electronic device (e.g., 100, 300, 500, 700) having a display (e.g., 702) and one or more input devices (e.g., an accelerometer (e.g., 168), a touch screen of the display (e.g., 702)). In some examples, the electronic device includes one or more biometric sensors (e.g., a fingerprint sensor, a contactless biometric sensor (e.g., a biometric sensor that does not require physical contact, such as a thermal or optical facial recognition sensor), an iris scanner). In some examples, one or more biometric sensors include one or more cameras. Some operations in method 800 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.
[0288] As described below, method 800 provides an intuitive way to provide an indication of an error condition during biometric authentication. The method reduces the cognitive burden on a user to perform biometric authentication, thereby creating a more efficient human-computer interface. For battery-powered computing devices, users can perform biometric authentication more quickly and efficiently, saving power and increasing the interval between battery charges.
[0289] An electronic device (e.g., 100, 300, 500, 700) receives (802) a request to perform an operation requiring authentication (e.g., biometric authentication) via one or more input devices (e.g., an accelerometer (e.g., 168), a touch screen of a display (e.g., 702). In some examples, the request to perform the operation requiring authentication includes a request to unlock the device (e.g., a swipe at a predefined location). In some examples, the request is triggered by lifting the device from a substantially horizontal position.
[0290] In response to a request (804) to perform an operation requiring authentication (e.g., biometric authentication) and in accordance with (806) determining that the authentication (e.g., biometric authentication) is successful, the electronic device performs an operation. In some examples, the authentication is successful when the user input (e.g., data obtained from one or more biometric sensors corresponding to the user's biometric features (e.g., face, fingers) , A password) corresponds to (e.g., matches) an authorization credential (e.g., a registered fingerprint, face, or password). In some examples, when the user input matches the authorization credential, the user input corresponds to the authorization credential.
[0291] In response to (804) a request to perform an operation requiring authentication (e.g., biometric authentication) and based on (808) determining that the authentication (e.g., biometric authentication) is unsuccessful and a set of error condition criteria are satisfied (e.g., an error condition exists), the electronic device (e.g., 100, 300, 500, 700) displays (810) an indication of an error condition (e.g., 714A-I) (e.g., in the set of error condition criteria) on a display (e.g., 702) and abandons (816) performing the operation. The indication includes (812) information about the cause of the error condition. In some examples, the authentication is unsuccessful when the user input (e.g., data obtained from one or more biometric sensors corresponding to the user's biometric features (e.g., face, finger)) does not correspond to (e.g., matches) an authorized credential (e.g., a registered fingerprint, face, or password). In some examples, the user input does not correspond to the authorized credential when the user input does not match the authorized credential. In some examples, the set of error condition criteria includes only one criterion. Displaying an indication of an error condition provides feedback to the user regarding the current state of the device (e.g., an error condition prevents successful biometric authentication) and prompts the user to take further action to correct the error condition. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently. In addition, when biometric authentication fails and an error condition is detected, abandoning execution of the operation enhances security and reduces instances of multiple resource-intensive retries of biometric authentication that may have failed due to the error condition. Providing improved security enhances the operability of the device and makes the user-device interface more efficient (e.g., by restricting unauthorized access), which in turn reduces power usage and extends the battery life of the device by limiting the execution of the constrained operations.
[0292] In some examples, in response to a request (804) to perform an operation requiring authentication and based on a determination (826) that the authentication (e.g., biometric authentication) was unsuccessful and the set of error condition criteria were not met, the electronic device (e.g., 100, 300, 500, 700) abandons (828) displaying an indication of the error condition on a display (e.g., 702) and abandons (830) performing the operation.
[0293] In some examples, the indication of the error condition (e.g., 714A-I) includes (814) an indication (e.g., a visual indication (e.g., a graphic or text)) of a user action that can be performed to correct the error condition (e.g., for a subsequent authentication attempt). In some examples, the indication of the user action indicates how to correct the error condition for a subsequent authentication attempt. Displaying the indication of the user action that can be performed to correct the error condition provides feedback to the user about what action to take so that the user can be biometrically authenticated in a subsequent authentication attempt. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some examples, the indicator is not displayed during biometric authentication.
[0294] In some examples, the indication of the error condition (e.g., 714A-I) includes information about the cause of the error condition (e.g., an indication of a user action and / or device state, a visual indication (e.g., a graphic or text)). Displaying an indication of the cause of the error condition provides feedback to the user about what action to take so that the user can be biometrically authenticated in a subsequent authentication attempt. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some examples, the indicator is not displayed during biometric authentication.
[0295] In some examples, the set of error condition criteria includes requirements that are met when a biometric feature (e.g., fingerprint, face) of a first type (e.g., corresponding to a type of authorized biometric feature) is detected using one or more biometric sensors (e.g., 703) of the electronic device. In some examples, if a potentially valid biometric feature is not detected (e.g., indicating that the user is not currently engaged with the device), an indication of the error condition (e.g., 714A-I) is not displayed. Forgoing displaying an indication of the error condition when no biometric feature is detected prevents possible confusion for the user because the user may not intend to perform biometric authentication if no biometric feature is detected. Therefore, forgoing displaying an indication in this case makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0296] In some examples, based on determining that authentication (e.g., biometric authentication) is successful, the electronic device (e.g., 100, 300, 500, 700) forgoes displaying an indication of an error condition (e.g., 714A-I) on a display (e.g., 702).
[0297] In some examples, after displaying an indication of an error condition (e.g., 714A-I) and based on a determination that the set of error condition criteria continue to be met, the electronic device (e.g., 100, 300, 500, 700) abandons (818) attempting (and optionally, disables further attempts) biometric authentication on the electronic device (e.g., biometric authentication functionality cannot be used on the device while the set of error conditions are met). In some examples, after displaying an indication of an error condition and based on a determination that the set of error condition criteria are no longer met, the electronic device enables (822) retrying biometric authentication on the electronic device (e.g., the error condition no longer exists (e.g., has been corrected (e.g., due to a user taking action to correct the error condition)). Automatically retrying biometric authentication when the set of error condition criteria is no longer met allows a user to quickly attempt to biometrically authenticate themselves without the user explicitly requesting biometric authentication. Performing optimization operations when a set of conditions have been met and no further user input is required enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0298] In some examples, after displaying an indication of the error condition and in response to determining that the set of error condition criteria is no longer met, the electronic device retries (824) authentication (e.g., biometric authentication) (e.g., automatically retrying authentication). In some examples, retrying authentication includes attempting to match biometric information obtained by one or more biometric sensors with authorized credentials (e.g., stored data that has been authorized for biometric authentication). In some examples, after (or in response to) receiving input for correcting the error condition, a determination that the error condition is not met occurs. In some examples, retrying authentication occurs (or occurs only) based on determining that the error condition is not met due to detecting a user input that causes the error condition to be unsatisfied.
[0299] In some examples, after determining that the set of error condition criteria is no longer met (e.g., detecting that the error condition has been corrected), the electronic device (e.g., 100, 300, 500, 700) receives an input corresponding to a request to retry authentication via one or more input devices (e.g., 724, 726). In some examples, the input is a touch gesture input (e.g., a tap, a swipe (e.g., an upward swipe)) or activation of a hardware button (e.g., a power button). In some examples, in response to receiving an input corresponding to a request to retry authentication, the electronic device retries authentication (e.g., biometric authentication) (e.g., automatically retrying authentication). In some examples, retrying authentication includes attempting to match biometric information obtained through one or more biometric sensors with authorized credentials (e.g., stored data that has been authorized for biometric authentication). In some examples, retrying authentication includes using one or more biometric sensors to obtain data of a user's biometric features (e.g., face, fingerprint).
[0300] In some examples, an indication of an error condition (e.g., 714A-I) is displayed including an animation (e.g., flashing) indicating that an authentication attempt is in progress. In some examples, the attempt to authenticate includes attempting to detect biometric information using one or more biometric sensors. Displaying a flashing animation indicating that an authentication attempt is in progress provides feedback to the user about the current state of the device and that no further action is required at this time. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some examples, no indicator is displayed during biometric authentication.
[0301] In some examples, after receiving a request to perform an operation that requires authentication (or in response thereto), and before displaying an indication of an error condition (e.g., 714A-I), an electronic device (e.g., 100, 300, 500, 700) performs authentication. In some examples, while performing authentication, the electronic device 700 displays a first indication (e.g., 710, 714A-I) (e.g., a ring rotating around a sphere, a flashing user interface object, wherein the user interface object includes an indication of an error condition) on a display (e.g., 702), the first indication being that the electronic device is using one or more biometric sensors (e.g., 703) of the electronic device to obtain information about a biometric feature. In some examples, displaying an indication of an error condition includes replacing the display of the first indication with the display of the indication of the error condition. Displaying an indication that biometric authentication is occurring provides feedback to the user about the current state of the device (e.g., biometric authentication is being performed) and the user does not need to take any action at this time. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0302] In some examples, when performing authentication, the electronic device (e.g., 100, 300, 500, 700) displays a first lock icon (e.g., 706) (e.g., an icon indicating a locked state of the electronic device) and a first animation that transitions from the first lock icon to the first indication on a display (e.g., 702). In some examples, after displaying an indication of an error condition (e.g., and based on determining that the authentication is successful) and after displaying the first animation, the electronic device displays a second animation on the display (e.g., 702), which transitions from an indication of the error condition to an unlock icon (e.g., 716) (e.g., an icon indicating a locked state of the electronic device). In some examples, the first animation and the second animation show a deformation from one object to the next. In some examples, the second animation includes displaying the first lock icon after the indication of the error condition and before the unlock icon.
[0303] In some examples, after displaying an indication of an error condition, the electronic device displays an animation on the display (e.g., 702) that transitions from the indication of the error condition to a second lock icon (e.g., 706), or from a second indication (e.g., 710, 714A-I) (e.g., a ring rotating around a sphere) to a second lock icon (e.g., an icon indicating a locked state of the electronic device), the second indication being that the electronic device uses one or more biometric sensors of the electronic device to obtain information about a biometric feature. In some examples, the second lock icon is the first lock icon. In some examples, the second indication is the first indication.
[0304] In some examples, after retrying authentication and after displaying an indication of an error condition and based on determining that there is no error condition, the electronic device displays a third indication (e.g., 710, 714A-I) on the display (e.g., a ring rotating around a sphere, a flashing user interface object, wherein the user interface object includes an indication of an error condition), the third indication being that the electronic device is using one or more biometric sensors of the electronic device to obtain information about a biometric feature. In some examples, the third indication is the first indication.
[0305] In some examples, before displaying the indication of the error condition, the electronic device (e.g., 100, 300, 500, 700) displays on the display (e.g., 702) a third lock icon (e.g., 706) (e.g., an icon indicating a locked state of the electronic device) at a location on the display. In some examples, the indication of the error condition (e.g., 714A-I) is displayed adjacent to (e.g., next to, adjacent to, at, within a predetermined distance of) the location on the display. In some examples, the third lock icon is the first lock icon and / or the second lock icon.
[0306] In some examples, when the electronic device is in a locked state, when a request to perform an operation requiring authentication is received and based on determining that the authentication is successful, the electronic device (e.g., 100, 300, 500, 700) transitions from a locked state to an unlocked state. In some examples, the operation requiring authentication transitions the electronic device from a locked state to an unlocked state. In some examples, when the electronic device is in a locked state, when a request to perform an operation requiring authentication is received and based on determining that the authentication is unsuccessful, the electronic device remains in a locked state. Keeping the device in a locked state when authentication is unsuccessful enhances device security by preventing fraudulent and / or unauthorized access to the device. Measures to improve the security of the device enhance the operability of the device by preventing unauthorized access to content and operations, and additionally reduce power usage and extend the battery life of the device by enabling users to use the device more efficiently.
[0307] In some examples, when the electronic device is in a locked state, and when a request to perform an operation requiring authentication is received and based on determining that the authentication is unsuccessful, the electronic device (e.g., 100, 300, 500, 700) remains in a locked state and retries authentication (e.g., biometric authentication) (e.g., automatically retrying authentication). In some examples, retrying authentication includes attempting to use one or more biometric sensors of the electronic device to obtain information about biometric features (e.g., face, fingerprint). In some examples, retrying authentication includes attempting to match biometric information obtained by one or more biometric sensors with authorized credentials (e.g., stored data authorized for biometric authentication). In some examples, after retrying authentication and based on determining that the authentication obtained by the retry authentication is successful, the electronic device transitions from a locked state to an unlocked state. In some examples, after retrying authentication and based on determining that the authentication obtained by the retry authentication is unsuccessful, the electronic device remains in a locked state.
[0308] In some examples, after receiving a request to perform an operation requiring authentication (or in response thereto), an electronic device (e.g., 100, 300, 500, 700) attempts authentication (e.g., biometric authentication). In some examples, while attempting authentication, the electronic device displays a third indication (e.g., 710, 714A-I) (e.g., a ring rotating around a sphere) on a display (e.g., 702), the third indication being that the electronic device is using one or more biometric sensors of the electronic device to obtain information about a biometric feature (e.g., face, fingerprint). In some examples, the indication is a scanning animation. In some examples, the third indication is the first indication and / or the second indication. In some examples, when authentication is retried, the electronic device maintains display of the third indication on the display (e.g., 702).
[0309] In some examples, based on determining that the authentication obtained by retrying the authentication is unsuccessful, the electronic device displays an animation alternating between a first position and a second position with a lock icon (e.g., 706) (e.g., an icon indicating the lock state of the electronic device) on a display (e.g., 702), the second position being different from the first position. In some examples, the animation with the lock icon is an animation of the lock icon shaking (e.g., rotating back and forth from one side to the other). In some examples, the electronic device displays an animation involving the lock icon to indicate that the biometric authentication has failed. In some examples, a tactile output is provided in conjunction with shaking the lock icon. In some examples, a tactile output is not provided. In some examples, based on determining that the biometric information captured using one or more biometric sensors does not correspond to or match the authorization credentials, the electronic device (e.g., 100, 300, 500, 700) maintains the lock state of the electronic device. The animation showing the lock icon shaking provides feedback to the user about the current state of the device (e.g., biometric authentication failed) and prompts the user to take further action. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0310] In some examples, the electronic device (e.g., 100, 300, 500, 700) includes a biometric sensor (e.g., 703), and the set of error condition criteria includes one or more of the following error condition criteria:
[0311] The distance of the biometric feature from the biometric sensor exceeds a first predetermined threshold distance (e.g., the biometric feature (e.g., face) is too far from the biometric sensor) or exceeds a maximum value of a distance range (e.g., 712). In some examples, exceeding the first predetermined threshold or the maximum value of the distance range is highly correlated to degradation or reduced accuracy of information about the biometric feature obtained by the biometric sensor. In some examples, the user can correct this error condition by moving the user's face closer to the biometric sensor.
[0312] The distance of the biometric feature from the biometric sensor falls below a second predetermined threshold distance (e.g., the biometric feature (e.g., face) is too close to the biometric sensor) or falls below a minimum value of a distance range (e.g., 712). In some examples, falling below the second predetermined threshold or the minimum value of the distance range is highly correlated to degradation or reduced accuracy of information about the biometric feature obtained by the biometric sensor. In some examples, the user can correct this error condition by moving the user's face further away from the biometric sensor.
[0313] The biometric sensor (e.g., 703) is occluded (e.g., partially occluded, completely occluded, occluded to an extent sufficient to inhibit operation of the sensor) (e.g., occluded by a portion of the user (e.g., a hand) while interacting with the electronic device). In some examples, the user can correct this error condition by moving the user's hand away from the biometric sensor.
[0314] A sub-portion of the detected biometric feature (e.g., eyes of the detected face) is not oriented toward the biometric sensor (e.g., one or more eyes are not focused on the electronic device (e.g., biometric sensor)). In some examples, the user can correct this error condition by opening the user's eyes or looking at the electronic device (e.g., biometric sensor).
[0315] At least a portion of the detected biometric feature is obscured (e.g., partially obscured, completely obscured, obscured enough to provide incomplete information about the biometric feature). In some examples, the user can correct this error condition by removing an accessory (e.g., sunglasses) or clothing item (e.g., scarf, hat) that is obstructing the user's face.
[0316] No biometric feature is detected within the field of view of the biometric sensor (e.g., 728).
[0317] The detected pose (e.g., orientation relative to the biometric sensor) of the biometric feature exceeds a threshold range (e.g., the biometric feature (e.g., face) is turned away from the biometric sensor). In some examples, exceeding the threshold range is highly correlated to degradation or reduced accuracy of information about the biometric feature obtained by the biometric sensor. In some examples, the user can correct the error condition by turning the user's face toward the electronic device (e.g., biometric sensor).
[0318] The electronic device detects (e.g., via one or more ambient light sensors) an amount of light (e.g., ambient light) that exceeds a predetermined light threshold (e.g., exceeding the predetermined light threshold is highly correlated to degradation or reduced accuracy of information about the biometric feature obtained by the biometric sensor). In some examples, the user can correct this error condition by turning the user's back toward the sun to reduce the amount of light detected by the electronic device or moving to a new location with less ambient light (e.g., indoors).
[0319] In some examples, the set of error condition criteria may be a first subset of the error conditions listed above. For example, the first subset may include one or more error condition criteria selected from: the distance of the biometric feature exceeds a first predetermined threshold distance, the distance of the biometric feature is less than a second predetermined threshold distance, the biometric feature is outside the field of view of the biometric sensor, and the posture of the biometric feature exceeds a threshold range. The first subset focuses on guiding the user to correct an error condition involving the positioning and / or orientation of the face. For another example, the second subset may include one or more error condition criteria selected from: the biometric sensor is blocked and the biometric feature is not detected within the field of view of the biometric sensor. The second subset focuses on guiding the user to correct an error condition in which the biometric sensor cannot obtain any information about the user's biometric feature. For another example, the third subset may include one or more error condition criteria selected from: the posture of the detected biometric feature exceeds a threshold range, and the biometric sensor is blocked. The third subset focuses on error conditions that may occur for a device of a certain form factor / size, such as a tablet device (e.g., iPad).
[0320] In some examples, an electronic device (e.g., 100, 300, 500, 700) includes a biometric sensor (e.g., 703) at a portion (e.g., location) of the electronic device (e.g., a portion that is not on a display). In some examples, in response to a request to perform an operation that requires authentication, the electronic device displays a progress indicator (e.g., 714A-I) on the display (e.g., 702) that is adjacent to (e.g., next to, next to, within a predetermined distance of) the portion of the electronic device, the progress indicator including an indication of an error condition. Displaying the progress indicator near the biometric sensor provides feedback to the user about the association of the biometric sensor with a process occurring at the device (e.g., an attempted authentication). Specifically, the user is aware of the biometric sensor during biometric authentication, making the user less likely to perform actions that interfere with the biometric sensor, or alternatively prompting the user to take corrective action. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0321] In some examples, the indication of the error condition (e.g., 714B) includes an indication that biometric authentication is not currently enabled on the electronic device based on a determination that biometric authentication is not currently enabled on the electronic device. In some examples, biometric authentication may become unavailable (or unenabled on the electronic device) when one or more of the following conditions are met: the electronic device has not been successfully authenticated or restarted since being turned on; the electronic device has not been unlocked for more than a predetermined amount of time (e.g., 48 hours); a password has not been used to unlock the device for more than a predetermined amount of time (e.g., 156 hours); biometric authentication using a biometric feature (e.g., face, fingerprint) has not been used to unlock the device for more than a predetermined amount of time (e.g., 4 hours); the electronic device has received a remote lock command; biometric authentication has failed more than a predetermined number of times (e.g., 5, 10, 15) since the last successful authentication using the device; the electronic device has received a shutdown and / or emergency SOS command, and an explicit request from the user to disable biometric authentication has been detected. Displaying an indication that biometric authentication is not currently enabled provides feedback to the user on the current state of the device and prompts the user to seek alternative methods to authenticate themselves. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0322] In some examples, the indication that biometric authentication is not currently enabled includes an enabling indication (e.g., 714B) (e.g., the indication is an enabling indication). In some examples, the electronic device (e.g., 100, 300, 500, 700) receives an input corresponding to the enabling indication (e.g., 720), and in response to receiving the input corresponding to the enabling indication, the electronic device (e.g., 100, 300, 500, 700) displays a credential input user interface (e.g., 722A) having multiple character input keys on the display (e.g., 702). In some examples, the credential input user interface includes a virtual keypad or a virtual keyboard. In some examples, the virtual keypad or the virtual keyboard includes multiple character input keys.
[0323] In some examples, an electronic device (e.g., 100, 300, 500, 700) detects a condition that triggers an authentication attempt (e.g., biometric authentication). In some examples, the request to perform an operation requiring authentication includes a request to unlock the device (e.g., a swipe at a predefined location). In some examples, in response to detecting a condition that triggers an authentication attempt (e.g., biometric authentication) and based on determining that the condition corresponds to an alert (e.g., 708) generated by the device in the absence of user input to the device (e.g., based on the satisfaction of a standard rather than detecting user input), while the biometric feature is detectable by one or more biometric sensors (e.g., a face is detected in the field of view of one or more facial detection sensors such as a depth camera), the electronic device displays a fifth indication (e.g., 710) (e.g., a ring rotating around a sphere), the fifth indication being that the electronic device is using one or more biometric sensors of the electronic device to obtain information about the biometric feature. In some examples, based on determining that the condition corresponds to an alert generated by the device in the absence of user input to the device (e.g., based on the satisfaction of a standard rather than the detection of user input), while the biometric feature is not available for detection by one or more biometric sensors (e.g., no face is detected in the field of view of one or more facial detection sensors such as a depth camera), the electronic device forgoes displaying a fifth indication (e.g., a ring rotating around a sphere), the fifth indication being that the electronic device is using one or more biometric sensors of the electronic device to obtain information about the biometric feature. In some examples, based on determining that the condition corresponds to user input to the device (e.g., a request unrelated to a notification; a request as a touch gesture input (e.g., a tap, a swipe (e.g., 724) (e.g., swipe up) or sensor data indicating movement of the device (e.g., lifting))), the electronic device displays a fifth indication, the fifth indication being that the electronic device is using one or more biometric sensors of the electronic device to obtain information about the biometric feature (e.g., regardless of whether the biometric feature is available for detection by one or more biometric sensors). Forgoing displaying the indication when a face is not detected prevents possible confusion for the user because if the biometric feature is not detected, the user may not intend to initiate biometric authentication. Thus, forgoing display instructions in this situation makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0324] Note that the process described above with respect to method 800 (eg, FIG. 8A to FIG. 8B) also apply in a similar manner to the methods described below. For example, method 1000, method 1200, and / or method 1400 optionally include one or more features of the various methods described above with reference to method 800. For example, error indications (e.g., 714A-I) as described with respect to method 800 may be used to provide an indication of an error condition during biometric authentication performed in the processes described with respect to method 1000, method 1200, and method 1400. For the sake of brevity, these details are not repeated below.
[0325] FIG. 9A to FIG. 9U An exemplary user interface for providing instructions about a biometric sensor during biometric authentication according to some examples is shown. The user interface in these figures is used to illustrate the process described below, including FIG. 10A to FIG. 10C process.
[0326] Fig. 9A An electronic device 900 (e.g., portable multifunction device 100, device 300, device 500) is shown. FIG. 9A to FIG. 9U In the exemplary example shown, the electronic device 900 is a tablet computer. In other examples, the electronic device 900 can be a different type of electronic device, such as a smart phone (e.g., the electronic device 700). The electronic device 900 includes a display 902, one or more input devices (e.g., a touch screen, a button 904, and a microphone of the display 902), a wireless communication radio component, and a biometric sensor 903. The electronic device 900 includes a biometric sensor 903. In some examples, the biometric sensor 903 includes one or more biometric sensors, which may include a camera, such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the biometric sensor 903 includes some or all of the features of the biometric sensor 703. In some examples, the biometric sensor 903 includes one or more fingerprint sensors (e.g., a fingerprint sensor integrated into a button). In some examples, the electronic device 900 also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the biometric feature (e.g., face) during the capture of the biometric data of the biometric feature by the biometric sensor 903. In some examples, the electronic device 900 includes multiple cameras separate from the biometric sensor 903. In some examples, the electronic device 900 includes only one camera separate from the biometric sensor 903.
[0327] exist Fig. 9A At , the user wishes to purchase goods using payment information stored on the electronic device 900. Fig. 9AAs shown, electronic device 900 is in split-screen (e.g., multitasking) mode. In split-screen mode, electronic device 900 simultaneously displays an application store user interface (UI) 906 in a left area 907 of display 902 and a browser UI 908 in a right area 909 of display 902. While simultaneously displaying application store UI 906 and browser UI 908, electronic device 900 receives input 910 at purchase affordance 912.
[0328] exist Fig. 9B In response to receiving input 910 at purchase enable indication 912, electronic device 900 swaps applications displayed in left area 907 and right area 909 of display 902. Specifically, electronic device 900 displays browser UI 908 in left area 907 and application store UI 906 in right area 909. Electronic device 900 swaps applications so as to place applications associated with purchased goods in an area closer to biometric sensor 903. By placing browser UI 908 in left area 907, electronic device 900 provides the user with an indication of the location of biometric sensor 903, which is used to authenticate the user before authorizing payment for the purchase of goods. Fig. 9B As shown, swapping applications also places an application associated with the purchased goods in an area closer to button 904. In some examples, when button 904 and biometric sensor 903 are not adjacent (e.g., on the same side), electronic device 900 swaps applications when necessary to place the application associated with the purchased goods in an area closer to biometric sensor 903. In some examples, when button 904 and biometric sensor 903 are not adjacent (e.g., on the same side), electronic device 900 swaps applications when necessary to place the application associated with the purchased goods in an area closer to button 904.
[0329] In addition, if Fig. 9B As shown, in response to receiving input 910 at purchase affordance 912, electronic device 900 dims browser UI 908 while dimming app store UI 906 to a greater degree than browser UI 908. By making browser UI 908 less dim than app store UI 906, electronic device 900 indicates to the user which application is associated with payment page interface 914 and the goods that the user wishes to purchase.
[0330] In addition, in response to receiving input 910 at purchase enable representation 912, electronic device 900 simultaneously displays payment page interface 914, which has information about the purchased goods and prompt 916 to prompt the user to double-click button 904 to initiate a process for authorizing payment of the goods. In addition, in response to receiving input 910 at purchase enable representation 912, electronic device 900 displays dynamic indication 918 to emphasize the location of button 904. While displaying payment page interface 914, the electronic device receives input 920 at button 904 (e.g., a double press of button 904). Prompt 916 instructs the user to provide one or more activations of button 904 (e.g., two presses of button 904). In some examples, prompt 916 is highlighted (on payment page interface 914) relative to one or more other displayed objects. In some examples, dynamic indication 918 highlights the location of button 904 on the device by continuously changing size (e.g., continuously alternating between becoming wider and narrower, or otherwise continuously changing size) on the display adjacent to the location of button 904, thereby allowing the user to more easily locate the button corresponding to the request of prompt 916. In some examples, payment page interface 914 includes the name of the application to which it corresponds (e.g., the name of the application from which the user initiated the process for authorizing payment).
[0331] exist Fig. 9C In response to receiving input 920 at button 904, electronic device 900 initiates a process for authorizing payment for goods. Authorizing payment for goods requires successful authentication of the user. Therefore, in response to receiving input 920, electronic device 900 initiates biometric authentication using biometric sensor 903. After initiating biometric authentication, electronic device 900 displays facial glyph 922, which provides an indication that electronic device 900 is attempting to biometrically authenticate the user (e.g., attempting to obtain biometric information about the user using biometric sensor 903). In some examples, facial glyph 922 includes a simulation of a representation of a biometric feature. In some examples, in response to receiving input 920 at button 904, the electronic device displays an animation of facial glyph 922 moving from the location of prompt 916 to the location of facial glyph 922, such as Fig. 9C In some examples, the animation causes the face glyph 922 to appear to slide out of the hint 916.
[0332] exist Fig.9DAt , after displaying the facial glyph 922, the electronic device transitions to displaying an authentication glyph 924, which provides an indication that the electronic device 900 is attempting to biometrically authenticate the user (e.g., continuing to attempt to obtain biometric information, attempting to match the obtained information with the stored authorization credentials). The authentication glyph 924 includes a plurality of spherically rotating rings. In some examples, the authentication glyph 924 provides an indication that the biometric data is being processed (e.g., compared to the stored authorization credentials).
[0333] While displaying the authentication glyph 924, the electronic device 900 detects that an error condition exists (e.g., a state that prevents the biometric sensor 903 from obtaining sufficient information about the user's face). Specifically, the electronic device 900 detects that the biometric sensor 903 is covered by a physical object (e.g., a user's hand). In some examples, the electronic device 900 does not detect an error condition and is able to obtain sufficient information about the user's face. In some examples, after obtaining sufficient information about the user's face and while displaying the authentication glyph 924, the electronic device 900 determines whether the obtained information meets the biometric authentication criteria (e.g., determining whether the obtained biometric information matches a biometric template associated with the user (e.g., a stored authorization credential) within a threshold). In some examples, upon determining that the biometric authentication is successful (e.g., the biometric authentication criteria are met), the electronic device 900 transitions to an unlocked state.
[0334] exist Fig.9E In response to detecting the presence of an error condition, the electronic device 900 displays an error indication 926 at a location at the top of the display 902 (e.g., relative to ground, relative to the user). Error indication 926 provides an indication of the error condition that currently exists. In addition, in response to detecting the presence of an error condition, the electronic device 900 displays an error icon 928 at a location of the display 902 adjacent to the biometric sensor 903, thereby providing an indication of the location of the biometric sensor 903. By providing an indication of the location of the biometric sensor 903, the error icon 928 suggests to the user the cause of the error condition. In some examples, in response to detecting the presence of an error condition, the electronic device 900 displays an error indication 926 at a location adjacent to the biometric sensor 903. In some examples, the error indication 926 includes some or all of the features of the error indication 714A, including a flashing effect.
[0335] exist Fig.9F In further response to detecting the presence of an error condition, the electronic device 900 displays the payment page interface 914 from its Fig.9E Move from the initial position in Fig.9F903. By moving the payment page interface toward the biometric sensor 903, the electronic device 900 indicates to the user the presence of the error icon 928 in addition to indicating the location of the biometric sensor 903 (thereby suggesting to the user the cause of the error condition).
[0336] In some examples, error icon 928 is displayed at different locations on display 902 depending on the positioning of the user's hand on display 902. Fig.9F As shown, the user's hand covers a portion of the display 902 adjacent to the biometric sensor 903. When the user's hand contacts the display 902, the electronic device 900 detects input due to the contact of the user's hand. In response to detecting the input, the electronic device 900 displays an error icon 928 at a location where the input is not detected. Figure 9G In the user's hand than the user's hand in Fig.9F In some examples, in response to detecting the user's hand in Figure 9G The electronic device 900 is connected with Fig.9F The error icon 928 is displayed at different positions in the Figure 9G The position ratio Fig.9F The position in is closer to the biometric sensor 903. Figure 9H , the user's hand covers a majority of the upper left side of display 902. In some examples, in response to detecting that the user's hand is Figure 9G The electronic device 900 is connected with FIG. 9F to FIG. 9G The error icon 928 is displayed at a different position from the position in the electronic device 900. Specifically, in some examples, the electronic device 900 displays the error icon 928 at a position close to (or substantially next to) the biometric sensor 903 rather than at a position where the input of the user's hand is detected.
[0337] exist Fig.9I , the user moves her hand away so that it no longer covers the biometric sensor 903. While the error indication 926 and the error icon 928 are displayed, the electronic device 900 detects that the error condition no longer exists.
[0338] exist Figure 9J In response to detecting that the error condition no longer exists, the electronic device 900 automatically retries the biometric authentication. When retrying the biometric authentication, the electronic device 900 displays the authentication glyph 924. When displaying the authentication glyph 924, the electronic device 900 attempts to authenticate the user in a biometric manner. Specifically, the electronic device 900 uses the biometric sensor 903 to obtain information about the user's face and determines whether the biometric authentication is successful (for example, the obtained information matches the stored authorization credentials).
[0339] When retrying the biometric authentication, the electronic device 900 determines that the biometric authentication is successful. Figure 9K At, when determining that the biometric authentication is successful, the electronic device 900 displays a success glyph 930, which provides an indication of the success of the biometric authentication. In some examples, the success glyph 930 replaces the authentication glyph 924.
[0340] exist Figure 9L In response to determining that the biometric authentication is successful, the electronic device 900 displays a processing indicator 932 that provides an indication that the payment transaction is being processed (e.g., the electronic device 900 transmits payment information (e.g., credentials) to an external device (e.g., a server) to authorize payment). In some examples, the processing indicator 932 has a pattern similar to or the same as the authentication glyph 924.
[0341] exist Figure 9M At, upon receiving an indication that payment has been completed (e.g., authorization), the electronic device 900 displays a completed indication 934, which provides an indication that payment has been completed. The completed indication 934 includes a check mark to indicate completion.
[0342] Figures 9N to 9S A technique for displaying an error indication 926 and an error icon 928 when the error indication 926 and the error icon 928 are to be displayed at approximately the same location is shown. Figure 9N In an application, a user wishes to unlock the device to access restricted content (e.g., home screen, recently used applications). Figure 9N The electronic device 900 is shown in a portrait orientation relative to the ground, with the user covering the biometric sensor 903 with her hand. In addition, the electronic device 900 displays a locked state UI 936 using a lock icon 938. The lock icon 938 provides an indication that the electronic device 900 is in a locked state.
[0343] While displaying the lock state UI 936, the electronic device 900 receives a request to unlock the device. For example, the electronic device 900 detects that the user lifts the device from a substantially horizontal position.
[0344] exist Fig.9O At , in response to receiving the request to unlock the device, the electronic device 900 attempts to authenticate the user biometrically. While attempting to authenticate the user biometrically, the electronic device 900 displays an authentication glyph 924. Additionally, while attempting to authenticate the user biometrically, the electronic device 900 detects that an error condition exists (e.g., a condition that prevents the biometric sensor 903 from obtaining sufficient information about the user's face). Specifically, the electronic device 900 detects that the biometric sensor 903 is covered by a physical object (e.g., a user's hand).
[0345] exist Figure 9P In response to detecting the presence of an error condition, the electronic device 900 displays an error icon 928 at a location of the display 902 that is near the biometric sensor 903 (e.g., at the top of the display 902). In addition, in response to detecting the presence of an error condition, the electronic device 900 determines to display an error indication 926 at approximately the same location as the error icon 928. Upon determining that the error indication 926 will be displayed at approximately the same location, the electronic device 900 does not immediately display the error indication 926, but instead displays the error indication 926 as part of an animation that transitions from the error icon 928 to the error indication 926 to the lock icon 938, as described below with respect to FIG. Figure 9Q to Figure 9R described.
[0346] exist Figure 9Q At , after displaying the error icon 928, the electronic device 900 displays (eg, replaces display of the error icon 928 with) an error indication 926 (as discussed above) providing an indication of the cause of the error condition.
[0347] While the error indication 926 is displayed, the user moves her hand away from the biometric sensor 903 so that it no longer covers the biometric sensor 903. In response to detecting that the error condition no longer exists, the electronic device 900 automatically retries the biometric authentication.
[0348] exist Figure 9R As shown in 9S, when it is determined that the authentication is successful by retrying the biometric authentication, the electronic device 900 changes from a locked state to an unlocked state. Specifically, the electronic device 900 displays (for example, the display of the error indication 926 is replaced by) an animation of the lock icon 938 changing to the unlock icon 940, which provides the user with an indication that the electronic device 900 has changed to an unlocked state. In some examples, instead of a successful biometric authentication, the electronic device 900 determines that the authentication is unsuccessful due to a retry of the biometric authentication. In some examples, when it is determined that the authentication is unsuccessful, the electronic device 900 uses an enable to indicate that a password input UI is displayed, which triggers a retry of the biometric authentication when activated. In some examples, when retrying the biometric authentication, the electronic device 900 dims all parts of the display 902, except for the user interface associated with the retry of the biometric authentication.
[0349] Figure 9T Techniques for displaying error icon 928 are shown when error icon 928 would be displayed at substantially the same location as one of the notifications (e.g., 944A-D) being displayed. In some examples, a user wishes to view limited content of one or more notifications (e.g., 944A-D) displayed when electronic device 900 is in a locked state. Figure 9TAs shown, when the electronic device is in a portrait orientation, the user is covering the biometric sensor 903 with her hand, where the biometric sensor 903 is located near the bottom of the device. In some examples, while attempting to biometrically authenticate the user to access restricted content of a notification, the electronic device 900 detects that an error condition exists due to the user covering the biometric sensor 903 with her hand. In response to detecting the existence of the error condition, the electronic device 900 determines to display an error icon 928 at approximately the same location as one of the notifications (e.g., 944A-D). Upon making this determination and in response to detecting the existence of the error condition, the electronic device 900 displays the UI element 942 (e.g., background) simultaneously with the error icon 928 to provide a displayed background with the error icon 928 superimposed thereon. As shown Figure 9T As shown, UI element 942 is opaque, so that the notification (e.g., 944D) on which the error icon 928 is superimposed is not visible to the user. In some examples, UI element 942 is transparent, so that the notification on which the error icon 928 is superimposed is visible to the user.
[0350] Figure 9T Also shown is a diagram for hiding the error icon 928 when it will be displayed at approximately the same location as the unlock indicator 905. Figure 9U In some examples, the electronic device 900 displays the unlock indication 905, which provides an indication of the approximate location on the display 902 from which the user can swipe up to initiate biometric authentication. In some examples, while displaying the unlock indication 905, the electronic device 900 detects that an error condition exists due to the user covering the biometric sensor 903 with her hand. In some examples, in response to detecting the presence of the error condition, the electronic device 900 determines to display the error icon 928 at approximately the same location as the unlock indication 905. In some examples, upon making this determination and in response to detecting the presence of the error condition, the electronic device 900 stops displaying the unlock indication 905 and displays the error icon 928 at approximately the same location as the unlock indication 905.
[0351] When the error icon 928 is displayed, the electronic device 900 detects that the error condition no longer exists (e.g., because the user has removed her hand from the biometric sensor 903). Figure 9U As shown, the user has removed her hand from the biometric sensor 903. Figure 9U Once it is detected that the error condition no longer exists, the electronic device 900 stops displaying the error icon 928 and redisplays the unlock indication 905 at the location where it was previously displayed.
[0352] FIG. 10A to FIG. 10C1000 is a flowchart illustrating a method for providing an indication about a biometric sensor during biometric authentication according to some examples. Method 1000 is performed at an electronic device (e.g., 100, 300, 500, 900) having a display (e.g., 902) and a biometric sensor (e.g., 903) located at a first portion (e.g., a portion that is not part of the display) of the electronic device (e.g., a first biometric sensor of a device having multiple biometric sensors) (e.g., a fingerprint sensor, a contactless biometric sensor (e.g., a biometric sensor that does not require physical contact, such as a thermal or optical facial recognition sensor), an iris scanner). In some examples, the biometric sensor includes one or more cameras. Some operations in method 1000 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.
[0353] As described below, method 1000 provides an intuitive way to provide instructions about biometric sensors during biometric authentication. The method reduces the cognitive burden of a user performing biometric authentication, thereby creating a more efficient human-computer interface. For battery-powered computing devices, users can perform biometric authentication more quickly and efficiently, saving power and increasing the interval between battery charges.
[0354] An electronic device (e.g., 100, 300, 500, 900) detects (1002) (e.g., in response to a request to perform an operation requiring authentication) that an error condition exists that prevents a biometric sensor (e.g., a contactless biometric sensor such as a thermal or optical facial recognition sensor) from obtaining biometric information about a user of the device by being occluded (e.g., partially occluded, fully occluded, occluded to an extent sufficient to inhibit operation of the sensor) (e.g., occluded by a portion of a user (e.g., a hand) while interacting with the electronic device).
[0355] In response to (1004) detecting the presence of an error condition, the electronic device (e.g., 100, 300, 500, 900) displays an error indication (e.g., 928) (e.g., a graphical icon) on a display (e.g., 902). In some examples, the error indication includes text (e.g., indicating that the sensor is blocked). In some examples, the error indication does not include text. The error indication is displayed (1006) at a location adjacent to a first portion of the electronic device. In some examples, the location is at or near a portion of the display closest to the location of the biometric sensor (e.g., 903). Displaying the error indication provides the user with feedback about the current state of the device (e.g., the error condition prevents successful biometric authentication) and prompts the user to take further action to correct the error condition. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in addition reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently. Displaying an error indication near a biometric sensor provides feedback to the user regarding the relevance of the biometric sensor to a process occurring at the device (e.g., an attempted authentication). Specifically, the user is made aware of the biometric sensor during biometric authentication, making the user less likely to perform actions that interfere with the biometric sensor, or alternatively prompting the user to take corrective action. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.
[0356] In some examples, the error indication (e.g., 928) includes (1008) a biometric sensor obstruction icon and a bar that provides an indication that the biometric sensor is obstructed. In some examples, the error indication is associated with an electronic device that performs biometric authentication (e.g., using a biometric sensor to obtain biometric information about a biometric feature (e.g., face, fingerprint)). Providing an indication that the biometric sensor is obstructed provides feedback to the user about the current state of the device (e.g., the biometric sensor is obstructed) and prompts the user to take further action to correct the error condition. Thus, providing improved feedback using indications of appropriate movement of the biometric feature enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0357] Based on (1010) determining that the user interface of the electronic device is in a first orientation relative to the biometric sensor, the electronic device (e.g., 100, 300, 500, 900) displays an error indication at a first position in the user interface that is adjacent to (e.g., next to, next to, within a predetermined distance from) a first part of the electronic device.
[0358] Based on (1012), determining that the user interface of the electronic device is in a second orientation relative to the biometric sensor, the electronic device (e.g., 100, 300, 500, 900) displays an error indication (e.g., 928) at a second position in the user interface that is adjacent to (e.g., next to, next to, within a predetermined distance from) a first part of the electronic device, and the first orientation is different from the second orientation.
[0359] In some examples, while attempting (1014) to obtain biometric information using a biometric sensor (e.g., 903), an electronic device (e.g., 100, 300, 500, 900) displays (1016) a first progress indicator (e.g., 924, 926, 938, 940) on a display (e.g., 902). In some examples, the first progress indicator provides an indication of a current state of the electronic device (e.g., locked state, unlocked state, performing biometric authentication, error condition, error condition). In some examples, based on (1018) determining that a user interface (e.g., 906, 908) of the electronic device is in a third orientation relative to the biometric sensor, the user interface in the third orientation has a first top side, and the electronic device displays the first progress indicator adjacent to (e.g., adjacent to, next to, within a predetermined distance of) the first top side of the user interface in the third orientation. In some examples, according to (1020), it is determined that the user interface of the electronic device is in a fourth orientation relative to the biometric sensor, the user interface in the fourth orientation has a second top side, and the electronic device displays a first progress indicator adjacent to (e.g., adjacent to, next to, within a predetermined distance of) the second top side of the user interface in the fourth orientation, and the third orientation is different from the fourth orientation. In some examples, the first progress indicator is displayed at a position on the display that is closest to or adjacent to (e.g., adjacent to, next to, within a predetermined distance of) the biometric sensor. Regardless of the orientation, displaying the first progress indicator near the top of the display ensures that the user is more likely to be aware of the feedback (e.g., progress indicator) provided to the user. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some examples, the indicator is not displayed during biometric authentication.
[0360] In some examples, an electronic device (e.g., 100, 300, 500, 900) displays a second progress indicator (e.g., 924, 926, 938, 940) of the electronic device on a display (e.g., 902). In some examples, the second progress indicator provides an indication of the current state of the electronic device (e.g., locked state, unlocked state, performing biometric authentication, error state). In some examples, the first progress indicator is the second progress indicator. In some examples, the second progress indicator is an animation having a first part (e.g., an indication (e.g., a rotating ring) that the electronic device performs biometric authentication using a biometric sensor (e.g., 924)) and a second part different from the first part (e.g., an error state or an indication of an error state (e.g., 926), an indication of the current locked or unlocked state of the electronic device (e.g., a lock icon (e.g., 938), an unlock icon (e.g., 940))). In some examples, based on determining to display the second progress indicator at a position adjacent to the first part of the electronic device, the electronic device displays the error indication (e.g., 928) as part of the animation after the first part and before the second part.
[0361] In some examples, an electronic device (e.g., 100, 300, 500, 900) displays a home affordance (e.g., 905) (e.g., an indication of the location of a gesture that, when executed, causes display of a home screen, such as an upward swipe gesture from an edge of the display or a tap gesture on the affordance) on a display (e.g., 902) at a third location in a user interface (e.g., a location adjacent to a side (e.g., bottom side) of the user interface). In some examples, based on determining that an error indication is displayed at the third location (e.g., 928), the electronic device stops displaying the home affordance (e.g., 905) while displaying the error indication at the third location. Stopping display of the home affordance while displaying the error indication enables a user to quickly realize that the home affordance is inaccessible because an error exists, and prompts the user to take further action to correct the error condition. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some examples, the indicator is not displayed during biometric authentication.
[0362] In some examples, after ceasing to display the home affordance (e.g., 905), the electronic device (e.g., 100, 300, 500, 900) detects a correction of an error condition that prevents a biometric sensor (e.g., 903) from obtaining biometric information about a device user. In some examples, the electronic device detects the presence of an error condition after displaying an error indication (e.g., 928) at a third location. In some examples, in response to detecting a correction of the error condition, the electronic device displays the home affordance at a third location in the user interface on the display (e.g., 902) (e.g., and ceasing to display the error indication (e.g., 928)).
[0363] In some examples, an electronic device (e.g., 100, 300, 500, 900) detects an input (e.g., a palm, a finger) at a position adjacent to (e.g., adjacent to, next to, within a predetermined distance of) a first portion of the electronic device. In some examples, in response to detecting an input at a position adjacent to the first portion of the electronic device, the electronic device displays an error indication (e.g., 928) at a different position on the display. In some examples, the different position is a position where no input is detected. In some examples, before displaying the error indication at the new position, the electronic device determines a different position based on the position of the input relative to the display. In some examples, the different position is adjacent to a position adjacent to the first portion of the electronic device. In some examples, the error indication is moved to the different position after being initially displayed at a first position adjacent to the first portion of the electronic device. In some examples, the error indication is initially displayed at a selected position so as to be away from any area of the display that is known to be blocked (e.g., blocked by a detected touch input). Displaying error indications at different locations depending on the location of the input (e.g., the user's hand) provides feedback to the user about the current state of the device (e.g., an error condition prevents successful biometric authentication) and prompts the user to take further action to correct the error condition. In addition, by adjusting the location, the device ensures that the error indication is visible to the user, so the user is more likely to take corrective action at the device. Providing improved feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0364] In some examples, an electronic device (e.g., 100, 300, 500, 900) displays a first transaction interface (e.g., 914) (e.g., a transaction (or payment) interface that is separate from (or overlaid on top of) a user interface and includes transaction information such as a credit card number, billing address, etc.) on a display (e.g., 902 at a location adjacent to (e.g., adjacent to, next to, within a predetermined distance of) a first portion of the electronic device. In some examples, the first transaction interface is displayed in response to receiving an input (e.g., 910) corresponding to an affordance (e.g., 912) of a user interface (e.g., 908) (e.g., an affordance for making a payment or completing a transaction).
[0365] In some examples, displaying the first transaction interface (e.g., 914) includes displaying an animation of the first transaction interface transitioning (e.g., translating) from an initial position to a position adjacent to the first portion of the electronic device, the initial position being substantially centered relative to the display. In some examples, the animation includes displaying the first transaction interface (e.g., maintaining display of the first transaction interface) while the first transaction interface transitions (e.g., translates) from the initial position to a position adjacent to the first portion of the electronic device. In some examples, the animation includes a visual effect in which the first transaction interface appears to float while transitioning.
[0366] In some examples, an electronic device (e.g., 100, 300, 500, 900) displays a prompt (e.g., 916) on a display (e.g., 902) to provide one or more activations of a hardware button (e.g., 904) of the electronic device. In some examples, the electronic device prompts the user by displaying "Double-click Apple Pay". In some examples, the prompt is displayed adjacent to the button. In some examples, the prompt is displayed when the device is displaying a transaction user interface area (e.g., 914) without receiving any indication that a transaction terminal is nearby and is requesting transaction credentials (e.g., displaying a prompt providing one or more activations of a button before the device has been placed in the NFC field of an NFC reader requesting payment information). In some examples, the hardware button is a mechanical button or a solid-state button. In some examples, the button is a switch or any other type of two-state element. In some examples, the button has a fixed position relative to the electronic device, and in particular, relative to the display of the electronic device, so that the electronic device can display a prompt based on the position of the button. In some examples, the button is a solid-state button that operates based on capacitive and / or resistive touch and / or in response to changes in input intensity, without having a mechanical switch that is pressed to activate the button, but instead monitors whether the intensity of the input is above an intensity threshold corresponding to the activation of the solid-state button. In some examples, an electronic device (e.g., 100, 300, 500, 900) receives one or more activations (e.g., 920) of a hardware button of the electronic device, and in response to receiving one or more activations of the hardware button, the electronic device displays an authentication progress indicator (e.g., 922, 924, 930, 932, 934) on a display (e.g., 902). In some examples, displaying the authentication progress indicator includes displaying an animation of the authentication progress indicator transitioning from the position of the prompt (e.g., 916) to the final position of the authentication progress indicator. In some examples, the authentication indicator provides the status of the authentication (e.g., in progress, successful, unsuccessful). In some examples, the animation includes displaying the authentication progress indicator (e.g., maintaining display of the authentication progress indicator) while the authentication progress indicator transitions (e.g., translates) from the position of the prompt to the final position. In some examples, the animation includes a visual effect in which the authentication progress indicator appears to slide out of the prompt. In some examples, the authentication progress indicator is displayed with (or superimposed on) a user interface (e.g., 914) (or a transaction user interface area). Prompting the user to activate a hardware button guides the user to perform an action at the device in order to complete the transaction. Prompting the user in this manner enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.Displaying the authentication progress indicator provides feedback to the user regarding the status of the authentication. The improved feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.
[0367] In some examples, an electronic device (e.g., 100, 300, 500, 900) simultaneously displays (1022) a first application (e.g., corresponding to 906, 908) in a first area (e.g., 907, 909) and a second application (e.g., corresponding to 906, 908) in a second area (e.g., 907, 909) on a display (e.g., 902), the second application being adjacent to (e.g., next to, adjacent to, within a predetermined distance of) the first application. In some examples, the electronic device displays (1024) a second transaction interface (e.g., 914) on the display. In some examples, the second transaction interface is the first transaction interface. In some examples, the second transaction interface is displayed superimposed on the first application and / or the second application. In some examples, based on (1026) determining that the second transaction interface corresponds to the first application, the electronic device modifies the first visual characteristic of the first application (e.g., shielding, dimming, blurring). In some examples, when the first application includes information about goods or services (or transactions) purchased (or completed) using (or through) the second transaction interface, the second transaction interface corresponds to the first application. In some examples, the determination is made when the second transaction interface is displayed. In some examples, based on (1030) determining that the second transaction interface corresponds to the second application, the electronic device (e.g., 100, 300, 500, 900) modifies the first visual characteristic of the second application (e.g., shielding, dimming, blurring). In some examples, when the first application includes information about goods or services (or transactions) purchased (or completed) using (or through) the second transaction interface, the second transaction interface corresponds to the first application. In some examples, the determination is made when the second transaction interface is displayed.
[0368] In some examples, modifying the first visual characteristic of the first application includes modifying the second visual characteristic of the second application. In some examples, modifying the second visual characteristic of the second application includes increasing the darkening and / or increasing blur radius of the blur effect applied to the second application to a greater degree (or amount) than the first application. In some examples, modifying the first visual characteristic of the second application includes modifying the second visual characteristic of the first application. In some examples, modifying the second visual characteristic of the first application includes increasing the darkening and / or increasing blur radius of the blur effect applied to the first application to a greater degree (or amount) than the second application. Modifying the visual characteristics of one application to a greater degree than another application provides feedback to the user about which application is more relevant at the time. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some examples, the indicator is not displayed during biometric authentication.
[0369] In some examples, modifying the first visual characteristic of the first application includes displaying (1028) the first application in the second area based on determining that the second area is closer (e.g., closer) to a first portion of the electronic device (e.g., a biometric sensor) than the first area. In some examples, displaying the first application in the second area includes ceasing to display the first application in the first area. In some examples, modifying the first visual characteristic of the second application includes displaying (1032) the second application in the first area based on determining that the first area is closer (e.g., closer) to a first portion of the electronic device (e.g., a biometric sensor) than the second area. In some examples, displaying the second application in the first area includes ceasing to display the second application in the second area. In some examples, the electronic device displays an animation in which the first application swaps positions with the second application.
[0370] In some examples, based on determining that the second transaction interface (e.g., 914) corresponds to the first application, the second transaction interface includes an indication of the first application (e.g., the name of the first application). In some examples, based on determining that the second transaction interface corresponds to the second application, the second transaction interface includes an indication of the second application (e.g., the name of the second application).
[0371] Note that the process described above with respect to method 1000 (eg, FIG. 10A to FIG. 10C) may also be similarly applicable to the methods described below / above. For example, method 800, method 1200, and / or method 1400 optionally include one or more features of the various methods described above with reference to method 1000. For example, error icon 928, as described in method 1000, may be used to indicate that the biometric sensor is blocked when biometric authentication is performed in the process described in conjunction with method 800, method 1200, and method 1400. For the sake of brevity, these details are not repeated below.
[0372] FIG. 11A to FIG. 11S An exemplary user interface for orienting a device to register a biometric feature (e.g., a face for later use in biometric authentication) is shown according to some examples. The user interfaces in these figures are used to illustrate the processes described below, including FIG. 12A to FIG. 12C process.
[0373] Fig.11A An electronic device 900 (e.g., portable multifunction device 100, device 300, device 500) is shown. FIG. 11A to FIG. 11S In the exemplary example shown, the electronic device 900 is a tablet computer. In other examples, the electronic device 900 can be a different type of electronic device, such as a smart phone (e.g., the electronic device 700). The electronic device 900 includes a display 902, one or more input devices (e.g., a touch screen, a button 904, and a microphone of the display 902), a wireless communication radio component, and a biometric sensor 903. The electronic device 900 includes a biometric sensor 903. In some examples, the biometric sensor 903 includes one or more biometric sensors, which may include a camera, such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the biometric sensor 903 includes some or all of the features of the biometric sensor 703. In some examples, the biometric sensor 903 includes one or more fingerprint sensors (e.g., a fingerprint sensor integrated into a button). In some examples, the electronic device 900 also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the biometric feature (e.g., face) during the capture of the biometric data of the biometric feature by the biometric sensor 903. In some examples, the electronic device 900 includes multiple cameras separate from the biometric sensor 903. In some examples, the electronic device 900 includes only one camera separate from the biometric sensor 903.
[0374] exist Fig.11AIn the example of FIG. 1 , a user wishes to set up biometric (e.g., facial) authentication on electronic device 900. Successfully setting up biometric authentication on the device enables the user to perform operations on the device that require authentication (e.g., unlocking the device) by presenting the user's face for biometric authentication. To set up biometric authentication on an electronic device, the user must first register her face. The process for registering a face may include: FIG. 11A to FIG. 11O Some or all of the characteristics (or processes) of.
[0375] like Fig.11A As shown, electronic device 900 displays an introduction user interface (UI) 1106 using an initiation affordance 1108. Electronic device 900 receives input 1110 at initiation affordance 1108 to begin the process of registering the user's face for biometric authentication.
[0376] exist Fig. 11B In response to receiving input 1110 at initiating enable indication 1108, electronic device 900 determines that the orientation of the device is not appropriate for registering the user's face. In some examples, the appropriate orientation for registering the user's face is a vertical (e.g., upright) portrait orientation, where the portrait orientation causes biometric sensor 903 to be located at the top of the device (e.g., on the side of the device farthest from the ground). In response to determining that the orientation of the device is not appropriate for registering the user's face, electronic device 900 displays (e.g., replaces the display of introduction UI 1106 with) one or more prompts to prompt the user to orient electronic device 900 to an appropriate orientation. More specifically, electronic device 900 determines that electronic device 900 is in a substantially horizontal orientation (e.g., approximately parallel to the ground). Therefore, as Fig. 11B As shown, the electronic device 900 displays a prompt 1112A to prompt the user to lift the electronic device 900 to a vertical position.
[0377] In some examples, in response to receiving input 1110 at initiating affordance 1108, the electronic device determines that the orientation of the device is suitable for registering the user's face. In some examples, upon determining that the orientation is suitable for registering the user's face, the electronic device 900 automatically initiates a process for registering the user's face, as described below in conjunction with Fig.11D described.
[0378] exist Fig. 11CIn response to determining that the electronic device 900 is in a vertical position but not in a portrait orientation (e.g., the user has lifted the device off the table in response to prompt 1112A), the electronic device 900 displays (e.g., replaces the display of prompt 1112A with) prompt 1112B to prompt the user (e.g., using the biometric sensor 903 at the top) to rotate the electronic device 900 to a portrait orientation. Specifically, prompt 1112B prompts the user to rotate in a specific direction (e.g., using text and / or arrows) so that minimal rotation is required to achieve the desired (or appropriate) orientation. For example, prompt 1112B prompts the user to rotate the electronic device 900 clockwise because less rotation is required to achieve the desired orientation than rotating the device counterclockwise. In some examples, prompt 1112B includes an animation of the electronic device 900 representing a 90 degree clockwise rotation to indicate to the user the action required to orient the device to an appropriate orientation for registering the user's face.
[0379] In some examples, the electronic device 900 displays different prompts based on the orientation of the device. For example, if the biometric sensor 903 is located near the right edge of the device (e.g., relative to the user), the electronic device 900 displays Figure 11Q In some examples, prompt 1112C prompts the user to rotate the device counterclockwise (e.g., via text and / or an illustration of the direction to rotate the device). In some examples, prompt 1112C includes an animation of electronic device 900 indicating the action required to rotate the device 90 degrees counterclockwise to indicate to the user the action required to orient the device to the appropriate orientation for registering the user's face. For another example, if biometric sensor 903 is located near the bottom edge of the device (e.g., relative to the user), electronic device 900 displays Figure 11R In some examples, prompt 1112D prompts the user to rotate the device 180 degrees (e.g., via text and / or an illustration of the direction to rotate the device). In some examples, prompt 1112D includes an animation of electronic device 900 representing rotating 180 degrees clockwise or counterclockwise to indicate to the user the actions required to orient the device to a suitable orientation for registering the user's face.
[0380] exist Fig.11D In response to determining that the electronic device 900 is in a suitable orientation, the electronic device 900 automatically initiates a process for registering the user's face. FIG. 11D to FIG. 11FAs shown, after initiating the process for registering the user's face, the electronic device 900 displays a face registration UI 1114. The face registration UI 1114 includes a facial image of the user. In some examples, the facial image is an image of the user captured by one or more cameras on the device 900. For example, the facial image is optionally a real-time preview of image data captured by the one or more cameras (e.g., a digital viewfinder), which is continuously updated as the field of view and / or field of view content of the camera changes. In some examples, background content is removed so that only the user's face is visible in the facial image. The face registration UI 1114 also optionally includes an orientation guide superimposed (e.g., superimposed) on the facial image. The orientation guide is optionally a set of curves that extend into the virtual z dimension (e.g., along an axis perpendicular to the plane of the display) and cross over the center of the facial image. Therefore, the curves of the orientation guide appear to bulge outward relative to the plane of the display 902 to give a sense of the position of the user's head in three-dimensional space.
[0381] The facial registration UI 1114 also includes a registration progress meter. The registration progress meter includes a set of display elements (e.g., progress elements) arranged around the facial image and orientation guide. Fig.11D In some examples, the progress element is a set of lines arranged in a circular pattern extending radially outward from the facial image. In some examples, the progress element indicates the orientation of the user's face required to register the corresponding facial features. For example, when the user's head is tilted upward, the progress element in the upper portion of the registration progress meter optionally moves, fills, stretches, and / or changes color, which allows the one or more cameras on device 900 to capture image data of the underside of the user's face. In some examples, device 900 displays the progress element in the registration progress meter in an unregistered state (e.g., the progress element is grayed out and unchanged).
[0382] The face registration UI 1114 also includes textual prompts that instruct the user to move (e.g., rotate and / or tilt) their head in a circular motion during the registration process. In some examples, the textual prompts are optionally accompanied by tactile and / or auditory prompts, depending on device settings and / or user selections. In some examples, the device 900 displays textual prompts on the face registration UI 1114 through the face registration process.
[0383] In some examples, rather than automatically initiating a process for registering a user's face (and displaying face registration UI 1114), electronic device 900 displays a face registration UI 1114 in response to determining that electronic device 900 is in a suitable orientation. Figure 11SThe registration import UI 1146 in FIG. 1146 includes a face glyph (e.g., a representation of a biometric feature (e.g., a face)) and a registration progress meter. The registration progress meter includes a set of display elements (e.g., progress elements) arranged around the glyph. In some examples, the progress elements include the following: Fig.11D In some examples, to trigger the display of face registration UI 1114 and continue to register the user's face, the user activates the continue enable indication 1148 on the registration introduction UI 1146. For example, Figure 11S As shown, electronic device 900 detects activation (e.g., selection) of continue enable indication 1148 via input 1150 (e.g., a tap gesture). In some examples, in response to detecting activation of continue enable indication 1148, electronic device 900 initiates a process for registering the user's face, as described above with respect to Fig.11D described.
[0384] exist Fig.11G In the example, after successfully completing the registration of the user's face, the electronic device 900 displays (e.g., replaces the display of the face registration UI 1114 with) a scan completion interface 1116, which includes a continue enable indication 1118. The scan completion interface 1116 includes a facial image and a success status meter. Fig.11G In some examples, the facial image is blurred, faded, darkened, or otherwise obscured to indicate that no additional image data is being collected as part of the facial scan. In some examples, the success status meter is a solid continuous green circle around the facial image that provides a visual indication that the first scan is complete. To provide further visual notification, scan completion interface 1116 also includes a text prompt (e.g., a completion message).
[0385] After completing the registration of the user's face, a second iteration of the registration process is performed without requiring the user to re-orient the device. Fig.11G As shown, while displaying the scan completion interface 1116, the electronic device 900 receives input 1120 at the continue enable indication 1118 to initiate a second iteration of the registration process.
[0386] exist Fig.11H In response to receiving input 1120 at the continue enable indication 1118, the electronic device 900 initiates a second iteration of the registration process, similar to the above with respect to FIG. 11D to FIG. 11F As shown. The electronic device 900 initiates the second iteration without prompting the user to re-orient the device to an orientation different from its current orientation. Initiating the second iteration of the registration process includes displaying a second facial registration UI 1122. The second facial registration UI 1122 includes some or all of the features of the facial registration UI 1114.
[0387] exist Fig.11I At , after successfully completing the second iteration of the registration process, electronic device 900 displays (e.g., replaces the display of second facial registration UI 1122 with) second scan completion interface 1124, which includes a continue enable indication 1126. Second scan completion interface 1124 includes some or all of the fe...
Claims
1. A method for indicating a sensor position, comprising: At an electronic device having a biometric sensor and a touch-sensitive display: detecting an occurrence of an error condition for detecting biometric information at the biometric sensor; in response to detecting the occurrence of the error condition, displaying on the touch-sensitive display an indication of the location of the biometric sensor on the electronic device; while displaying the indication of the location of the biometric sensor on the electronic device, detecting a request to unlock the electronic device using the biometric sensor; as well as In response to detecting the request to unlock the electronic device using the biometric sensor: Upon determining that the error condition still occurs: ceasing to display the indication of the position of the biometric sensor; and displaying a touch-based user interface for inputting touch-based authentication information; and Based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
2. The method according to claim 1, further comprising: detecting a touch input sequence on the touch-sensitive display while displaying the touch-based user interface for inputting touch-based authentication information; In response to detecting the touch input sequence: Transitioning the electronic device from a locked state to an unlocked state based on determining that the touch input sequence matches the authorization credential; as well as Based on determining that the touch input sequence does not match the authorization credentials, the electronic device is maintained in a locked state.
3. The method according to any one of claims 1 to 2, wherein: Determining that the error condition still occurs includes determining that the error condition still occurs at a corresponding time that occurs after detecting the request to unlock the electronic device, and wherein the corresponding time is a time that occurs after a predetermined delay period from detecting the request to unlock the electronic device using the biometric sensor.
4. The method according to any one of claims 1 to 2, wherein attempting to unlock the electronic device comprises: upon determining that the biometric authentication is successful, unlocking the electronic device; as well as Based on determining that the biometric authentication is unsuccessful, an alternative authentication user interface is displayed on the touch-sensitive display.
5. The method of any one of claims 1 to 2, wherein the occurrence of the error condition includes a requirement that the biometric sensor be covered in order for the error condition to occur.
6. The method of any one of claims 1 to 2, wherein the occurrence of the error condition comprises a requirement that the touch-sensitive display be turned on in order for the error condition to occur.
7. A method according to any one of claims 1 to 2, wherein the occurrence of the error condition includes the following requirement: the input corresponding to the request to attempt biometric authentication has been satisfied in order for the error condition to occur.
8. The method of any one of claims 1 to 2, wherein the occurrence of the error condition comprises a requirement that a maximum number of failed authentication attempts has not been reached in order for the error condition to occur.
9. A method according to any one of claims 1 to 2, wherein the occurrence of the error condition includes the following requirement: the electronic device is oriented so that the biometric sensor is not pointed at a part of the biometric feature that can be used for biometric authentication so that the error condition occurs.
10. The method according to any one of claims 1 to 2, wherein the indication comprises instructions for changing a posture of a biometric feature towards the biometric sensor.
11. The method of any one of claims 1 to 2, wherein the indication comprises a textual description of the location of the biometric sensor.
12. The method of any one of claims 1 to 2, wherein the indication comprises a graphical indication located proximate to the biometric sensor.
13. The method of any one of claims 1 to 2, wherein the indication comprises a graphical representation of the location of the biometric sensor.
14. The method of any one of claims 1 to 2, wherein the indication comprises an animation showing the position of the biometric sensor.
15. The method according to any one of claims 1 to 2, wherein when the error condition is of a first type, the occurrence of the error condition is detected, the method further comprising: In response to detecting an occurrence of a second type of error condition, an indication of the occurrence of the second type of error condition is displayed without displaying an indication of the position of the biometric sensor.
16. A method according to any one of claims 1 to 2, wherein the determination that the error condition no longer occurs is made after the request to unlock the electronic device is detected using the biometric sensor and when the indication of the position of the biometric sensor is displayed.
17. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having a biometric sensor and a touch-sensitive display, the one or more programs comprising instructions for: detecting an occurrence of an error condition for detecting biometric information at the biometric sensor; in response to detecting the occurrence of the error condition, displaying on the touch-sensitive display an indication of the location of the biometric sensor on the electronic device; while displaying the indication of the location of the biometric sensor on the electronic device, detecting a request to unlock the electronic device using the biometric sensor; as well as In response to detecting the request to unlock the electronic device using the biometric sensor: Upon determining that the error condition still occurs: ceasing to display said indication of said position of said biometric sensor; as well as displaying a touch-based user interface for inputting touch-based authentication information; as well as Based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
18. The computer-readable storage medium of claim 17, wherein the one or more programs include instructions for: detecting a touch input sequence on the touch-sensitive display while displaying the touch-based user interface for inputting touch-based authentication information; In response to detecting the touch input sequence: Transitioning the electronic device from a locked state to an unlocked state based on determining that the touch input sequence matches the authorization credential; as well as Based on determining that the touch input sequence does not match the authorization credentials, the electronic device is maintained in a locked state.
19. The computer-readable storage medium according to any one of claims 17 to 18, wherein: Determining that the error condition still occurs includes determining that the error condition still occurs at a corresponding time that occurs after detecting the request to unlock the electronic device, and wherein the corresponding time is a time that occurs after a predetermined delay period from detecting the request to unlock the electronic device using the biometric sensor.
20. The computer-readable storage medium of any one of claims 17 to 18, wherein attempting to unlock the electronic device comprises: upon determining that the biometric authentication is successful, unlocking the electronic device; as well as Based on determining that the biometric authentication is unsuccessful, an alternative authentication user interface is displayed on the touch-sensitive display.
21. The computer-readable storage medium of any one of claims 17 to 18, wherein the occurrence of the error condition comprises a requirement that the biometric sensor be covered in order for the error condition to occur.
22. The computer-readable storage medium of any one of claims 17-18, wherein the occurrence of the error condition comprises a requirement that the touch-sensitive display be turned on in order for the error condition to occur.
23. The computer-readable storage medium of any one of claims 17 to 18, wherein the occurrence of the error condition comprises a requirement that an input corresponding to a request to attempt biometric authentication has been satisfied in order for the error condition to occur.
24. The computer-readable storage medium of any one of claims 17-18, wherein the occurrence of the error condition comprises a requirement that a maximum number of failed authentication attempts has not been reached in order for the error condition to occur.
25. A computer-readable storage medium according to any one of claims 17 to 18, wherein the occurrence of the error condition includes the requirement that the electronic device is oriented so that the biometric sensor is not pointing at a portion of a biometric feature that can be used for biometric authentication in order for the error condition to occur.
26. The computer-readable storage medium of any one of claims 17 to 18, wherein the indication comprises instructions for changing a posture of a biometric feature toward the biometric sensor.
27. The computer-readable storage medium of any one of claims 17 to 18, wherein the indication comprises a textual description of the location of the biometric sensor.
28. The computer-readable storage medium of any one of claims 17 to 18, wherein the indication comprises a graphical indication located proximate to the biometric sensor.
29. The computer-readable storage medium of any one of claims 17 to 18, wherein the indication comprises a graphical representation of the location of the biometric sensor.
30. The computer-readable storage medium of any one of claims 17 to 18, wherein the indication comprises an animation showing the position of the biometric sensor.
31. The computer-readable storage medium of any one of claims 17 to 18, wherein when the error condition is of a first type, the occurrence of the error condition is detected, the one or more programs include instructions for: In response to detecting an occurrence of a second type of error condition, an indication of the occurrence of the second type of error condition is displayed without displaying an indication of the position of the biometric sensor.
32. A computer-readable storage medium according to any one of claims 17 to 18, wherein the determination that the error condition no longer occurs is made after the request to unlock the electronic device is detected using the biometric sensor and when the indication of the position of the biometric sensor is displayed.
33. An electronic device comprising: Biometric sensors; Touch-sensitive display; one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: detecting an occurrence of an error condition for detecting biometric information at the biometric sensor; in response to detecting the occurrence of the error condition, displaying on the touch-sensitive display an indication of the location of the biometric sensor on the electronic device; while displaying the indication of the location of the biometric sensor on the electronic device, detecting a request to unlock the electronic device using the biometric sensor; as well as In response to detecting the request to unlock the electronic device using the biometric sensor: Upon determining that the error condition still occurs: ceasing to display said indication of said position of said biometric sensor; as well as displaying a touch-based user interface for inputting touch-based authentication information; as well as Based on determining that the error condition no longer occurs, attempting to unlock the electronic device using the biometric sensor.
34. The electronic device of claim 33, wherein the one or more programs include instructions for performing the following operations: detecting a touch input sequence on the touch-sensitive display while displaying the touch-based user interface for inputting touch-based authentication information; In response to detecting the touch input sequence: Transitioning the electronic device from a locked state to an unlocked state based on determining that the touch input sequence matches the authorization credential; as well as Based on determining that the touch input sequence does not match the authorization credentials, the electronic device is maintained in a locked state.
35. The electronic device according to any one of claims 33 to 34, wherein: Determining that the error condition still occurs includes determining that the error condition still occurs at a corresponding time that occurs after detecting the request to unlock the electronic device, and wherein the corresponding time is a time that occurs after a predetermined delay period from detecting the request to unlock the electronic device using the biometric sensor.
36. The electronic device of any one of claims 33 to 34, wherein attempting to unlock the electronic device comprises: upon determining that the biometric authentication is successful, unlocking the electronic device; as well as Based on determining that the biometric authentication is unsuccessful, an alternative authentication user interface is displayed on the touch-sensitive display.
37. An electronic device according to any one of claims 33 to 34, wherein the occurrence of the error condition includes the requirement that the biometric sensor be covered in order for the error condition to occur.
38. An electronic device as claimed in any one of claims 33 to 34, wherein the occurrence of the error condition comprises a requirement that the touch-sensitive display be turned on in order for the error condition to occur.
39. An electronic device according to any one of claims 33 to 34, wherein the occurrence of the error condition includes the requirement that an input corresponding to a request to attempt biometric authentication has been satisfied in order for the error condition to occur.
40. An electronic device as claimed in any one of claims 33 to 34, wherein the occurrence of the error condition comprises a requirement that a maximum number of failed authentication attempts has not been reached in order for the error condition to occur.
41. An electronic device according to any one of claims 33 to 34, wherein the occurrence of the error condition includes the requirement that the electronic device is oriented so that the biometric sensor is not pointing at a portion of a biometric feature that can be used for biometric authentication in order for the error condition to occur.
42. An electronic device according to any one of claims 33 to 34, wherein the indication comprises instructions for changing a posture of the biometric feature toward the biometric sensor.
43. An electronic device according to any one of claims 33 to 34, wherein the indication includes a text description of the location of the biometric sensor.
44. An electronic device according to any one of claims 33 to 34, wherein the indication comprises a graphical indication located near the biometric sensor.
45. An electronic device according to any one of claims 33 to 34, wherein the indication includes a graphical representation of the location of the biometric sensor.
46. An electronic device according to any one of claims 33 to 34, wherein the indication includes an animation showing the position of the biometric sensor.
47. An electronic device according to any one of claims 33 to 34, wherein when the error condition is of a first type, the occurrence of the error condition is detected, and the one or more programs include instructions for performing the following operations: In response to detecting an occurrence of a second type of error condition, an indication of the occurrence of the second type of error condition is displayed without displaying an indication of the position of the biometric sensor.
48. An electronic device according to any one of claims 33 to 34, wherein the determination that the error condition no longer occurs is made after the request to unlock the electronic device is detected using the biometric sensor and when the indication of the position of the biometric sensor is displayed.
49. An electronic device comprising: Biometric sensors; Touch-sensitive display; as well as Device for carrying out the method according to any one of claims 1 to 16.
50. A computer program product comprising one or more programs configured to be executed by one or more processors of an electronic device having a touch-sensitive display and one or more biometric sensors, the one or more programs including instructions for performing a method according to any one of claims 1 to 16.
