DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR ALLOWING SECURE OPERATION - Patent application

Abstract

In some examples, a change in a user's current viewpoint is detected while the three-dimensional environment is visible. In some examples, user permission for a secure operation using the virtual user interface object is enabled in accordance with a determination that at least a threshold amount of the object is visible from the user's viewpoint. In some examples, at a first time, a user is biometrically authenticated to perform a first type of biometric authentication. In some examples, at a second time, a request to perform a secure operation is received. In some examples, after receiving the request, the secure operation is performed without performing the first type of biometric authentication in accordance with a determination that the user has met the respective criteria between the first and second times.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application Ser. No. 18 / 167,767, entitled "DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR AUTHORIZING A SECURE OPERATION," filed February 10, 2023, and U.S. Provisional Application No. 63 / 314,900, entitled "DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR AUTHORIZING A SECURE OPERATION," filed February 28, 2022, the entire contents of each of which are incorporated herein by reference in their entirety for all purposes. [Technical field]

[0002] The present disclosure generally relates to computer systems in communication with a display generation component, one or more input devices, and (optionally) a biometric sensor, that provide computer-generated experiences, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary extended reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays for computer systems and other electronic computing devices are used to interact with the virtual / extended reality environment. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements such as buttons and other graphics. Summary of the Invention

[0004] Methods and interfaces that allow secure operation in environments that include some virtual elements (e.g., augmented reality, mixed reality, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide solutions for automatically filling out various text fields, facilitating payment for items or other services, and providing access to third-party applications (e.g., social networking, games, business services, etc.) are complex, cumbersome, and error-prone, creating significant cognitive load for users and undermining the experience in the virtual / augmented reality environment. In addition, the methods take longer than necessary, thereby wasting energy of the computer system. This latter consideration is particularly important in battery-operated devices.

[0005] Thus, there is a need for a computer system having improved methods and interfaces for authorizing secure operations to make interaction with the computer system more efficient and intuitive for users. Such methods and interfaces, optionally complementing or replacing conventional methods for authorizing secure operations, reduce the number, extent and / or type of inputs from a user by helping the user understand the connection between a provided input and a device response to that input, thereby creating a more efficient human-machine interface.

[0006] The above disadvantages and other problems associated with user interfaces of computer systems are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a watch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generating components, the output devices including one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on a touch-sensitive surface, movements of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, creating spreadsheets, playing games, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, playing digital music, taking notes, and / or playing digital videos. Executable instructions to perform those functions are optionally contained in a transitory and / or non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for authorizing secure operation. Such methods and interfaces may complement or replace conventional methods for authorizing secure operation. Such methods and interfaces reduce the number, extent, and / or type of inputs from a user, creating a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces conserve power and increase the time between battery charges. Furthermore, these methods and interfaces help reduce the number of repeated inputs from a user by automatically filling in or otherwise inputting sensitive information such as passwords or payment information. Such methods and interfaces also reduce processing power and display usage by reducing the amount of time a user spends interacting with a device when authorizing secure operation.

[0008] According to some embodiments, a method is described. The method is performed in a computer system in communication with one or more input devices and a display generating component. The method includes detecting a change in a user's current viewpoint via the one or more input devices while a three-dimensional environment is visible via the display generating component, the three-dimensional environment including virtual user interface objects that include information regarding a secure operation, and in response to detecting the change in the user's viewpoint, enabling user permission of the secure operation using the virtual user interface objects in accordance with a determination that at least a threshold amount of the virtual user interface objects are visible from the user's viewpoint and the user is authorized to perform the secure operation, and withdrawing enabling user permission of the secure operation using the virtual user interface objects in accordance with a determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a computer system in communication with one or more input devices and a display generating component, the one or more programs including instructions for detecting a change in a user's current viewpoint via the one or more input devices while a three-dimensional environment is visible via the display generating component, the three-dimensional environment including virtual user interface objects that include information regarding a secure operation, and in response to detecting the change in the user's viewpoint, enabling user permission of the secure operation using the virtual user interface objects in accordance with a determination that at least a threshold amount of the virtual user interface objects are visible from the user's viewpoint and the user is authorized to perform the secure operation, and withdrawing enabling of the user permission of the secure operation using the virtual user interface objects in accordance with a determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint.

[0010] According to some embodiments, a computer system is described, the computer system being in communication with one or more input devices and a display generating component, the computer system comprising one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for detecting a change in a user's current viewpoint via the one or more input devices while a three-dimensional environment is visible via the display generating component, the three-dimensional environment including virtual user interface objects that include information regarding a secure operation, and in response to detecting the change in the user's viewpoint, enabling user permission of the secure operation using the virtual user interface objects in accordance with a determination that at least a threshold amount of the virtual user interface objects are visible from the user's viewpoint and the user is authorized to perform the secure operation, and ceasing to enable user permission of the secure operation using the virtual user interface objects in accordance with a determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint.

[0011] According to some embodiments, a computer system is described that is in communication with one or more input devices and a display generating component that includes means for detecting a change in a user's current viewpoint via the one or more input devices while a three-dimensional environment including virtual user interface objects that include information regarding a secure operation is visible via the display generating component, and means for enabling user permission of the secure operation using the virtual user interface objects in response to detecting the change in the user's viewpoint in accordance with a determination that at least a threshold amount of the virtual user interface objects are visible from the user's viewpoint and the user is authorized to perform the secure operation, and means for withdrawing enabling user permission of the secure operation using the virtual user interface objects in accordance with a determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint.

[0012] According to some embodiments, a method is described. The method is performed in a computer system in communication with one or more input devices, a display generating component, and a biometric sensor. The method includes biometrically authenticating a user of the device at a first time using the biometric sensor to perform a first type of biometric authentication, receiving a request to perform a secure operation at a second time after the first time, and, in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first time and the second time based on a plurality of sensor measurements obtained at a plurality of intermediate times between the first time and the second time, the sensor measurements including a sensor measurement obtained at the first intermediate time and a sensor measurement obtained at the second intermediate time, the respective criteria detecting that a same user used the device between the first time and the second time, and aborting the performance of the secure operation in accordance with a determination that a continuity criterion was not satisfied between the first time and the second time.

[0013] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with one or more input devices, a display generating component, and a biometric sensor, the one or more programs including instructions for: at a first time, biometrically authenticating a user of the device using the biometric sensor to perform a first type of biometric authentication; receiving a request to perform a secure operation at a second time after the first time; in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first time and the second time based on a plurality of sensor measurements obtained at a plurality of intermediate times between the first time and the second time, including a sensor measurement obtained at the first intermediate time and a sensor measurement obtained at the second intermediate time, the respective criteria detecting that a same user used the device between the first time and the second time; and aborting performance of the secure operation in accordance with a determination that a continuity criterion was not satisfied between the first time and the second time.

[0014] According to some embodiments, a computer system is described in communication with one or more input devices, a display generating component, and a biometric sensor. The computer system includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: at a first time, biometrically authenticating a user of the device using the biometric sensor to perform a first type of biometric authentication; receiving a request to perform a secure operation at a second time after the first time; in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first time and the second time based on a plurality of sensor measurements obtained at a plurality of intermediate times between the first time and the second time, the sensor measurements including a sensor measurement obtained at the first intermediate time and a sensor measurement obtained at the second intermediate time, the respective criteria detecting that a same user used the device between the first time and the second time; and aborting the performance of the secure operation in accordance with a determination that a continuity criterion was not satisfied between the first time and the second time.

[0015] According to some embodiments, a computer system is described that is in communication with one or more input devices and a display generating component, the computer system including means for biometrically authenticating a user of the device at a first time using a biometric sensor to perform a first type of biometric authentication, means for receiving a request to perform a secure operation at a second time after the first time, means for performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in response to receiving the request to perform the secure operation, in accordance with a determination that a user of the device satisfied respective criteria between the first time and the second time based on a plurality of sensor measurements taken at a plurality of intermediate times between the first time and the second time, the sensor measurements including a sensor measurement taken at the first intermediate time and a sensor measurement taken at the second intermediate time, the respective criteria detecting that a same user used the device between the first time and the second time, and means for abandoning performance of the secure operation in accordance with a determination that a continuity criterion was not satisfied between the first time and the second time.

[0016] According to some embodiments, a method is described in a computer system in communication with one or more input devices and a display generating component, the method including receiving a request via the one or more input devices for display of a user interface associated with performing a secure operation, and displaying via the display generating component, in response to the request for display of a user interface associated with performing a secure operation, a first user interface including a separate prompt for providing additional input for authorizing the device to perform the secure operation, wherein, in accordance with a determination that the computer system is operating in a first mode, the separate prompt is a first prompt providing a physical input for authorizing the secure operation, and in accordance with a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt for displaying a second user interface, the second user interface enabling user authorization of the secure operation without the use of a physical input.

[0017] According to some embodiments, a non-transitory computer-readable storage medium is described, the non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with one or more input devices and a display generating component, the one or more programs including instructions for receiving via the one or more input devices a request for display of a user interface associated with performing a secure operation, and displaying via the display generating component a first user interface including a separate prompt for providing additional input for authorizing the device to perform the secure operation, in response to the request for display of the user interface associated with performing the secure operation, in accordance with a determination that the computer system is operating in a first mode, the separate prompt being a first prompt providing a physical input for authorizing the secure operation, and in accordance with a determination that the computer system is operating in a second mode different from the first mode, the separate prompt being a second prompt for displaying a second user interface, the second user interface enabling user authorization of the secure operation without the use of a physical input.

[0018] According to some embodiments, a computer system is described. The computer system is in communication with one or more input devices and a display generating component. The computer system includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for receiving a request via the one or more input devices for displaying a user interface associated with performing a secure operation, and displaying via the display generating component a first user interface including a separate prompt for providing additional input for authorizing the device to perform the secure operation in response to the request for displaying the user interface associated with performing the secure operation, wherein, in accordance with a determination that the computer system is operating in a first mode, the separate prompt is a first prompt providing a physical input for authorizing the secure operation, and in accordance with a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt for displaying a second user interface, the second user interface enabling user authorization of the secure operation without the use of a physical input.

[0019] According to some embodiments, a computer system is described, the computer system being in communication with one or more input devices and a display generating component, the computer system including means for receiving via the one or more input devices a request for display of a user interface associated with performing a secure operation, and means for displaying via the display generating component, in response to the request for display of the user interface associated with performing a secure operation, a first user interface including a separate prompt for providing additional input for authorizing the device to perform the secure operation, in accordance with a determination that the computer system is operating in a first mode, the separate prompt being a first prompt providing a physical input for authorizing the secure operation, and in accordance with a determination that the computer system is operating in a second mode different from the first mode, the separate prompt being a second prompt for displaying a second user interface, the second user interface enabling user authorization of the secure operation without the use of a physical input.

[0020] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, the specification, and the claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and description, and not for the purpose of defining or limiting the subject matter of the present invention. [Brief description of the drawings]

[0021] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0022] [Figure 1]FIG. 1 is a block diagram illustrating an operating environment for a computer system for providing an XR experience, according to some embodiments.

[0023] [Diagram 2] FIG. 1 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience, according to some embodiments.

[0024] [Diagram 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user, according to some embodiments.

[0025] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0026] [Diagram 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0027] [Figure 6A] FIG. 1 is a flow diagram illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0028] [Figure 6B] FIG. 1 illustrates exemplary devices connected via one or more communication channels, according to some embodiments.

[0029] [Figure 7A] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7B]FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7C] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7D] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7E] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7F] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7G] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7H] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7I] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7J] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7K] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7L] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7M] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments. [Figure 7N] FIG. 1 illustrates an example technique for allowing secure operation, according to some embodiments.

[0030] [Figure 8] FIG. 1 is a flow diagram of a method for facilitating user consent for secure operation, according to various embodiments.

[0031] [Figure 9]FIG. 1 is a flow diagram of a method of authentication continuity for secure operation according to various embodiments.

[0032] [Figure 10] FIG. 1 is a flow diagram of a method for allowing secure operation via an accessibility interface, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The present disclosure relates to a user interface that provides an extended reality (XR) experience to a user, according to some embodiments.

[0034] 1-6B provide an illustration of an example computer system for providing an XR experience to a user. FIGS. 7A-7N illustrate an example technique for authorizing secure operation, according to some embodiments. FIG. 8 is a flow diagram of a method for facilitating informed consent for secure operation, according to various embodiments. The process of FIG. 8 is described using the user interfaces of FIGS. 7A-7N. FIG. 9 is a flow diagram of a method of authentication continuity for secure operation, according to various embodiments. The process of FIG. 9 is described using the user interfaces of FIGS. 7A-7N. FIG. 10 is a flow diagram of a method for authorizing secure operation via an accessibility interface, according to various embodiments. The process of FIG. 10 is described using the user interfaces of FIGS. 7A-7N.

[0035] The process described below is directed to enhancing device operability and streamlining the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device and reducing user errors) through various techniques, including providing improved visual feedback to the user, reducing the number of inputs required to perform an action, adding control options without cluttering the user interface by adding displayed controls, performing an action without requiring further user input when a set of conditions are met, improving privacy and / or security, and / or other techniques. These techniques also reduce power usage and improve the device's battery life by allowing the user to use the device more quickly and efficiently.

[0036] Furthermore, in methods described herein in which one or more steps are contingent on one or more conditions being met, it should be understood that the described method can be repeated in multiple iterations such that over the course of the iterations, all of the conditions on which the steps of the method are contingent are met in different iterations of the method. For example, if a method requires performing a first step if a condition is met and performing a second step if the condition is not met, one skilled in the art would understand that the steps recited in the claim are repeated in a particular order until the conditions are met and are no longer met. Thus, a method described with one or more steps that are dependent on one or more conditions being met can be rewritten as a method that is repeated until each condition recited in the method is met. However, this is not required for system or computer readable medium claims in which the system or computer readable medium includes instructions to perform a conditional action based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency is met without explicitly repeating the steps of the method until all conditions on which the steps of the method are contingent are met. Those skilled in the art will also appreciate that, as with methods having conditional steps, the system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0037] In some embodiments, as shown in Fig. 1, an XR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generating component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a home appliance, a wearable device, etc.). In some embodiments, one or more of the input device(s) 125, output device(s) 155, sensors 190, and peripheral devices 195 are integrated with the display generation component 120 (e.g., within a head-mounted or handheld device).

[0038] When describing an XR experience, various terms are used to specifically refer to several related but distinct environments that a user may sense and / or interact with (e.g., with inputs detected by computer system 101 that cause the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the XR experience). The following are a subset of these terms:

[0039] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can sense and / or interact with the physical environment directly, such as through sight, touch, hearing, taste, and smell.

[0040] Extended reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In XR, a subset of a person's body motions or a representation thereof is tracked, and one or more properties of one or more virtual objects simulated in the XR environment are adjusted accordingly to behave with at least one law of physics. For example, an XR system may detect a person's head rotation and adjust the graphical content and sound field presented to the person accordingly, in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustment of a property(ies) of a virtual object(s) in an XR environment may be made in response to a representation of body motions (e.g., voice commands). A person may sense and / or interact with an XR object using any one of these senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person may sense and / or interact with only audio objects.

[0041] Examples of XR include virtual reality and mixed reality.

[0042] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a number of virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0043] Mixed reality: A mixed reality (MR) environment refers to a simulated environment designed to incorporate sensory input from or representations of a physical environment in addition to including computer-generated sensory input (e.g., virtual objects) as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between but not including a complete physical environment at one end and a virtual reality environment at the other end. In some MR environments, the computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track location and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system may take into account movement so that a virtual tree appears stationary relative to the physical ground.

[0044] Examples of mixed reality include extended reality and augmented virtuality.

[0045] Extended reality: An extended reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or semi-transparent display, whereby a person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment that are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. The person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, a video of a physical environment shown on an opaque display is referred to as a "pass-through video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An extended reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may transform one or more sensor images to impose a selected perspective (e.g., viewpoint) other than the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof, such that the modified portion becomes a modified, but non-photorealistic, version of the original captured image.As a further example, the representation of the physical environment may be altered by graphically removing or obscuring parts of it.

[0046] Augmented Virtuality: An augmented virtuality (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically reproduced from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0047] Perspective-locked virtual object: A virtual object is perspective-locked when the computer system displays the virtual object in the same location and / or position within the user's perspective, even as the user's perspective shifts (e.g., changes). In embodiments in which the computer system is a head-mounted device, the user's perspective is locked to the forward-facing orientation of the user's head (e.g., the user's perspective is at least a portion of the user's field of view when the user is looking straight ahead). Thus, the user's perspective remains fixed even as the user's line of sight moves without moving the user's head. In embodiments in which the computer system has a display generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's perspective is an extended reality view that is being presented to the user on the display generating component of the computer system. For example, a perspective-locked virtual object that is displayed in the upper left corner of the user's perspective when the user's perspective is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's perspective when the user's perspective changes to a second orientation (e.g., the user's head is facing west). In other words, the location and / or position at which a viewpoint-locked virtual object is displayed in a user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments in which the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head such that the virtual object is also referred to as a "head-locked virtual object."

[0048] Environment-Locked Virtual Object: A virtual object is environment-locked (or "world-locked") when a computer system displays the virtual object at a location and / or position in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., physical environment or virtual environment). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, resulting in the environment-locked virtual object being displayed at a different location and / or position in the user's viewpoint. For example, an environment-locked virtual object locked to a tree directly in front of the user will be displayed at the center of the user's viewpoint. If the user's viewpoint shifts to the right (e.g., the user's head is turned to the right) and the tree becomes more left in the user's viewpoint (e.g., the position of the tree in the user's viewpoint shifts), the environment-locked virtual object locked to the tree will be displayed more left in the user's viewpoint. In other words, the location and / or position at which the environment-locked virtual object is displayed in the user's viewpoint depends on the position and / or orientation of the location and / or object in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a coordinate system that is fixed to a fixed location and / or object in the physical environment) to determine a position at which to display an environment-locked virtual object in the user's viewpoint. The environment-locked virtual object can be locked to a stationary portion of the environment (e.g., a floor, wall, table, or other stationary object) or can be locked to a moving portion of the environment (e.g., a vehicle, an animal, a person, or a representation of a part of the user's body that moves independent of the user's viewpoint, such as the user's hand, wrist, arm, leg, etc.), such that the virtual object moves as the viewpoint or the part of the environment moves in order to maintain a fixed relationship between the virtual object and the part of the environment.

[0049] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits a delayed-following behavior that reduces or delays the motion of the environment-locked or viewpoint-locked virtual object relative to the movement of a reference point that the virtual object is following. In some embodiments, when exhibiting a delayed-following behavior, the computer system intentionally delays the movement of the virtual object when it detects movement of a reference point that the virtual object is following (e.g., a part of the environment, the viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 and 300 cm from the viewpoint). For example, when the reference point (e.g., a part of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops or slows down its movement, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits a delayed-following behavior, the device ignores small amounts of movement of the reference point (e.g., ignores movement of the reference point that is less than a threshold amount of movement, such as 0-5 degrees movement or 0-50 cm movement). For example, when the reference point (e.g., a viewpoint or portion of the environment to which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position with respect to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked) and when the reference point (e.g., a viewpoint or portion of the environment to which the virtual object is locked) moves by a second amount that is greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position with respect to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked) and then decreases as the amount of movement of the reference point increases beyond a threshold (e.g., a “lag following” threshold) as the virtual object is moved by the computer system to maintain a fixed or substantially fixed position with respect to the reference point.In some embodiments, a virtual object maintaining a substantially fixed position relative to a reference point includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the reference point in one or more dimensions (e.g., above / below, left / right, and / or forward / backward relative to the position of the reference point).

[0050] Hardware: There are many different types of electronic systems that allow a person to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated viewing capabilities, windows with integrated viewing capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display rather than an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to the person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In an embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology that projects a graphical image onto the person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the XR experience for the user.In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., an HMD, a display, a projector, a touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of one or more of the display generating components 120 (e.g., an HMD, or a portable electronic device including a display and one or more processors, etc.), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0051] In some embodiments, display generation component 120 is configured to provide an XR experience (e.g., at least a visual component of an XR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0052] According to some embodiments, the display generation component 120 provides an XR experience to the user while the user is virtually and / or physically present within the scene 105.

[0053] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). Thus, the display generating component 120 includes one or more XR displays provided to display XR content. For example, in various embodiments, the display generating component 120 surrounds the user's field of view. In some embodiments, the display generating component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, where the user holds a device with a display directed to the user's field of view and a camera directed to the scene 105. In some embodiments, the handheld device is optionally disposed in a housing worn on the user's head. In some embodiments, the handheld device is optionally disposed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generating component 120 is an XR chamber, housing, or room configured to present XR content without the user wearing or holding the display generating component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) can also be realized on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in the space in front of a handheld or tripod-mounted device can be realized similarly to an HMD where the interactions occur in the space in front of the HMD and the responses of the XR content are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) can be realized similarly to an HMD where the movement is caused by movement of the HMD relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0054] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0055] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communications interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communications buses 204 for interconnecting these and various other components.

[0056] In some embodiments, the one or more communications buses 204 include circuitry that interconnects and controls communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0057] The memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, the memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220, or the non-transitory computer-readable storage medium of the memory 220, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and an XR experience module 240:

[0058] The operating system 230 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for respective groups of one or more users). To that end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0059] 1 , and optionally one or more of the input devices 125, the output devices 155, the sensors 190, and / or the peripheral devices 195. To that end, in various embodiments, the data acquisition unit 241 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0060] In some embodiments, the tracking unit 242 is configured to map the scene 105 and track the position / location of at least the display generation components 120 relative to the scene 105 of FIG. 1 and optionally relative to one or more of the input devices 125, the output devices 155, the sensors 190, and / or the peripheral devices 195. To that end, in various embodiments, the tracking unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand and / or the motion of one or more parts of the user's hand relative to the scene 105 of FIG. 1, relative to the display generation components 120, and / or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 244 is described in more detail below with respect to FIG. 4. In some embodiments, eye tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)), or relative to XR content displayed via display generation component 120. Eye tracking unit 243 is described in more detail below with respect to FIG.

[0061] In some embodiments, the coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by the display generation components 120 and, optionally, by one or more of the output devices 155 and / or peripheral devices 195. To that end, in various embodiments, the coordination unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0062] In some embodiments, data dissemination unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments data dissemination unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0063] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 can be located within separate computing devices.

[0064] Moreover, Figure 2 is intended more to illustrate the functionality of various features that may be present in certain embodiments, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. For example, in various embodiments, some functional modules shown separately in Figure 2 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0065] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, and by way of non-limiting example, in some embodiments the display generating component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communications interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communications buses 304 for interconnecting these and various other components.

[0066] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0067] In some embodiments, the one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, the one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, the one or more XR displays 312 correspond to a waveguide display, such as diffractive, reflective, polarized, holographic, etc. For example, the display generating component 120 (e.g., HMD) includes a single XR display. In another example, the display generating component 120 includes an XR display for each eye of the user. In some embodiments, the one or more XR displays 312 can present MR or VR content. In some embodiments, the one or more XR displays 312 can present MR or VR content.

[0068] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene as the user would view it if the display generating component 120 (e.g., an HMD) were not present (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0069] The memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, the memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320, or the non-transitory computer-readable storage medium of the memory 320, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and an XR presentation module 340:

[0070] The operating system 330 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to a user via one or more XR displays 312. To that end, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0071] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0072] In some embodiments, the XR presentation unit 344 is configured to present XR content via one or more XR displays 312. To that end, in various embodiments, the XR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0073] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate an extended reality) based on the media content data. To that end, in various embodiments, the XR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0074] In some embodiments, data dissemination unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least controller 110, and optionally to one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, data dissemination unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0075] Although the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 are shown as being present on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0076] Moreover, FIG. 3 is intended more to illustrate the functionality of various features that may be present in a particular implementation, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in FIG. 3 can be implemented within a single module, and various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0077] 4 is a schematic diagram of an example embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (FIG. 1) is controlled by a hand tracking unit 244 (FIG. 2) to track the location / position of one or more parts of a user's hand and / or the motion of one or more parts of a user's hand relative to the scene 105 of FIG. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generating components 120, or relative to a coordinate system defined relative to a part of the user (e.g., the user's face, eyes, or head) and / or the user's hand). In some embodiments, the hand tracking device 140 is part of the display generating components 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generating components 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0078] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for distinguishing fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or images of the entire body, and can have either zoom capabilities or dedicated sensors with high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or serves as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0079] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data as well) to the controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives the display generation component 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving their hand 406 to change the posture of their hand.

[0080] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spots of the pattern. This approach is advantageous in that it does not require the user to hold or wear any kind of beacon, sensor, or other marker. It gives the depth coordinate of a point in the scene relative to a predefined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines a set of orthogonal x, y, z axes such that the depth coordinate of a point in the scene corresponds to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0081] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps including the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D locations of the user's wrist joints and fingertips.

[0082] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with motion tracking functionality, such that patch-based pose estimation is performed only once every two frames (or more), and tracking is used to find pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. The program can, for example, move and modify an image presented on the display generation component 120 or perform other functions in response to the pose and / or gesture information.

[0083] In some embodiments, the gesture includes an air gesture, which is a gesture that is detected without (or independent of) the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) and is based on detected motion of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) that is in the air, including motion of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), motion of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute motion of the user's body part (e.g., a tap gesture that includes movement of a hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture that includes a predetermined speed or amount of rotation of the user's body part).

[0084] In some embodiments, input gestures used in various examples and embodiments described herein include air gestures performed by movement of a user's finger(s) relative to other finger(s) or part(s) of the user's hand to interact with an XR environment (e.g., a virtual or mixed reality environment), according to some embodiments. In some embodiments, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and is based on detected motion of a part of the user's body, including motion of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), motion of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one of the user's hands, and / or movement of the user's finger relative to another finger or part of the user's hand), and / or absolute motion of a part of the user's body (e.g., a tap gesture that includes movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture that includes rotation of a part of the user's body a predetermined speed or amount).

[0085] In some embodiments where the input gesture is an air gesture (e.g., in the absence of physical contact with an input device that provides a computer system with information about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., in the case of direct input, as described below). Thus, in implementations that include air gestures, the input gesture is detected attention (e.g., gaze) to a user interface element in combination (e.g., simultaneously) with the movement of the user's finger(s) and / or hand to perform pinch and / or tap input, as described in more detail below.

[0086] In some embodiments, an input gesture directed at a user interface object is performed directly or indirectly with respect to the user interface object. For example, the user input is performed directly at the user interface object in concert with the user's hand performing the input gesture at a position corresponding to the position of the user interface object in the three-dimensional environment (e.g., determined based on the user's current viewpoint). In some embodiments, the input gesture is performed indirectly at the user interface object in concert with the user performing the input gesture while detecting the user's attention (e.g., gaze) to the user interface object while the position of the user's hand is not at a position corresponding to the position of the user interface object in the three-dimensional environment. For example, in the case of a direct input gesture, the user can direct the user's input to the user interface object by initiating the gesture at or near a position corresponding to the displayed position of the user interface object (e.g., within a distance of 0.5 cm, 1 cm, 5 cm, or 0-5 cm, measured from an outer edge of an option or a central portion of an option). For indirect input gestures, a user can direct the user's input to a user interface object by paying attention to the user interface object (e.g., by gazing at the user interface object), and while paying attention to the option, the user initiates an input gesture (e.g., in any position detectable by the computer system) (e.g., in a position that does not correspond to the displayed position of the user interface object).

[0087] In some embodiments, input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch inputs and tap inputs for interacting with a virtual or mixed reality environment, according to some embodiments. For example, pinch inputs and tap inputs, described below, are performed as air gestures.

[0088] In some embodiments, the pinch input is part of an air gesture, including one or more of a pinch gesture, a long pinch gesture, a pinch-and-drag gesture, or a double pinch gesture. For example, the pinch gesture, which is an air gesture, includes moving two or more fingers of a hand to contact each other, i.e., optionally with a short interruption (e.g., within 0-1 seconds) after contacting each other. The long pinch gesture, which is an air gesture, includes moving two or more fingers of a hand to contact each other for at least a threshold amount of time (e.g., at least 1 second) before detecting an interruption in contacting each other. For example, the long pinch gesture includes a user holding a pinch gesture (e.g., when two or more fingers are in contact), and the long pinch gesture continues until an interruption in contact between the two or more fingers is detected. In some embodiments, the double pinch gesture, which is an air gesture, includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected directly (e.g., within a predefined period of time) in succession to each other. For example, a user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaking contact between two or more fingers), and performs a second pinch input within a predetermined period of time (e.g., within 1 second or within 2 seconds) after releasing the first pinch input.

[0089] In some embodiments, a pinch-and-drag gesture that is an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., followed by) a drag input that changes the position of the user's hand from a first position (e.g., a start position of the drag) to a second position (e.g., an end position of the drag). In some embodiments, the user maintains the pinch gesture while performing the drag input and releases the pinch gesture (e.g., spreading two or more fingers apart) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers together to touch each other and moves the same hand to a second position in the air in a drag gesture). In some embodiments, the pinch input is performed by a first hand of the user and the drag input is performed by a second hand of the user (e.g., the second hand of the user moves from a first position to a second position in the air while the user continues the pinch input with the first hand of the user). In some embodiments, an input gesture that is an air gesture includes an input (e.g., a pinch input and / or a tap input) performed using both of a user's hands. For example, an input gesture includes two (e.g., or more) pinch inputs performed in conjunction with each other (e.g., simultaneously or within a predefined time period). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) performed using a first hand of a user and a second pinch input performed using the other hand (e.g., a second hand of the user's two hands) in conjunction with performing the pinch input using the first hand. In some embodiments, a movement between the user's hands (e.g., to increase and / or decrease the distance or relative orientation between the user's hands).

[0090] In some embodiments, a tap input performed as an air gesture (e.g., directed towards a user interface element) includes a movement of a user's finger(s) towards a user interface element, a movement of a user's hand towards a user interface element, optionally with the user's finger(s) extended towards the user interface element, a downward motion of the user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture, moving the finger or hand away from the user's viewpoint and / or towards the object that is the target of the tap input followed by an end of the movement. In some embodiments, an end of the movement is detected based on a change in movement characteristics of the finger or hand performing the tap gesture (e.g., an end of the movement away from the user's viewpoint and / or towards the object that is the target of the tap input, a reversal of the finger or hand movement direction, and / or a reversal of the acceleration direction of the finger or hand movement).

[0091] In some embodiments, the user's attention is determined to be directed to a portion of the three-dimensional environment based on detection of a gaze directed to the portion of the three-dimensional environment (optionally without requiring other conditions). In some embodiments, the user's attention is determined to be directed to a portion of the three-dimensional environment based on detection of a gaze directed to the portion of the three-dimensional environment with one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., dwell time) while the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment for the device to determine that the user's attention is directed to the portion of the three-dimensional environment, and / or requiring the gaze to be directed to the portion of the three-dimensional environment, and if one of the additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until one or more additional conditions are met).

[0092] In some embodiments, detection of a ready state configuration of a user or a portion of a user is detected by the computer system. Detection of a ready state configuration of the hands is used by the computer system as an indication that the user is likely preparing to interact with the computer system using one or more air gesture inputs performed by the hands (e.g., a pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of a hand is determined based on whether the hand has a predefined hand geometry (e.g., a pre-pinch geometry with the thumb and one or more fingers extended and spaced apart, ready to perform a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), whether the hand is in a predefined position relative to the user's viewpoint (e.g., below the user's head, above the user's waist, extended at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular manner (e.g., above the user's waist, moved toward an area in front of the user below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface is responsive to attentional (e.g., gaze) input.

[0093] In scenarios where input is described with reference to air gestures, it will be understood that similar gestures may also be detected using a hardware input device attached to or held by one or more of the user's hands, where the position of the hardware input device in space may be tracked using optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers, and / or one or more inertial measurement units, and the position and / or movement of the hardware input device is substituted for the position and / or movement of the hand or hands in the corresponding air gesture(s). In scenarios where input is described with reference to air gestures, it should be understood that similar gestures can also be detected using a hardware input device attached to or held by one or more of the user's hands, where user input can be detected using controls included in the hardware input device, such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, one or more hand or finger covers, and / or other hardware input device controls that can detect the position or change in position of parts of the hands and / or fingers relative to each other, relative to the user's body, and / or relative to the user's physical environment, where user input using controls included in the hardware input device is used in place of hand and / or finger gestures such as an air tap or air pinch in the corresponding air gesture(s). For example, a selection input described as being made with an air tap or air pinch input can alternatively be detected with a button press, a tap on a touch-sensitive surface, a press on a pressure-sensitive surface, or other hardware input.As another example, the movement input described as being provided by an air pinch and drag may alternatively be detected based on interaction with a hardware input control, such as a press and hold of a button, a touch on a touch-sensitive surface, a press on a pressure-sensitive surface, or based on hardware input following the movement of another hardware input device in space (e.g., accompanying the hand with which the hardware input device is associated). Similarly, two-handed input, including the movement of both hands relative to one another, may be provided using one air gesture and one hardware input device held in the hand not performing the air gesture, two hardware input devices held in separate hands, or two air gestures performed by separate hands, using various combinations of air gestures and / or input detected by one or more of the hardware input devices described above.

[0094] In some embodiments, the software may be downloaded in electronic form to the controller 110, for example over a network, or may alternatively be provided on a tangible non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in a memory associated with the controller 110. Alternatively or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in FIG. 4 as a separate unit from the image sensor 404, by way of example, some or all of the processing functionality of the controller may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit within the housing of the image sensor 404 (e.g., a hand tracking device), or in another manner. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generating component 120 (e.g., in a television set, handheld device, or head-mounted device), or with any other suitable computerized device, such as a game console or media player. The sensing functions of the image sensor 404 may similarly be integrated into a computer or other computerized device that is controlled by the sensor output.

[0095] FIG. 4 further includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels with respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and the wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with darker shades increasing with increasing depth. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics may include, for example, the overall size, shape, and frame-to-frame motion of the sequence of depth maps.

[0096] 4 also illustrates, in accordance with some embodiments, a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406. In FIG. 4, the hand skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand that connects to the wrist, etc.) and optionally the wrist or arm connected to the hand are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0097] FIG. 5 illustrates an exemplary embodiment of the eye tracking device 130 (FIG. 1). In some embodiments, the eye tracking device 130 is controlled by the eye tracking unit 243 (FIG. 2) to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generating component 120. In some embodiments, the eye tracking device 130 is integrated with the display generating component 120. For example, in some embodiments, if the display generating component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed in a wearable frame, the head-mounted device includes both a component for generating XR content for viewing by the user and a component for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generating component 120. For example, if the display generating component is a handheld device or an XR chamber, the eye tracking device 130 is, optionally, a device separate from the handheld device or the XR chamber. In some embodiments, the eye tracking device 130 is a head-mounted device or part of the head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with head-mounted or non-head-mounted display generating components. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally used in combination with head-mounted display generating components. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally part of non-head-mounted display generating components.

[0098] In some embodiments, the display generating component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generating component may include left and right optical lenses (referred to herein as eyepieces) located between the display and the user's eyes. In some embodiments, the display generating component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generating component may have a transparent or semi-transparent display that allows the user to view the physical environment directly and display virtual objects on the transparent or semi-transparent display. In some embodiments, the display generating component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, whereby an individual using the system can observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0099] As shown in FIG. 5, in some embodiments, eye tracking device 130 (e.g., gaze tracking device) includes at least one eye tracking camera (e.g., infrared (IR) camera or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye tracking camera may be aimed at the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, may be aimed at a "hot" mirror disposed between the user's eye and a display panel that reflects the IR or NIR light from the eye to the eye tracking camera while allowing visible light to pass through. Eye tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to controller 110. In some embodiments, the user's eyes are tracked separately by respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a separate eye tracking camera and lighting source.

[0100] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for the particular operating environment 100, e.g., 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility before delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. The user-specific calibration process may include estimation of the particular user's eye parameters, e.g., pupil location, central vision location, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters are determined for the eye tracking device 130, images captured by the eye tracking camera can be processed using a glint-assisted method to determine the user's current visual axis and gaze point relative to the display.

[0101] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and a gaze tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be located between the user's eye(s) 592 and the display 510 (e.g., a left or right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) and may be pointed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown in the top of FIG. 5), or may be pointed at the user's eye(s) 592 to receive reflected IR or NIR light from the eye(s) 592 (e.g., as shown in the bottom of FIG. 5).

[0102] In some embodiments, the controller 110 renders the AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses gaze tracking input 542 from the eye tracking camera 540 for various purposes, e.g., in processing the frames 562 for display. The controller 110 optionally estimates the user's point of gaze on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540, using a glint-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0103] Below, some possible use cases of the user's current gaze direction are described, which are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 can generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller can position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller can display certain virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the XR experience and focus in the determined direction. The autofocus mechanism of the external camera can then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another exemplary use case, the eyepiece 520 may be a focusable lens, and the eye tracking information is used by the controller to adjust the focus of the eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. The controller 110 can leverage the eye tracking information to orient and focus the eyepiece 520 so that a nearby object the user is looking at appears at the correct distance.

[0104] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR LED or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR light or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0105] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR ranges, and therefore does not introduce noise into the eye tracking system. Note that the locations and angles of the eye tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0106] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0107] FIG. 6A illustrates a glint-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is realized by a glint-assisted gaze tracking system (e.g., eye tracking device 130 as shown in FIG. 1 and FIG. 5). The glint-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted gaze tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted gaze tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0108] As shown in FIG. 6A, an eye-tracking camera may capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system may continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0109] At 610, if the tracking status is "yes" for the currently captured image, the method proceeds to element 640. If the tracking status is "no" at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eyes.

[0110] At 640, proceeding from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from previous frames. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to verify that the tracking or detection results are reliable. For example, the results may be checked to determine whether the pupils and a sufficient number of glints to perform gaze estimation have been successfully tracked or detected in the current frame. At 650, if the results are not reliable, the tracking state is set to "no" at element 660 and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to "yes" (if not already "yes") and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0111] 6A is intended to serve as an example of eye-tracking technology that may be used in certain implementations. As will be appreciated by those skilled in the art, other eye-tracking technologies, either currently existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with an XR experience according to various embodiments.

[0112] 6B illustrates exemplary devices connected via one or more communication channels to participate in a transaction, according to some embodiments. One or more exemplary electronic devices (e.g., devices 602 and 604) are optionally configured to detect input (e.g., specific user input, NFC field) and optionally transmit payment information (e.g., using NFC). One or more electronic devices optionally include NFC hardware and are configured to be NFC enabled.

[0113] The electronic device (e.g., devices 602 and 604) is optionally configured to store payment account information associated with each of one or more payment accounts. The payment account information includes, for example, one or more of a personal or company name, a billing address, a login, a password, an account number, an expiration date, a security code, a phone number, a bank associated with the payment account (e.g., an issuing bank), and a card network identifier. In some embodiments, the payment account information includes an image, such as a photo of a payment card (e.g., taken by and / or received at the device). In some embodiments, the electronic device receives a user input including at least some payment account information (e.g., receives a credit, debit, account, or gift card number and expiration date entered by a user). In some embodiments, the electronic device detects at least some payment account information from an image (e.g., a payment card captured by a camera sensor of the device). In some embodiments, the electronic device receives at least some payment account information from another device (e.g., another user device or a server). In some embodiments, the electronic device receives payment account information from a server associated with another service (e.g., an app for renting or selling audio and / or video files) for which the user or user device or account for the identified payment account data previously made a purchase.

[0114] In some embodiments, a payment account is added to an electronic device (e.g., devices 602 and 604) such that the payment account information is securely stored on the electronic device. In some embodiments, after a user initiates such a process, the electronic device sends information for the payment account to a transaction coordination server, which then communicates with a server operated by the payment network for the account (e.g., a payment server) to ensure the validity of the information. The electronic device is optionally configured to receive a script from the server that enables the electronic device to program the payment information for the account onto the secure element.

[0115] In some embodiments, communication between electronic devices 602 and 604 facilitates a transaction (e.g., a general or specific transaction). For example, a first electronic device (e.g., 602) can act as a provisioning or management device and can send notifications of new or updated payment account data (e.g., information about new accounts, updated information about existing accounts, and / or alerts for existing accounts) to a second electronic device (e.g., 604). In another example, a first electronic device (e.g., 602) can send data to a second electronic device that reflects information about a payment transaction facilitated at the first electronic device. The information optionally includes one or more of the payment amount, the account used, the time of purchase, and whether the default account has been changed. The second device (e.g., 604) optionally uses such information to update the default payment account (e.g., based on a learning algorithm or explicit user input).

[0116] The electronic devices (e.g., 602, 604) are configured to communicate with each other over any of a variety of networks. For example, the devices communicate using a Bluetooth connection 616 (including, for example, a conventional Bluetooth connection or a Bluetooth low energy connection) or a WiFi network 614. Communications between user devices are optionally coordinated to reduce the possibility of inappropriate sharing of information between devices. For example, communications related to payment information require the communicating devices to be paired (e.g., associated with each other via explicit user interaction) or associated with the same user account.

[0117] In some embodiments, the electronic devices (e.g., 602, 604) are optionally used to communicate with a point-of-sale (POS) payment terminal 606 that is NFC enabled. This communication is optionally performed using various communication channels and / or technologies. In some embodiments, the electronic devices (e.g., 602, 604) communicate with the payment terminal 606 using an NFC channel 618. In some embodiments, the payment terminal 606 communicates with the electronic devices (e.g., 602, 604) using a peer-to-peer NFC mode. The electronic devices (e.g., 602, 604) are optionally configured to transmit a signal to the payment terminal 606 that includes payment information for a payment account (e.g., a default account or an account selected for a particular transaction).

[0118] In some embodiments, proceeding with the transaction includes transmitting a signal including payment information for an account, such as a payment account. In some embodiments, proceeding with the transaction includes reconfiguring the electronic device (e.g., 602, 604) to respond as a contactless payment card, such as an NFC-enabled contactless payment card, and then transmitting the account credentials via NFC to, such as, the payment terminal 606. In some embodiments, after transmitting the account credentials via NFC, the electronic device reconfigures to not respond as a contactless payment card (e.g., authorization is required before reconfiguring to respond as a contactless payment card via NFC again).

[0119] In some embodiments, the generation and / or transmission of the signal is controlled by a secure element in the electronic device (e.g., 602, 604). The secure element optionally requires a specific user input before releasing the payment information. For example, the secure element optionally requires detection of the electronic device being worn, detection of a button press, detection of the entry of a passcode, detection of a touch, detection of one or more option selections (e.g., received during interaction with an application), detection of a fingerprint signature, detection of a voice or voice command, and / or detection of a gesture or movement (e.g., rotation or acceleration). In some embodiments, if a communication channel (e.g., an NFC communication channel) is established with another device (e.g., payment terminal 606) within a defined period of time from when the input is detected, the secure element releases the payment information to be transmitted to the other device (e.g., payment terminal 606). In some embodiments, the secure element is a hardware component that controls the release of the secure information. In some embodiments, the secure element is a software component that controls the release of the secure information.

[0120] In some embodiments, the protocols involved in engaging in a transaction depend, for example, on the device type. For example, the conditions for generating and / or transmitting payment information may be different for a wearable device (e.g., device 604) than for a phone (e.g., device 602). For example, the conditions for generating and / or transmitting on a wearable device may include detecting a button press (e.g., after security verification), while the conditions for a phone may not require a button press, but instead detection of a specific interaction with an application. In some embodiments, the conditions for transmitting and / or publishing payment information include receiving a specific input on each of a plurality of devices. For example, publishing payment information may optionally require detection of a fingerprint and / or passcode on a device (e.g., device 602) and detection of a mechanical input (e.g., a button press) on another device (e.g., device 604).

[0121] The payment terminal 606 optionally uses the payment information to generate a signal that it sends to a payment server 612 to determine if the payment is authorized. The payment server 612 optionally includes any device or system configured to receive payment information associated with a payment account and determine if the proposed purchase is authorized. In some embodiments, the payment server 612 includes an issuing bank's server. The payment terminal 606 communicates with the payment server 612 directly or indirectly via one or more other devices or systems (e.g., an acquiring bank's server and / or a card network's server).

[0122] The payment server 612 optionally uses at least a portion of the payment information to identify a user account in a database of user accounts (e.g., 608). For example, each user account includes payment information. The account is optionally located by locating an account with particular payment information that matches the payment information from the POS communication. In some embodiments, the payment is rejected when the provided payment information is inconsistent (e.g., an expiration date does not correspond to a credit, debit, or gift card number) or when there is no account with payment information that matches the payment information from the POS communication.

[0123] In some embodiments, the data for the user account further identifies one or more limitations (e.g., credit limit), current or previous balance, previous transaction date, location, and / or amount, account status (e.g., active or frozen), and / or authorization instructions. In some embodiments, the payment server (e.g., 612) uses such data to determine whether to authorize a payment. For example, the payment server denies a payment when the purchase amount added to the current balance would exceed an account limit, when the account is frozen, when previous transaction amounts exceed a threshold, or when the number or frequency of previous transactions exceed a threshold.

[0124] In some embodiments, the payment server 612 responds to the POS payment terminal 606 with an indication as to whether the proposed purchase was permitted or denied. In some embodiments, the POS payment terminal 606 sends a signal to the electronic device (e.g., 602, 604) to identify the outcome. For example, the POS payment terminal 606 sends a receipt to the electronic device (e.g., 602, 604) when the purchase was permitted (e.g., via a transaction coordination server that manages the transaction app on the user device). In some cases, the POS payment terminal 606 presents an output (e.g., a visual or audio output) indicating the outcome. Payment may be sent to the merchant as part of the authorization process or may be sent at a later time.

[0125] In some embodiments, the electronic device (e.g., 602, 604) engages in a transaction that is completed without the intervention of the POS payment terminal 606. For example, upon detecting that machine input has been received, a secure element within the electronic device (e.g., 602, 604) may expose payment information to allow an application on the electronic device to access the information (e.g., transmit the information to a server associated with the application).

[0126] In some embodiments, the electronic device (e.g., 602, 604) is in a locked or unlocked state. In the locked state, the electronic device is powered on and operational, but is prevented from performing a predefined set of actions in response to user input. The predefined set of actions may include navigation between user interfaces, activation or deactivation of a predefined set of features, and activation or deactivation of specific applications. The locked state may be used to prevent unintended or unauthorized use of some functionality of the electronic device, or activation or deactivation of some features on the electronic device. In the unlocked state, the electronic device 602 is powered on and operational, and is not prevented from performing at least a portion of the predefined set of actions that cannot be performed in the locked state.

[0127] When a device is in a locked state, the device is said to be locked. In some embodiments, a device in a locked state may respond to a limited set of user inputs, including inputs corresponding to an attempt to transition the device from an unlocked state or inputs corresponding to powering off the device.

[0128] In some embodiments, the secure element is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms. In some embodiments, the secure element provides (or exposes) the payment information (e.g., an account number and / or a transaction-specific dynamic security code). In some embodiments, the secure element provides (or exposes) the payment information in response to the device receiving authorization such as user authentication (e.g., fingerprint authentication, passcode authentication, detecting a double press of a hardware button by providing authentication credentials to the device when the device is in an unlocked state and optionally when the device is continuously on the user's wrist from the time the device was unlocked, etc., where the continued presence of the device on the user's wrist is determined by periodically checking whether the device is in contact with the user's skin). For example, the device detects a fingerprint with a fingerprint sensor (e.g., a fingerprint sensor integrated into a button) of the device. The device determines whether the fingerprint matches a registered fingerprint. In response to a determination that the fingerprint matches the registered fingerprint, the secure element provides (or exposes) the payment information. Following a determination that the fingerprint does not match a registered fingerprint, the secure element refrains from providing (or releasing) the payment information.

[0129] In the present disclosure, various input methods are described with respect to interaction with a computer system. If one example is provided using one input device or input method and another example is provided using another input device or input method, it should be understood that each example may be compatible with and optionally utilize the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. If one example is provided using one output device or output method and another example is provided using another output device or output method, it should be understood that each example may be compatible with and optionally utilize the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. If one example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with and optionally utilize the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User Interface and Related Processes

[0130] Attention now turns to embodiments of user interfaces (UIs) and associated processes that may be implemented in a computer system, such as a portable multifunction device or a head-mounted device, in communication with a display generating component, one or more input devices, and (optionally) a biometric sensor.

[0131] Figures 7A-7N illustrate examples of authorizing secure operation. Figure 8 is a flow diagram of an example method 800 for facilitating user consent for secure operation. Figure 9 is a flow diagram of a method 900 of continuity of authentication for secure operation. Figure 10 is a flow diagram of an example method 1000 for authorizing secure operation via an accessibility interface. The user interfaces of Figures 7A-7N are used to illustrate the processes described below, including the processes of Figures 8, 9, and 10.

[0132] FIG. 7A illustrates an electronic device 700 including a display 700a. In FIG. 7A, a user 704 is interacting with the electronic device 700. The electronic device 700 may correspond to a tablet device, a wearable device (e.g., a head-mounted display), a smartphone, and / or a smartwatch. In addition, the display 700a may include or be coupled to a display generating component (e.g., a display controller, a touch-sensitive display system, a display (e.g., built-in and / or connected), a 3D display, a transparent display, a projector, and / or a head-up display). The electronic device 700 displays one or more interface objects on the display 700a, such as a display login user interface 702. In some embodiments, the login user interface 702 corresponds to a virtual interface object in the XR environment. The login user interface 702 optionally includes various display fields, such as a username field 706 and a password field 708. The login user interface 702 also optionally includes affordances 710 for initiating a process of a secure action, such as authorization of various displayed fields (e.g., “autofill”). In some embodiments, the secure action corresponds to a payment, such as providing payment information to a third party service to purchase an item, such as providing credit card information to an e-commerce website to purchase shoes (as described in more detail with respect to FIG. 7N). In some embodiments, the secure action corresponds to providing access credentials to access an application. For example, the secure action provides access credentials to an application associated with a third party service (e.g., a stock trading application) that requires user authentication before accessing that application. In some embodiments, the electronic device 700 allows the user 704 to manually enter information into fields 706 and / or 708 (e.g., using a displayed keyboard and / or using voice commands).

[0133] In some embodiments, as shown in FIG. 7A , the electronic device 700 detects a selection of an affordance 710 by a user 704 to initiate an autofill function that automatically fills in information in fields 706 and / or 708. For example, the display 700a is a touch-sensitive display such that the electronic device 700 detects touch input (e.g., a tap or a tap-and-hold) on the affordance 710. In some embodiments, when the device 700 is implemented in an XR environment, the electronic device 700 optionally detects the selection of the affordance 710 using a controller communicatively coupled to the electronic device 700. Additionally, the electronic device 700 detects the selection of the affordance 710 using a gaze, for example, by detecting that the user 704 is looking at the affordance 710 for a predetermined period of time and / or (optionally) while detecting one or more gestures. For example, electronic device 700 optionally corresponds to a head mounted display, such that electronic device 700 detects that user 704 is performing one or more gestures, such as an air gesture, to activate affordance 710. Electronic device 700 optionally displays additional / other elements on display 700a, such as additional user interfaces (e.g., an email application), representations of other users (e.g., via a video call, an avatar of another user in an XR environment), and / or representations of a physical environment, such as the physical environment of user 704.

[0134] In FIG. 7B, in response to activation of affordance 710, electronic device 700 displays authorization interface 712. In some embodiments, authorization interface 712 corresponds to a displayed virtual interface object in the XR environment. Authorization interface 712 includes search function 714 and one or more credential options 716a and 716b. Credential option 716a is associated with (e.g., used to select) information, such as, for example, username "janeapples@mail.com" to log into website "abc.com". In some embodiments, electronic device 700 detects that a credential option, such as credential option 716b (shown in FIG. 7B), has been selected. Credential option 716b is associated with information, such as, for example, username "janeapples@mail.com" to log into website "xyz.com". In some embodiments, the credential option is associated with a separate password, which may be displayed as part of authorization interface 712 or may not be displayed to protect privacy. In some embodiments, the electronic device 700 receives information (e.g., text and / or voice) via the search functionality 714 and, in response, displays one or more search result options that correspond to the received information for use in the autofill functionality.

[0135] In some embodiments, various requirements (a set of one or more criteria) must be met for the electronic device 700 to enable user authorization of a secure operation, including a visibility criterion and / or a user authentication criterion. For example, the visibility criterion is met when a threshold amount of one or more interface objects, such as the login user interface 702 and / or the authorization interface 712, are visible from the perspective of the user 704 (e.g., the threshold amount may correspond to any portion of the interface object, a non-zero amount of the interface object, or the entire interface object). In addition, the user authentication criterion is met when the electronic device 700 receives a request to perform a secure operation. For example, the electronic device 700 detects the selection by the user 704 of a displayed individual credential option, such as the credential option 716b (as shown in FIG. 7B). Once the credential option is selected, the user 704 may provide an input (e.g., a double press of a hardware button) that authorizes the performance of the secure operation, as described in more detail with respect to FIG. 7D. Pursuant to determining that input permitting performance of a secure operation has been received, the electronic device 700 causes the individual credential information to be automatically filled into displayed fields, such as the username field 706 and password field 708, using the individual credential option, provided that the set of one or more criteria is met. In FIG. 7B, the electronic device 700 receives a selection of credential option 716b.

[0136] In response to receiving a selection of the credentials option 716b in FIG. 7C, the electronic device 700 updates the authorization interface as shown in FIG. 7C. In FIG. 7C, the electronic device 700 detects a user gaze direction 718 that is related to (e.g., corresponds to) a gaze direction of the user 704. Typically, the user gaze direction 718 is not part of the displayed user interface of the electronic device 700, but is provided to aid in the description of the present technique. While the user 704 is interacting with the device 700, the electronic device 700 detects a change in the user's 704 perspective (e.g., a change from a first perspective to a second perspective different from the first perspective) based on a change in the user gaze direction 718. In some embodiments, the user gaze direction is detected within the context of an XR environment (e.g., via a head-mounted display). The user gaze direction 718 optionally indicates that the user's 704 gaze is not directed toward the login user interface 702 or the authorization interface 712. Thus, the electronic device 700 determines that the login user interface 702 and the permission interface 712 are sufficiently invisible from the viewpoint of the user 704. For example, the user gaze direction 718 may be directed toward a portion of the display 700a that is not proximate to the location where the login user interface 702 and the permission interface 712 are displayed. In some embodiments, portions of the login user interface 702 or the permission interface 712 that are within a predetermined distance (e.g., a predetermined number of pixels and / or a predetermined length) from the center of the user gaze direction 718 are determined to be visible from the viewpoint of the user 704. For example, 10% of the permission interface 712 may be determined to be within a predetermined distance from the center of the user gaze direction 718, and 5% of the login user interface 702 may be determined to be within a predetermined distance from the center of the user gaze direction 718. In this example, the electronic device 700 determines that less than a threshold amount (e.g., a 75% threshold amount or a 90% threshold amount) of the login user interface 702 and / or the permission interface 712 is visible from the viewpoint of the user.

[0137] 7C , pursuant to a determination that less than a threshold amount of the login user interface 702 and / or the permission interface 712 are visible from the user's perspective, the electronic device 700 disables and / or deactivates the user's permission of the autofill function. In some embodiments, the electronic device 700 modifies the appearance of one or more interface objects to indicate that the user cannot request performance of a secure operation (e.g., by graying out the permission interface 712). For example, in response to detecting a selection of the credentials option 716b, the electronic device 700 does not automatically fill in the username field 706 and the password field 708 while not enough of the login user interface 702 and the permission interface 712 are visible from the user's perspective (e.g., based at least on the user gaze direction 718).

[0138] In Fig. 7D, the electronic device 700 determines that the user 704 is authorized to perform a secure operation, and thus, user permission of the autofill function is enabled. In some examples, the user permission of the secure operation is enabled conditioned on determining that the user 704 is authorized to perform the secure operation. For example, the electronic device 700 determines that the user 704 is authorized to perform the secure operation based on gaze criteria or other biometric criteria. The gaze criteria is optionally met when the user 704 gazes at a separate interface object, such as the login user interface 702 and / or the permission interface 712. Optionally, the login user interface 702 and / or the permission interface 712 are displayed in a central portion of the display 700a. In some embodiments, the electronic device 700 determines that the user 704 is not authorized to perform a secure operation when the user is not gazing at the login user interface 702 and / or the authorization interface 712 (e.g., the user's eyes are closed or the user is looking at a corner of the display 700a (away from the interfaces 702 and / or 712)). In some embodiments, user authorization of a secure operation is enabled conditioned on a determination that the user has or provides a particular motion of a biometric feature. For example, such a particular motion may include the user gazing at a particular portion of the login user interface 702 (e.g., the affordance 710) or a particular portion of the authorization interface 712 (e.g., a particular credential option), the user providing a particular facial rotation, and / or the user moving a finger along a particular path.

[0139] Typically, the electronic device 700 operates in various modes. For example, the electronic device 700 optionally operates in a first mode, such as a standard mode in which accessibility features are not enabled and / or a mode in which secure operation is permitted using a default physical input mechanism, such as a hardware button 722 (as described with respect to FIG. 7D). In some embodiments, the device 700 optionally operates in a second mode, such as an accessibility mode and / or a mode in which input with a physical input mechanism is not required to permit secure operation (as described in more detail with respect to FIGS. 7L-7M). Returning to FIG. 7D, pursuant to a determination that at least a threshold amount of the login user interface 702 and / or the permission interface 712 is visible from the perspective of the user 704, the electronic device 700 enables user permission of the autofill function. For example, the user gaze direction 718 may be directed toward the permission interface 712. Thus, the target of the user's gaze is determined to be the permission interface 712, resulting in a determination that 95% or 100% of the permission interface 712 is visible from the viewpoint of the user 704. As a result, the electronic device 700 determines that at least a threshold amount (e.g., a 75% threshold amount or a 90% threshold amount) of the login user interface 702 and / or the permission interface 712 is visible from the viewpoint of the user. In some embodiments, the electronic device 700 then enables user permission of the secure operation and indicates to the user that user permission of the secure operation has been enabled via an indication 720 that includes a textual indicator that reads "double-click to allow." Specifically, the indication 720 informs the user that the autofill function is permitted by activating a hardware button, such as hardware button 722, with a double press. In some embodiments, user permission for the autofill feature remains enabled for a period of time (e.g., up to 1 second, up to 3 seconds, etc.) after it is determined that the login user interface 702 and / or the permission interface 712 are visible from the user's perspective, and is optionally disabled after that period of time.In some embodiments, the device 700 visually indicates that the viewpoint includes the permissions interface 712 and therefore its individual permissions are enabled (e.g., by un-graying the permissions interface 712 and / or a button or the like of the permissions interface 712).

[0140] If the autofill permission is enabled, the user 704 may proceed to allow a secure operation. In particular, the electronic device 700 optionally modifies the appearance of one or more interface objects and / or displays one or more additional interface objects to indicate to the user that the user can request to perform a secure operation. For example, if the electronic device 700 is implemented in an XR environment, the electronic device 700 displays an indication, such as instructions to perform one or more air gestures (e.g., instructions to move one or more hands or fingers in a particular motion) and / or instructions to perform one or more inputs in a controller compatible with the XR environment.

[0141] In FIG. 7D , the electronic device 700 receives user authorization for the autofill function via a double click of the hardware button 722. In response to receiving user authorization for the autofill function, a determination is optionally made as to whether a set of one or more performance criteria is met. For example, visibility may be required to be met before, upon, and / or after receiving a user authorization request to perform a secure operation. In some embodiments, the one or more performance criteria optionally also include a requirement that at least a threshold amount of the login user interface 702 and / or the authorization interface 712 are visible from the perspective of the user 704 after the user 704 requests to perform a secure operation. In some embodiments, the set of one or more performance criteria includes a user authentication criterion that is met when the user is authenticated (e.g., via a password and / or biometric authentication). 7F, the criteria optionally include a positive indication of a first level (e.g., full or high fidelity) iris feature verification, a positive indication of continued (e.g., second level) iris feature verification from the time the user 704 begins interacting with the device 700, and / or a positive indication of proper passcode authentication. As a result, upon receiving a request to perform a secure operation, pursuant to a determination that a set of one or more performance criteria have been met, the electronic device 700 begins performing the secure operation.

[0142] In FIG. 7E, the electronic device 700 provides an indication to the user 704 that user authorization of the autofill feature was successful. Specifically, the electronic device 700 modifies (or replaces with an additional interface) the authorization interface 712 to include a positive indicator that user authorization of the autofill feature was successful (e.g., a "check" mark next to the text "Done"). In addition, the electronic device 700 updates the login user interface 702 to indicate successful user authorization of the autofill feature by updating the username field 706 to include the username of the username corresponding to the selected credential option and updating the password field 708 to include obscured characters that conceal the password corresponding to the selected credential option. In some embodiments, the electronic device 700 does not conceal the password corresponding to the selected credential option and updates the password field 708 to include readable characters.

[0143] FIG. 7F illustrates a user authentication process. Specifically, continuing from FIG. 7D, in response to receiving a request to perform a secure operation, a determination is made as to whether the user 704 meets user authentication criteria to perform the secure operation. In some embodiments, a determination is made that a set of execution criteria is not met. Typically, a first type of biometric authentication is performed at a first time (e.g., a first time of device interaction), so that the electronic device 700 biometrically authenticates the user 704 using one or more biometric sensors of the device 700. The first time may correspond to the first time when the user activates the electronic device 700 and / or when the user wears the electronic device 700 on a part of the user's body. In an embodiment in which the device 700 corresponds to a head-mounted display, the first time may correspond to the time when the user 704 secures / wears the electronic device 700 on the user's 704's head. The first type of biometric authentication may include a first level (e.g., full or high fidelity) of iris feature verification. The first level of iris feature verification includes a retinal scan to obtain one or more images and / or measurements of at least one eye of the user 704. In some embodiments, different users have different iris features. The one or more images and / or measurements are then compared to one or more stored images and / or measurements corresponding to the registered retinal scan of the user to determine a match and / or similarity between the obtained image(s) / measurements and the registered image(s) / measurements. If the obtained image(s) / measurements and the registered image(s) match with sufficient similarity (e.g., 95% or 99% similarity), the electronic device 700 determines that the user 704 is authenticated. In some embodiments, the first type of biometric authentication optionally includes (in addition to or instead of) iris feature verification, fingerprint verification, facial recognition verification, and / or voiceprint verification.

[0144] The user authentication criteria used to determine whether to perform a secure operation is optionally based on sensor measurements taken during a particular time frame, specifically after an initial time when the user 704 starts interacting with the electronic device 700 and / or at intermediate times after the user 704 wears the electronic device 700. The sensor measurements optionally correspond to measurements taken to detect whether the same user (e.g., the user 704) is using the device 700 throughout the duration of the interaction. For example, in an embodiment in which the electronic device 700 corresponds to a head-mounted display, the electronic device 700 optionally authenticates the user 704 via a first type of biometric authentication (e.g., full or high fidelity) when the user 704 attaches / wears the electronic device 700 on the user 704's head. If authentication via the first type of biometric authentication is successful, the electronic device 700 repeats and takes sensor measurements using a second type of biometric authentication. In some embodiments, the first type of biometric authentication corresponds to a high-fidelity biometric verification, such that the electronic device 700 analyzes characteristics associated with the entire eye of the user 704 or a portion of the eye of the user 704, such as an iris feature. In some embodiments, the second type of biometric authentication corresponds to one or more sensor measurements that directly or indirectly verify the continued presence of a biometric feature from the time the high-fidelity biometric verification was performed for that biometric feature. The second type of biometric authentication optionally analyzes the same (or a similar) amount of biometric features as the first type of biometric authentication, or optionally analyzes fewer biometric features compared to the first type of biometric authentication. In some embodiments, the second type of biometric authentication may be referred to as a continuity verification, since the second type of biometric authentication is used to verify that a user authenticated using the first type of biometric authentication has remained a user of the electronic device 700 (e.g., without interrupting the continuity of use of the biometric features) since the first type of biometric authentication was performed.

[0145] In some embodiments, both the first type and the second type of biometric authentication are enabled while the electronic device 700 is operating in a first mode (e.g., a standard mode in which accessibility features are not enabled and / or a mode that allows secure operation using a default physical input mechanism). In some embodiments, the first type of biometric authentication is enabled and the second type of biometric authentication is disabled while the electronic device 700 is operating in a second mode (e.g., an accessibility mode and / or a mode in which input at a physical input mechanism is not required to allow secure operation). Thus, in some embodiments, while operating in the second mode (as described in more detail with respect to FIGS. 7L and 7M ), the first type of biometric authentication (e.g., full or high fidelity) is optionally required in response to receiving a request to perform a secure operation.

[0146] In some embodiments, at least one (e.g., the same or different) biometric sensor is used to perform both the first type of biometric authentication and the second type of biometric authentication (e.g., a retinal scanner, a depth camera, and / or a proximity sensor).

[0147] For example, the second type of biometric authentication optionally focuses on detecting features and characteristics such as pupils, eye glints, and iris features (e.g., shape, size, and / or color pattern) of user 704. Specifically, electronic device 700 captures information about the appearance of the user's eyes and records information about the detected pupils, eye glints, and / or iris features. Electronic device 700 again captures information about the appearance of the user's eyes at a subsequent time (e.g., 1 second later, 2 seconds later, etc.) and stores information about the detected pupils, eye glints, and / or iris features. A comparison is made between the information recorded at the first time and the information recorded at the second time to determine if the same user (e.g., user 704) was using the device at both times. For example, if the first detected pupil, eye glint, and / or iris feature matches (e.g., completely and / or with sufficient confidence) the second detected pupil, eye glint, and / or iris feature, the electronic device 700 determines that the same user used the device at both times and thus the respective criteria remain satisfied. This process may be repeated until a certain time, such as when the user provides a request to perform a secure operation. If at this particular time a determination has been made (since the first type of biometric authentication) that the same user has not used the device at any given time (e.g., the first detected information did not match the second detected information), the respective criteria are determined to be not satisfied and, as a result, the performance of the secure operation is cancelled (e.g., until further authentication is performed). For example, if the sensor measurements indicate that the presence of eyes, iris features, pupils, and / or eye glints was not detected, the respective criteria will be determined to be not satisfied. The lack of presence of the eye feature may be based on the user's eyes being closed or otherwise not being detectable by the sensor (e.g., a head mounted display being improperly mounted). For example, in an embodiment in which the electronic device 700 represents a head mounted display, the user may have removed the head mounted display.In embodiments in which the electronic device 700 corresponds to a tablet device or a smartphone device, the user 704 may be holding the device 700 such that the eyes of the user 704 are not within the field of view of the sensor (e.g., the electronic device 700 is placed upside down on a surface). In other words, each criterion optionally includes an "eyes open" criterion that is met when at least one eye of the user 704 does not remain continuously closed for more than a threshold period (e.g., a non-zero period such as 1 second or 2 seconds).

[0148] In some embodiments, the first type of biometric authentication is based on different biometric features than the second type of biometric authentication. Specifically, the first type of biometric authentication optionally corresponds to a full iris feature verification, such that the electronic device 700 analyzes characteristics related to the entire eye of the user 704 or a portion of the eye of the user 704, such as pupil, eye glint, and / or iris features. The second type of biometric authentication may correspond to a partial face recognition verification. The partial face recognition optionally includes analyzing features related to the portion of the face surrounding the eye (and excluding the eye), such as the eyebrows, eyelids, skin surrounding the eye, forehead, cheeks, and / or nose of the user 704. For example, the electronic device 700 captures the appearance of the portion of the face of the user 704 surrounding the eye. At a later time (e.g., one or two seconds later), the electronic device 700 again captures the appearance of the portion of the face of the user 704 surrounding the same eye. The electronic device 699 compares the information recorded at the first time with the information recorded at the second time to determine whether the same user (e.g., user 704) used the device at both times. For example, if the first detected eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose match (e.g., perfectly or with sufficient confidence) with the second detected eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose, it is determined that the same user used the device at both times, and thus the respective criteria remain satisfied. The process may repeat until a particular time, such as when the electronic device 700 receives a request from the user to perform a secure operation. If at this particular time, there is a determination that the same user did not use the device from the time of the first type of authentication (e.g., the first detected information did not match the second detected information), it is determined that the respective criteria are not satisfied, and as a result, the electronic device 700 ceases to perform the secure operation (e.g., until further authentication is performed). In other words, each criterion optionally includes non-eye criteria that may be met even if the eyes of the user 704 are not detectable (e.g., not detectable for a threshold period of time, such as 1, 5, 10, 15, or 60 seconds).

[0149] In some embodiments, authentication is improved by using a first biometric feature for the first type of biometric authentication and another biometric feature for the second type of biometric authentication. For example, analyzing eye features for the first type of biometric authentication and facial features for the second type of biometric authentication improves authentication when eye features are not always available for analysis. Specifically, the user 704 is initially authenticated based on eye features, but facial features (e.g., eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose) are used for the second type of biometric authentication. In this case, even in situations where the eyes of the user 704 cannot be detected by one or more sensors of the electronic device 700, the electronic device 700 can still detect the user's facial features and perform the second type of authentication to allow the execution of a secure operation.

[0150] Returning to FIG. 7F, in response to receiving the request to perform the secure operation, the electronic device 700 determines whether the user 704 meets the respective authentication criteria to perform the secure operation. In some embodiments, the request to perform the secure operation is received at a second time (after the first time) because the user 704 was biometrically authenticated at a first time. In this case, if a determination is made that the user 704 did not meet the respective criteria between the first and second times, an authentication guidance user interface 724 is provided to the user to biometrically authenticate the user 704 using the first type of biometric authentication. In other words, when the electronic device 700 cannot verify that the same user has been using the device since the first first type of biometric authentication (or can verify that the same user has not been using the device since the first first type of biometric authentication), the electronic device 700 attempts to verify the user's identity again using the first type of biometric authentication. Failure to meet the respective authentication criteria may be the result of various factors. For example, the eyes of the user 704 may be undetectable for a period between the first time and the second time. In some embodiments, the initial authentication may be based on another type of biometric authentication (e.g., fingerprint authentication) or based on a non-biometric authentication (e.g., passcode entry). Certain initial authentications, such as fingerprint authentication or passcode authentication, optionally require the user 704 to provide a specific first type of biometric authentication (e.g., full or high fidelity iris feature verification) to allow secure operation, since the respective authentication criteria are not automatically met. In some embodiments, the authentication guidance user interface 724 includes instructions for performing the first type of biometric authentication, such as instructions requesting the user to provide a specific movement of an individual biometric feature. For example, the authentication guidance user interface 724 may include an initial prompt such as "Biometric authentication required" with an option for the user to "continue" and another option for the user to "cancel". In FIG. 7F, the electronic device 700 detects that the "continue" option has been activated.

[0151] In response to detecting activation of the "continue" option in FIG. 7G, the electronic device 700 initiates a first type of biometric authentication. When the first type of biometric authentication corresponds to iris feature verification, the electronic device 700 instructs the user to look at various portions of the display when the biometric authentication begins (e.g., via a "look here" prompt). During the first type of biometric authentication, the electronic device 700 optionally analyzes a gaze 726 corresponding to the user 704 to obtain characteristics related to the entire eye of the user 704 or a portion of the eye of the user 704, such as iris features, pupils, and / or glints. When the first type of biometric authentication corresponds to face verification, the electronic device 700 instructs the user 704 to rotate his / her face at a particular angle. In some embodiments, when the first type of biometric authentication corresponds to fingerprint authentication, the electronic device 700 instructs the user to move a finger along a particular path and / or touch a finger to a touch-sensitive contact of the device 700 or another device in a particular manner.

[0152] 7H illustrates a passcode entry prompt 728. In some embodiments, the user authentication criteria is met when the passcode entry matches a registered passcode entered via the passcode entry prompt 728. For example, the electronic device 700 is optionally configured such that user authentication (or user authentication to perform secure operations) occurs via a passcode or password (e.g., the user has disabled iris feature verification, face verification, and / or other types of biometric authentication). Alternatively, following a determination that the user authentication criteria is not met based on a failure of the first type of biometric authentication (e.g., the device 700 was unable to successfully complete the first type of biometric authentication), the electronic device 700 displays the passcode entry prompt 728 to provide an additional attempt to meet the user authentication criteria. In some embodiments, a determination may be made that an initial first type of biometric authentication was successful (e.g., the user successfully passed the biometric authentication when first accessing the device), but a second type of biometric authentication to detect continuity may have failed (e.g., the user's 704 eyes were undetectable for a period of time after the initial time). In this case, the electronic device 700 attempts to authenticate the user by displaying a passcode entry prompt 728 to the user.

[0153] In some embodiments, if the continuity verification fails, the electronic device 700 first attempts to authenticate the user by means other than passcode verification before attempting passcode verification. For example, the electronic device 700 attempts a first type of biometric authentication, such as full or high-fidelity iris feature verification (e.g., as shown in FIG. 7G), or attempts another type of full or high-fidelity biometric authentication, such as fingerprint authentication and / or face recognition authentication. If the full or high-fidelity biometric authentication fails, passcode verification is optionally used as a default option. In some embodiments, the passcode entry prompt 728 includes a passcode entry portion and an affordance for rejecting the passcode entry. If the user selects the option to reject the passcode entry, the user permission for secure operation is withdrawn.

[0154] FIG. 7I illustrates a change in the viewpoint of the user 704, following FIG. 7D. Typically, the visible amount of an interface object may change based on a change in the viewpoint of the user 704. In some embodiments, at least a portion of the login user interface 702 and the permission interface 712 are invisible to the user 704, based on, for example, a portion of the login interface and a portion of the permission interface not being fully displayed on the display 700a. In some embodiments, when the electronic device 700 is implemented in an XR environment, certain user motions optionally cause interface objects to be invisible or partially invisible. For example, a user turns their head in a particular direction while wearing the device 700 (e.g., a head-mounted display). The login user interface 702 and the permission interface 712 are optionally environment-locked in the environment, so that the interfaces remain in the same general location in the environment regardless of the user's head movements or other motions. Thus, at least a portion of the interface is optionally invisible to the user 704 based on the interface being toward the side of the user's 704 field of view (or behind the user in the XR environment). Thus, based on the user's 704 movement, a visible amount of the login user interface 702 and the permission interface 712 may be reduced. As a result, a determination is made that less than a threshold amount of the login user interface 702 and / or the permission interface 712 is visible from the user's 704's perspective. Based on this determination, the electronic device 700 modifies the appearance of the permission interface 712, for example, to indicate that the user 704 cannot request performance of the secure operation.

[0155] In FIG. 7J, an exemplary environment is depicted on the display 700a, including a background region 730. In some embodiments, the environment is implemented in the context of an XR environment, so that the electronic device 700 corresponds to a head-mounted display. The background region 730 optionally corresponds to a representation of the user's 704's physical environment, virtual environment, or a combination of physical and virtual environments. The user 704 may interact with various interface objects, virtual objects, or other objects depicted in the environment. For example, the electronic device 700 displays a messaging interface 732, through which the user 704 can send and receive messages. In addition, the electronic device 700 also displays a login user interface 702 and an authorization interface 712. While the electronic device 700 detects the user 704 interacting with an interface, the interface may be fully visible to the user 704, such that the interface is not occluded by objects or other elements in the environment. The electronic device 700 optionally detects that the user 704 is currently looking at the permission interface 712, as depicted by the user gaze direction 718. As described herein, user permission of secure operation is optionally enabled. Thus, the appearance (e.g., not grayed out) of the permission interface 712 and the indication 734 inform the user that the user can authorize the autofill function, for example, by performing a function indicated by the indication 734 (e.g., double pressing the button 722).

[0156] FIG. 7K illustrates an exemplary environment including a change in the viewpoint of the user 704. In some embodiments, the visible amount of an object depicted in the environment changes based on occlusion by other objects in the environment. Such occlusion occurs based on a movement of the (occluded or occluding) object itself or a change in the viewpoint of the user 704. For example, with reference to FIG. 7J, the electronic device 700 detects that the user 704 has turned left in the environment. Specifically, if the environment is implemented in the context of an XR environment, the electronic device 700 detects that the user 704 has turned his head to the left while wearing the electronic device 700 (e.g., a head mounted display). Various objects in the environment may be viewpoint locked or environment locked. Specifically, the messaging interface 732 may be viewpoint locked such that the messaging interface 732 appears in the same location and / or position in the viewpoint of the user 704 as the viewpoint of the user 704 changes. Alternatively, the login user interface 702 and the permission interface 712 may be environment locked such that the login user interface 702 and the permission interface 712 are displayed in a position based on the location in the XR environment within the user's viewpoint (e.g., in front of a representation of a couch). Thus, when the user 704 adjusts his viewpoint to the left (relative to FIG. 7J), the messaging interface 732 appears to move over the login user interface 702 and the permission interface 712. In other words, the login user interface 702 and the permission interface 712 remain in the same general location relative to the background of the XR environment from the user's viewpoint, and the user's viewpoint movement results in the messaging interface 732 being overlaid on top of the login user interface 702 and the permission interface 712. Based on the viewpoint change, the overlay by the messaging interface 732 reduces the visible amount of the login user interface 702 and the permission interface 712.The electronic device 700 detects the user gaze direction 718 as remaining directed toward the permission interface 712, but a determination is made that less than a threshold amount of the login user interface 702 and / or the permission interface 712 is visible from the user's perspective, and therefore the appearance of the permission interface 712 is adjusted (e.g., grayed out) to reflect that the user's permission for secure operation has been disabled.

[0157] In FIG. 7L, the electronic device 700 displays an interface for authorizing secure operation while the electronic device 700 is operating in a second mode different from the first mode (described with respect to FIGS. 7A-7K). In some embodiments, the electronic device 700 operates in the second mode, such as an accessibility mode and / or a mode in which input with a physical input mechanism is not required to authorize secure operation. While in the second mode, the electronic device 700 displays a login user interface 736. The login user interface 736 corresponds to a login interface that includes functionality to facilitate interaction for users with accessibility requirements (e.g., limited fine motor skills, vision, and / or hearing abilities). The electronic device 700 also optionally displays an authorization interface 738 (e.g., an "autofill" feature) that facilitates the process for secure operation, such as authorizing information into certain displayed fields.

[0158] In some embodiments, the login user interface 736 and the authorization interface 738 correspond to displayed virtual interface objects in the XR environment. The authorization interface 738 includes a search function 742 and one or more credential options 744a and 744b. In some embodiments, the credential options are associated with a username and / or password. In some embodiments, the electronic device 700 receives a search query for additional credential options via the search function 742 and provides any populated result options based on the search. While in the second mode, the user 704 may interact with the authorization interface 738 (e.g., select one of the credential options 744a or 744b or search for additional credential options via the search function 742) using an input or combination of inputs, such as gaze input, controller input, and / or voice input, that are compatible with the accessibility features. In some embodiments, the electronic device 700 detects the selection of a credential option (e.g., credential option 744a). After the credential option is selected, the user 704 may activate affordance 740 using an input or combination of inputs, such as gaze input, controller input, and / or voice input, that are compatible with the accessibility features. Specifically, affordance 740 may be displayed with the text "Confirm with Assistive Touch" to indicate to the user that the device is operating in the second mode, and thus no physical input mechanism is required to allow the autofill operation. In FIG. 7L, electronic device 700 detects selection of affordance 740.

[0159] In FIG. 7M, in response to detecting the selection of affordance 740, electronic device 700 displays assistive input interface 742 including a representation of multiple functions for facilitating various tasks. In some embodiments, user 704 must gaze at affordance 740 while activating affordance 740 to trigger the selection of affordance 740. For example, user 704 gazes at affordance 704 and activates a special input while gazing at affordance 740 (described in more detail in paragraphs

[0272] -

[0273] ). In addition, in response to user selection of affordance 740, electronic device 700 also modifies the appearance of authorization interface 738 to indicate that a credential option was selected by the user. In some embodiments, the position or location of authorization interface 738 is adjusted (e.g., in an XR environment, authorization interface 738 is moved to the side of the user's field of view).

[0160] The functions included in the assistive input interface 742 include various options. For example, the various options include an option to navigate to a main screen (e.g., a “Home” option), an option to display device notifications (e.g., “Notifications”), and / or an option to display a settings menu (e.g., a “Control Menu”). The assistive input interface 742 includes a secure operation permission affordance 744. The functions can optionally be invoked using a specific input type, which may be pre-configured by the electronic device 700 or a specific user. For example, while in the first mode, the option to navigate to the main screen is executed in response to the electronic device 700 receiving a primary input type (e.g., a “swipe” input at a specific location), while while in the second mode, the option to navigate to the main screen is executed in response to the electronic device 700 receiving an accessibility-based input type (e.g., a user gazing at a “Home” icon in combination with a long press on a controller). Additionally, while in the first mode, allowing secure operation is performed in response to the electronic device 700 receiving a primary input type, such as a double press of a particular hardware button, while while in the second mode, allowing secure operation is performed by an accessibility-based input type, such as activating a secure operation allowance affordance (e.g., a user gazing at the secure operation allowance affordance 744 in combination with a long press of the controller).

[0161] In some embodiments, in response to receiving an input to allow secure operation, the electronic device 700 performs user authentication. For example, a first type of biometric authentication is performed so that the user 704 is biometrically authenticated using one or more biometric sensors of the device 700. The first type of biometric authentication optionally includes a first level (e.g., full or high fidelity) of iris feature verification. In particular, while operating in the second mode, the second type of biometric authentication (e.g., passive iris feature verification, in which the electronic device 700 repeatedly analyzes certain characteristics associated with the entire eye of the user 704 or a portion of the eye of the user 704 while the user 704 interacts with the device 700) is disabled. Thus, the first type of biometric authentication is required when the input to allow secure operation is received. In particular, (referring back to FIG. 7F) the electronic device 700 provides an authentication guidance user interface to the user to biometrically authenticate the user 704 using the first type of biometric authentication. The authentication guidance user interface includes instructions for performing a first type of biometric authentication, such as instructions requesting a user to provide a particular movement of an individual biometric feature. For example, the authentication guidance user interface includes an initial prompt such as "Biometric authentication required," with an option for the user to "continue" and another option for the user to "cancel."

[0162] When the electronic device 700 detects that the option to "continue" has been activated, the electronic device 700 initiates a first type of biometric authentication. If the first type of biometric authentication corresponds to iris feature verification, the electronic device 700 optionally instructs the user to look at various portions of the display when the biometric authentication begins (e.g., via a "look here" prompt). During the first type of biometric authentication, the electronic device 700 analyzes a gaze corresponding to the user 704 to obtain characteristics related to the entire eye of the user 704 or a portion of the eye of the user 704, such as iris features, pupils, and / or glints. If the first type of biometric authentication corresponds to face verification, the electronic device 700 instructs the user 704 to rotate the face of the user 704 at a specific angle. In some embodiments, if the first type of biometric authentication corresponds to fingerprint authentication, the electronic device 700 instructs the user 704 to move a finger along a specific path and / or touch a finger to a touch-sensitive contact of the device 700 or another device in a specific manner. In some embodiments, the first type of biometric authentication is based on another type of biometric authentication (eg, voice authentication) or is based on a non-biometric authentication (eg, passcode entry).

[0163] In FIG. 7N, an additional exemplary user interface for allowing secure operations is displayed on the display 700a of the device 700. Specifically, FIG. 7N illustrates a secure operation related to payment. In some embodiments, the secure operation related to payment is implemented in the context of an XR environment, so that the device 700 corresponds to a head-mounted display. The electronic device 700 receives the user input and, in response, navigates to a displayed e-commerce website via a web browser 748 to purchase an item (e.g., shoes). In some embodiments, when the electronic device 700 is operating in a first mode (e.g., a standard mode without accessibility features enabled and / or a mode that allows secure operations using a default physical input mechanism), the electronic device 700 detects a selection of an affordance 750 to initiate a payment function. For example, since the display 700a is a touch-sensitive display, the user 704 presses the affordance 750 with a finger. In some embodiments, when the device 700 is implemented in an XR environment, the electronic device 700 detects a selection of the affordance 750 using a controller communicatively coupled to the electronic device 700. Additionally, the electronic device 700 detects the selection of the affordance 750 using gaze, for example, by detecting that the user 704 is looking at the affordance 750 for a predetermined period of time and / or receiving one or more gestures. In some embodiments, since the electronic device 700 corresponds to a head mounted display, the electronic device 700 detects that the user 704 is making one or more gestures to activate the affordance 750, such as an air gesture. When the electronic device 700 detects activation of the affordance 750, the electronic device 700 displays a payment details interface 752. The payment details interface 752 includes details regarding information for completing an order to purchase the item displayed in the associated web browser 748. For example, the payment details interface 752 includes credit card billing information, shipping information, and / or price information.

[0164] In some embodiments, the electronic device 700 enables user permission of the payment operation conditional on a determination that the user 704 is authorized to perform the payment operation. For example, the user is determined to be authorized to perform the secure operation based on a gaze criterion with respect to the user gaze direction 754. The gaze criterion is met when the user 704 gazes at an interface object, such as the web browser 748 and / or the payment details interface 752. For example, the electronic device 700 optionally displays the web browser 748 and / or the payment details interface 752 in a central portion of the display 700a. In some embodiments, the electronic device 700 determines that the user is not authorized to perform the secure operation when the user is not gazed at the web browser 748 and / or the payment details interface 752 (e.g., the user's eyes are closed or the user is looking at a corner of the display 700a (away from the interfaces 748 and 752)). In some embodiments, the user permission of the secure operation is enabled conditional on a determination that the user 704 has or provides a particular movement of a biometric characteristic. For example, the particular movement may include the user gazing at a particular portion of the web browser 748 (e.g., gazing at affordance 750) or at the payment details interface 752 (e.g., the displayed price), the user providing a particular facial rotation, or the user moving a finger along a particular path.

[0165] Once authorization of the payment feature is enabled, the user proceeds to authorize the payment feature. Specifically, the electronic device 700 modifies the appearance of one or more interface objects and / or displays one or more additional interface objects to indicate to the user that the user can request to perform a payment operation. The indications 756 include the instruction "double-click to authorize," which may be displayed in proximity to the hardware button 758. In some embodiments, when the electronic device 700 is implemented in an XR environment, the electronic device 700 displays the indication, such as instructions to perform one or more air gestures (e.g., instructions to move one or more hands or fingers in a particular motion) and / or instructions to perform one or more inputs on an XR compatible controller.

[0166] In some embodiments, in response to receiving a request to perform a secure operation, the electronic device 700 determines whether the user 704 meets user authentication criteria for performing the payment operation, as discussed with respect to Figures 7F-7H. If the user does not meet the user authentication criteria, the user's permission for the payment operation is revoked. Thus, a user authentication process is performed, as discussed with respect to Figures 7F-7H. If the user does not meet the user authentication criteria, the electronic device 700 initiates the payment transaction and provides feedback to the user regarding the payment transaction.

[0167] Additional description regarding FIGS. 7A-7N is provided below with reference to the method 800 described with respect to FIGS. 7A-7N.

[0168] 8 is a flow diagram of an example method 800 for facilitating user consent for secure operation, according to some embodiments. In some embodiments, the method 800 is executed in a computer system (e.g., computer system 101 of FIG. 1 (e.g., a smartphone, a smartwatch, a tablet, and / or a wearable device)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a display controller, a touch-sensitive display system, a display (e.g., embedded and / or connected), a 3D display, a transparent display, a projector, and / or a head-up display) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and / or one or more input devices. In some embodiments, the method 800 is governed by a plurality of instructions that are stored in a non-transitory (or transitory) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 of FIG. 1). Some operations of method 800 are optionally combined and / or the order of some operations is optionally changed.

[0169] As described below, method 800 provides an intuitive method for user consent for secure operations. This method reduces the cognitive burden on the user while performing secure operations, thereby creating a more efficient human-machine interface. For battery-powered computing devices, facilitating user consent for secure operations to be faster and more efficient conserves power and extends the time between battery charges.

[0170] Detect (802) a change in a current viewpoint (e.g., from a first viewpoint to a second viewpoint different from the first viewpoint) of a user (e.g., user 704) via one or more input devices (e.g., 125 and / or 150) while a three-dimensional environment (e.g., 730) including virtual user interface objects (e.g., 702, 712, 736, 738, 748 and / or 752) including information regarding a secure operation (e.g., 706, 708, 710, 716a, 716b, 744a, and / or 744b) is visible via a display generation component (e.g., 120 and / or 700a) (e.g., receiving a request to autofill input fields (e.g., with a password and / or email information) and / or detecting activation of a payment affordance).

[0171] In response to detecting a change in a user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), enable (804) user authorization of a secure action using the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., enabling an autofill authorization option to automatically fill in fields in a second user interface (e.g., a form user interface) and / or enabling an activatable purchase authorization option) in accordance with determining that at least a threshold amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are visible (e.g., a gaze is directed at the user interface and / or the target of the user's gaze is the user interface) from the user's viewpoint (e.g., 718, 746, and / or 754) and that the user (e.g., 704) is authorized to perform the secure action. In some embodiments, the computer system (e.g., 700) visually indicates that the viewpoint (e.g., 718, 746, and / or 754) includes a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and thus a respective permission option is enabled (e.g., by not graying out a portion of the second user interface (e.g., 702, 712, 736, 738, 748, and / or 752) or a button (e.g., 710)). In some embodiments, the activatable purchase permission options include a single press, a long press, a single press followed by a long press, a press followed by a rotation, a rotation followed by a press, and / or multiple presses in succession (e.g., a double press, a triple press).

[0172] The virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) that are visible from the viewpoint (e.g., 718, 746, and / or 754) of the user (e.g., 704) are less than a threshold amount (e.g., the gaze (e.g., 718, 746, and / or 754) is not directed toward the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the subject of the user's gaze (e.g., 718, 746, and / or 754) is not directed toward the user interface (e.g., 7 7, pursuant to a determination that the virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) is not a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752), withdrawing (806) user authorization of the secure operation using the virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., disabling an autofill authorization option to automatically fill in fields in a second user interface (e.g., a form user interface) and / or disabling enabling an activatable purchase authorization option (e.g., a double press procedure of a physical button)). In some embodiments, pursuant to a determination that the viewpoint (e.g., 718, 746, and / or 754) does not include a user interface (e.g., 702, 712, 736, 738, 748, and / or 752), the computer system (e.g., 700) visually indicates that the viewpoint (e.g., 718, 746, and / or 754) does not include a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and therefore a respective permission option is not enabled (e.g., by graying out a portion of the second user interface (e.g., 702, 712, 736, 738, 748, and / or 752) or a button (e.g., 710)). Enabling user permission of a secure operation when a separate user interface is visible provides improved security / privacy by ensuring that the user has an opportunity to review information related to the secure operation before allowing the secure operation to proceed.

[0173] In some embodiments, an input to authorize the performance of a secure operation (e.g., double-clicking a hardware button (e.g., 722 and / or 758) and / or activating a particular affordance) is received from a user (e.g., 704). In some embodiments, the input to authorize includes a single press, a long press, a single press followed by a long press, a press followed by a rotate, a rotate followed by a press, multiple presses in succession (e.g., a double press, a triple press). In some embodiments, in response to receiving an input authorizing performance of a secure operation, performance of the secure operation is initiated (e.g., filling in fields of a second user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., a form user interface) and / or payment) pursuant to determining that one or more sets of execution criteria are met (e.g., the gaze (e.g., 718, 746, and / or 754) is directed at a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the user's gaze (e.g., 718, 746, and / or 754) is directed at a user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). a new transaction), where the set of one or more execution criteria includes a visibility criterion, and the visibility criterion is met when at least a threshold amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., the threshold amount corresponds to any portion of the interface objects (e.g., 702, 712, 736, 738, 748, and / or 752), a non-zero amount of the interface objects (e.g., 702, 712, 736, 738, 748, and / or 752), or the entirety of the interface objects (e.g., 702, 712, 736, 738, 748, and / or 752)) are visible from a viewpoint (e.g., 718, 746, and / or 754) of a user (e.g., 704).In some embodiments, pursuant to a determination that one or more sets of execution criteria are not met (e.g., the gaze (e.g., 718, 746, and / or 754) is not directed toward a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the user's gaze (e.g., 718, 746, and / or 754) is not directed toward a user interface (e.g., 702, 712, 736, 738, 748, and / or 752)), execution of a secure operation is not initiated (e.g., refrain from filling in fields (e.g., 706 and / or 708) of a second user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., a form user interface) and / or refrain from initiating a payment transaction). Initiating performance of a secure operation when a threshold amount of a distinct user interface is visible provides improved security / privacy by ensuring that a user has an opportunity to review information related to the secure operation before allowing the secure operation to proceed.

[0174] In some embodiments, the set of one or more performance criteria includes user authentication criteria that are met when a user (e.g., 704) is authenticated (e.g., the criteria include positive indications of continuous iris feature verification from the time the user (e.g., 704) engages the device (e.g., 700), a first level (e.g., full or high fidelity) iris feature verification, and / or a positive indication of proper passcode authentication). Requiring user authentication in addition to user authorization for secure operation provides improved security / privacy by adding an additional layer of verification specific to the user requesting authorization.

[0175] In some embodiments, biometric information is detected from a user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed and / or a second level (e.g., tracking eye information (e.g., detected glint information and / or detected pupil information) for continuity) iris feature verification is performed). In some embodiments, detecting the biometric information includes receiving input from the user (e.g., 704) (e.g., image(s) of the user's face) in response to a prompt (e.g., 724) to gaze at a particular portion (e.g., 726) of the screen (e.g., 700a). In some embodiments, detecting the biometric information includes passively detecting the biometric information while the user (e.g., 704) is using the device (e.g., wearing a head mounted display). In some embodiments, detecting the biometric information includes detecting the biometric information while a hardware button (e.g., 722 and / or 758) is activated (e.g., a single press, a long press, a long press after a single press, a press and a rotation, a press after a rotation, multiple consecutive presses (e.g., double press, triple press)) or in response to a hardware button (e.g., 722 and / or 758) being activated. In some examples, the biometric information is compared to enrolled biometric information (e.g., comparing an acquired iris scan to a stored iris print and / or comparing current eye information to eye information from a previous frame), and a determination of whether a user authentication criterion is met is based on a comparison of the biometric information to the enrolled biometric information (e.g., the user authentication criterion is met if the comparison meets a sufficient similarity threshold (e.g., 95% similarity). In some embodiments, a first level (e.g., full or high fidelity) iris feature verification is performed and a second level (e.g., tracking eye information for continuity) iris feature verification is performed at a later time (e.g., one hour) to tie the first level iris feature verification to an execution criterion for initiating execution of a secure operation.Requiring biometric authentication in addition to user authorization for secure operation provides improved security / privacy by adding a biometric verification layer specific to the user requesting authorization.

[0176] In some embodiments, a determination that the user authentication criteria has been met is made in response to receiving an input that allows a secure operation to be performed (e.g., in response to activation of a hardware button (e.g., 722 and / or 758) (e.g., single press, long press, single press followed by long press, single press followed by rotation, press followed by rotation, multiple presses in succession (e.g., double press, triple press), verifying that the user authentication criteria has been met). Requiring user authentication at the time of a user request to perform a secure operation provides improved security / privacy by verifying the identity of the user requesting the operation.

[0177] In some embodiments, determining whether user authentication criteria are met includes retrieving stored authentication information, where the stored authentication information is indicative of one or more previously performed biometric authentications (e.g., retrieving a continuity result indicating a positive result (e.g., iris feature continuity and / or facial continuity is maintained) or a negative result (e.g., iris feature continuity and / or facial continuity is not maintained)). Utilizing the stored authentication information in addition to user permissions for secure operations creates an efficient and uninterrupted user experience for biometric authentication, encourages users to use biometric authentication, and therefore provides improved security / privacy by providing a more secure experience.

[0178] In some embodiments, the user authentication criteria includes performing a first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed), and after performing the first type of biometric authentication (e.g., after successfully performing the first level (e.g., full or high fidelity) iris feature verification), performing a second type of biometric authentication (e.g., a second level (e.g., tracking eye information for continuity) iris feature verification), receiving an input from a user (e.g., 704) including the first individual biometric information (e.g., obtaining eye information from a current frame), and tracking the received input with the second individual biometric information (e.g., receiving the second individual biometric information from the user (e.g., 704). and repeatedly performing a second type of biometric authentication (e.g., every 2 frames (e.g., a frame includes a still image or set of images obtained from a sensor (e.g., biometric sensor) of the device), every 3 frames, every 4 frames, every 5 frames, every 6 frames, every 7 frames, every 8 frames, every 9 frames, or every 510 frames)). Obtaining multiple sensor measurements at multiple intermediate times between the first and second times provides improved security / privacy by verifying that the same user has been using the device since the time of the initial authentication.

[0179] In some embodiments, performing the first type of biometric authentication includes detecting the presence of a first type of biometric feature (e.g., a first level (e.g., full or high fidelity) verification based on eye features, or a first level (e.g., full or high fidelity) verification based on facial features), and performing the second type of biometric authentication includes detecting a second biometric feature type (e.g., the second level verification is based on eye features (e.g., tracking eye information for continuity) or the second level verification is based on facial features (tracking eyebrows, eyelids, forehead, and / or nose for continuity)), where the first biometric feature type and the second biometric feature type correspond to the same biometric feature type (e.g., the first level verification and / or the second level verification are based on eye features, or the first level verification and / or the second level verification are based on facial features). For secure operation, continuously verifying user presence based on the same biometric characteristics as the initial authentication, in addition to user authorization, creates an efficient and uninterrupted user experience for biometric authentication, encourages users to use biometric authentication, and therefore provides security / privacy improvements by providing a more secure experience.

[0180] In some embodiments, the user authentication criteria is met when the user (e.g., 704) is authenticated based on an eye biometric (e.g., detecting a first pattern based on a plurality of features corresponding to the user's (e.g., 704) eye (e.g., a pattern formed based on eye fibers, pits, grooves, coronas, and / or rings). In some embodiments, the detected first pattern is compared to a reference pattern, the reference pattern based on a plurality of features corresponding to the user's eye (e.g., a stored pattern based on full or high fidelity iris feature verification, or a stored pattern based on partial iris feature verification for continuity). Requiring authentication based on eye biometrics in addition to user authorization for secure operation provides improved security / privacy by adding a biometric verification layer specific to the user requesting authorization.

[0181] In some embodiments, the user authentication criteria is met when the passcode input matches a registered passcode (e.g., receiving multiple numeric inputs (e.g., 0-9) and comparing with a previously registered passcode (e.g., a passcode containing the digits 0-9)). Requiring passcode authentication in addition to user authorization for secure operation provides improved security / privacy by providing an additional method for verifying the user's identity.

[0182] In some embodiments, following a determination that the user authentication criteria is not met (e.g., the device is unable to obtain sufficient iris feature scans, results of the first level iris feature verification indicate that the iris feature scan does not match the reference scan, or the user declines to provide iris feature verification), a prompt (e.g., 728) (e.g., a button that the user activates to invoke a passcode entry interface) is provided (e.g., displayed via the display generation components (e.g., 120 and / or 700a)) to allow authentication via a passcode. In some embodiments, the prompt (e.g., 704) includes a passcode entry portion and an affordance for declining the passcode entry. Requiring passcode authentication after failed biometric authentication provides improved security / privacy by providing a backup option for verifying the user's identity.

[0183] In some embodiments, user authorization for a secure operation (e.g., initiating a payment operation and / or initiating an autofill operation) using a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) includes user activation (e.g., single press, long press, single press followed by long press, rotation and press of the hardware user interface element, rotation and press, two, three, or more consecutive presses) of a hardware user interface element (e.g., 722 and / or 758). Utilizing a hardware element for authorization provides improved security / privacy by reducing the likelihood of erroneous authorization and / or preventing unauthorized input from malicious software.

[0184] In some embodiments, the secure action is a payment (e.g., providing payment information to a third party service to purchase an item, such as providing credit card information to an e-commerce website to purchase shoes). Enabling user permission of the payment action when a separate user interface is visible provides improved security / privacy by ensuring that the user has an opportunity to review information related to the secure action before allowing the secure action to proceed.

[0185] In some embodiments, the secure operation includes autofilling user credentials (e.g., providing login information (e.g., 706 and / or 708) (e.g., username and / or password) to access secure information (e.g., a banking website, a stock trading application, and / or user profile information). Enabling user permission of the autofill operation when a separate user interface is visible provides improved security / privacy by ensuring that the user has an opportunity to review information related to the secure operation before allowing the secure operation to proceed.

[0186] In some embodiments, in response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), the amount of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) that are visible to the user (e.g., 704) changes (e.g., a viewpoint shift (e.g., looking left or right) results in the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) being partially (e.g., as shown in Figures 7I and 7K) or completely removed from view, or a viewpoint shift results in the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) being completely removed from view and then becoming visible). Enabling user permission for secure actions based on a change in the user's viewpoint provides improved security / privacy by verifying that the user is still viewing details related to the secure action.

[0187] In some embodiments, the amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) that are visible varies based on occlusion by physical (e.g., real) objects. In some embodiments, the visible amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) from a viewpoint (e.g., 718, 746, and / or 754) of a user (e.g., 704) is detected, and the viewpoint (e.g., 718, 746, and / or 754) of the user includes physical objects represented in the three-dimensional environment. In some embodiments, a user (e.g., 704) views a physical object (e.g., a computer monitor) through an additional display, or the user (e.g., 704) views a representation of the physical object reproduced on an opaque display, where the physical object is closer to the user (e.g., 704) than the representations of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752). In some embodiments, a reduction in the visibility amount is detected in response to detecting a change in a user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), where the detected change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes detecting a physical object appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) with respect to the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754). In some embodiments, the reduction in the visibility amount is based on the detected physical object appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., less of the virtual interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are displayed than were displayed prior to the viewpoint change).In some embodiments, a physical object may move in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., a physical balloon floats in front of the central field of view of the user (e.g., 704)). In some embodiments, the changed user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes a physical object in close proximity to the user (e.g., 704) that occludes the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., the user (e.g., 704) turns toward a tree that is one foot in front of the user (e.g., 704)). When a physical object is described as occluding a virtual object (e.g., 702, 712, 736, 738, 748, and / or 752), this refers to the device (e.g., 700) optionally withdrawing or obscuring at least a portion of the virtual object (e.g., 702, 712, 736, 738, 748, and / or 752) that is within the same line of sight as the physical object but at a greater simulated distance from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) than the physical object, effectively virtually occluding the virtual object (e.g., 702, 712, 736, 738, 748, and / or 752) (or a portion of the object) that would otherwise overlap the physical object. Enabling user permission for a secure action based on the details of the secure action being occluded by a physical object provides improved security / privacy by verifying that the user is still viewing details related to the secure action.

[0188] In some embodiments, the amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) that are visible changes based on occlusion by the virtual object (e.g., 732). In some embodiments, a reduction in the visibility is detected (e.g., a portion of the interface becomes invisible or the entire interface becomes invisible) in response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754). In some embodiments, the detected change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes detecting a virtual object (e.g., 732) appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) with respect to the user's (e.g., 704) viewpoint (e.g., as shown in Figures 7J and 7K). In some embodiments, the reduction in visibility is based on a detected virtual object (e.g., 732) appearing in front of a virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752). Enabling user permission for a secure action based on details of the secure action being occluded by a virtual object provides improved security / privacy by verifying that the user is still viewing details related to the secure action.

[0189] In some embodiments, the visibility of a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on the virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) moving out of the field of view of the user (e.g., 704) (e.g., as shown in FIG. 7I). In some embodiments, in response to detecting a change in a user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) becomes undetectable within the user's (e.g., 704) current viewpoint (e.g., 718, 746, and / or 754) (e.g., the user (e.g., 704) looks away from the interface (e.g., 702, 712, 736, 738, 748, and / or 752) so that it is located "behind" the user in the environment). Enabling user permission for a secure action based on the details of the secure action being outside of the user's field of view provides improved security / privacy by verifying that the user is still viewing details related to the secure action.

[0190] In some embodiments, the visible amount of a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on the virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) moving beyond a threshold distance from the center of the user's (e.g., 704) field of view (e.g., 718, 746, and / or 754) (e.g., the user (e.g., 704) looks away from an interface (e.g., 702, 712, 736, 738, 748, and / or 752) such that only a portion of the interface is visible (e.g., visible toward the side of the user's (e.g., 704) field of view), or the interface (e.g., 702, 712, 736, 738, 748, and / or 752) is moved by the user (e.g., 704) or another user). Enabling user permission for a secure action based on the details of the secure action being too far from the center of the user's field of view provides improved security / privacy by verifying that the user is still looking at the details related to the secure action.

[0191] In some embodiments, the visibility of a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on occlusion by a physical object. In some embodiments, movement of the physical object is detected (e.g., a balloon floats in front of the user (e.g., 704)), and the detected movement includes movement of the physical object in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) relative to the user's viewpoint (e.g., 718, 746, and / or 754) (e.g., a balloon floats in front of the user (e.g., 704) on an additional display (e.g., a display including a transparent portion through which the user (e.g., 704) views the actual surrounding physical environment (e.g., 730)). The movement of the balloon may cause portions of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) to disappear, or on an opaque display (e.g., a display coupled to a sensor and / or camera through which a user (e.g., 704) views an on-display representation (e.g., 730) of the surrounding physical environment), the movement of the balloon may cause a representation of the balloon to be displayed that obscures portions of the virtual interface objects (e.g., 702, 712, 736, 738, 748, and / or 752). Enabling user permission for a secure action based on the details of the secure action being occluded by a physical object may provide improved security / privacy by verifying that the user is still viewing details related to the secure action.

[0192] In some embodiments, determining that a user (e.g., 704) is authorized to perform a secure operation includes determining whether a gaze criterion is met when the user (e.g., 704) gazes at a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752). In some embodiments, the secure operation is initiated pursuant to determining that the user (e.g., 704) gazes at a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) while a request to perform a secure operation is received (e.g., the user (e.g., 704) gazes at selected login information (e.g., 716a, 716b, 744a, and / or 744b) or the user (e.g., 704) gazes at payment information). Requiring the user to view details regarding the secure operation when requesting authorization provides improved security / privacy by ensuring that the user has an opportunity to read details regarding the operation.

[0193] In some embodiments, determining that a user (e.g., 704) is authorized to perform a secure operation includes determining whether an eye criterion is met when the user's (e.g., 704) eyes are open. In some embodiments, the secure operation is initiated pursuant to a determination that both of the user's (e.g., 704) eyes are open or at least one of the user's (e.g., 704) eyes is open while a request to perform a secure operation is received. Requiring the user's eyes to be open to provide permission for a secure operation provides improved security / privacy by ensuring that the user has an opportunity to read details regarding the operation.

[0194] In some embodiments, a user (e.g., 704) may request performance of a secure operation (e.g., an indicator) pursuant to determining that at least a threshold amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are visible from a user's (e.g., 704) perspective (e.g., a line of sight (e.g., 718, 746, and / or 754) is directed toward the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the user's line of sight (e.g., 718, 746, and / or 754) is directed toward the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). The appearance of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) may be modified (e.g., as shown in FIG. 7D ) to indicate to the user (e.g., 704) that a secure operation may be performed (e.g., via 720, 734, and / or 756) (e.g., the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) may not be grayed out, a button may be marked as active, or text (e.g., “Please proceed with your request”) may be provided in the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) indicating that the user (e.g., 704) may request performance of the secure operation).In some embodiments, a user (e.g., 704) may determine that an amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) that are visible from a user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) is less than a threshold amount (e.g., the line of sight (e.g., 718, 746, and / or 754) is not directed at the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the user's line of sight (e.g., 718, 746, and / or 754) is not directed at the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). The appearance of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) is modified (e.g., as shown in FIG. 7C ) to indicate to the user (e.g., 704) that the secure operation cannot be requested to be performed (e.g., the user interfaces (e.g., 702, 712, 736, 738, 748, and / or 752) are grayed out, buttons are marked as inactive, and / or text (e.g., “See to request”) is provided in the user interfaces (e.g., 702, 712, 736, 738, 748, and / or 752) indicating that the user (e.g., 704) cannot request the secure operation to be performed). Providing a visual indication that user permission can be provided for the secure operation provides improved visual feedback by informing the user that the secure operation can be initiated by the user.

[0195] In some embodiments, the three-dimensional environment (e.g., 730) includes a virtual object associated with the secure action (e.g., a virtual representation of a shoe), and includes attaching a virtual user interface object to the virtual object associated with the secure action (e.g., attaching a virtual representation of a sheet containing payment information for the shoe to the shoe), such that when the virtual object is moved, the attached virtual user interface object moves with the virtual object (e.g., when the shoe is moved (e.g., placing the shoe on a table), the sheet remains attached to the shoe and moves with the shoe (e.g., the sheet remains attached to the shoe while the shoe is on the table)). Attaching the corresponding object to an interface having details regarding the secure action provides improved visual feedback by informing the user that the secure action is associated with the corresponding object.

[0196] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchanged, substituted, and / or added between these methods. For example, the determination of whether to enable user permission for secure operation in method 800 is optionally used as part of method 900 to determine whether to enable user permission. For the sake of brevity, those details will not be repeated here.

[0197] Additional description regarding FIGS. 7A-7N is provided below with reference to the method 900 described with respect to FIGS. 7A-7N.

[0198] 9 is a flow diagram of an exemplary method 900 of continuity of authentication for secure operation, according to some embodiments. In some embodiments, the method 900 is executed in a computer system (e.g., computer system 101 of FIG. 1 (e.g., a smartphone, a smartwatch, a tablet, and / or a wearable device)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a display controller, a touch-sensitive display system, a display (e.g., embedded and / or connected), a 3D display, a transparent display, a projector, and / or a head-up display)), one or more input devices, and a biometric sensor (e.g., a facial recognition device, a fingerprint recognition device, and / or an eye tracking device). In some embodiments, the method 900 is governed by a plurality of instructions that are stored in a non-transitory (or transitory) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 of FIG. 1). Some operations of method 900 are optionally combined and / or the order of some operations is optionally changed.

[0199] As described below, method 900 provides an intuitive way to facilitate authentication continuity for secure operations. The method reduces the cognitive burden on the user while performing secure operations, thereby creating a more efficient human-machine interface. For battery-powered computing devices, facilitating authentication continuity for secure operations to be faster and more efficient conserves power and extends the time between battery charges.

[0200] At a first time, a user of the device (e.g., 704) biometrically authenticates (902) using a biometric sensor to perform a first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed and / or a first level (e.g., full or high fidelity) fingerprint verification is performed). At a second time after the first time, a request to perform a secure operation is received (e.g., receiving a request to autofill (904) an input field (e.g., 706 and / or 708) (e.g., autofill a password and / or email information) and / or detect activation of a payment affordance).

[0201] In some embodiments, in response to receiving 906 a request to perform a secure operation, the secure operation is performed 908 (e.g., using autofill to enter pre-defined values ​​into fields (e.g., 706 and / or 708) of a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or initiating a payment transaction) in accordance with a determination that a respective criterion (e.g., a continuity criterion) was satisfied by a user of the device (e.g., 704) between the first and second times. In some embodiments, each criterion is based on sensor measurements taken at multiple intermediate times between the first time and the second time, including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, and detecting that the same user has used the device (e.g., 700) between the first time and the second time (e.g., the eye tracking continuity policy has not had a “false” result since the user (e.g., 704) previously performed a first level (e.g., full or high fidelity) iris feature verification and / or the fingerprint continuity policy has not had a “false” result since the user (e.g., 704) previously performed a first level (e.g., full or high fidelity) fingerprint verification).

[0202] In some embodiments, the secure operation is cancelled (910) upon a determination that the user (e.g., 704) of the device (e.g., 700) did not meet the respective criteria between the first and second times (e.g., an eye tracking continuity policy evaluates to “false” (e.g., eye gaze was undetectable for a period of time, the user (e.g., 704) was previously authenticated by passcode only, and / or the user (e.g., 704) was previously authenticated by fingerprint only), and / or a fingerprint tracking continuity policy evaluates to “false”). Ensuring that biometric continuity conditions are met when a user requests the secure operation to be performed provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0203] In some embodiments, in response to receiving a request to perform a secure operation, following a determination that a user (e.g., 704) of the device did not meet the respective criteria between the first and second times (e.g., an eye tracking continuity policy is evaluated as “false” (e.g., eyes are undetectable for a period of time, the user (e.g., 704) was previously authenticated by passcode only, and / or the user (e.g., 704) was previously authenticated by fingerprint only), and / or a fingerprint tracking continuity policy is evaluated as “false”), a user (e.g., 704) of the device is biometrically authenticated using a biometric sensor to perform a first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, and / or a first level (e.g., full or high fidelity) fingerprint verification is performed). Authenticating a user when a biometric continuity condition is not met provides improved security / privacy by verifying that an authorized user is interacting with the device.

[0204] In some embodiments, the first type of biometric authentication is based on a biometric feature of the eye (e.g., the whole eye or a portion of the eye such as an iris feature) and the second type of biometric authentication is based on a biometric feature of the eye (e.g., the whole eye or a portion of the eye such as an iris feature). Ensuring that the biometric continuity condition is met based on eye features provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0205] In some embodiments, the first type of biometric authentication is based on facial biometric features (e.g., eyebrows, eyelids, skin surrounding the eyes, forehead, nose) and the second type of biometric authentication is based on facial biometric features (e.g., eyebrows, eyelids, skin surrounding the eyes, forehead, nose). Ensuring that the biometric continuity condition is met based on facial features provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0206] In some embodiments, the user (e.g., 704) of the device meeting the respective criteria includes repeatedly performing a second type of biometric authentication (e.g., a second level (e.g., tracking eye information for continuity) iris feature verification) using one or more sensors of the computer system (e.g., 101 and / or 700), the second type of biometric authentication including receiving an input from the user (e.g., 704) including the first individual biometric information (e.g., obtaining eye information from a current frame) and comparing the received input to a previously received input including the second individual biometric information (e.g., comparing eye information from the current frame to eye information from a previous frame). Obtaining multiple sensor measurements at multiple intermediate times between the first and second times provides improved security / privacy by verifying that the same user has been using the device since the time of the initial authentication. In some embodiments, the one or more sensors used to perform the second type of biometric authentication include a biometric sensor used to perform the first type of biometric authentication (e.g., the one or more sensors include a biometric sensor used to obtain a new biometric authentication). Obtaining multiple sensor readings at multiple intermediate times between the first and second times provides improved security / privacy by verifying that the same user has been using the device since the time of the initial authentication.

[0207] In some embodiments, the first type of biometric authentication is based on a first biometric characteristic (e.g., eye and / or iris characteristics of a user (e.g., 704)) and the second type of biometric authentication is based on a second type of biometric characteristic (e.g., continuous wearing of the computer system and / or a fingerprint of the user (e.g., 704)), where the first biometric characteristic is different from the second biometric characteristic (e.g., iris feature verification is performed for initial authentication and facial features are monitored for continuity). In some embodiments, at a first time, a user (e.g., 704) of the device is biometrically authenticated based on a first biometric characteristic of the user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints). In some embodiments, a second biometric feature of the user (e.g., 704) is periodically detected (e.g., detecting the eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and / or cheeks of the user (e.g., 704)), and each criterion is based on the periodic detection of the second biometric feature of the user (e.g., 704) (e.g., the continuity criterion depends on detected details related to the presence of the eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and / or cheeks). Utilizing a separate biometric feature for initial authentication and a different biometric feature for biometric continuity verification provides improved security / privacy by improving authentication when features used for initial authentication are unavailable for continuity verification.

[0208] In some embodiments, the first type of biometric authentication is based on a first biometric feature (e.g., eye and / or iris features of a user (e.g., 704), such that iris feature verification is performed for initial authentication) and the second type of biometric authentication is based on the first type of biometric feature (e.g., eye and / or iris features of a user (e.g., 704), such that iris feature verification is performed for continuity). In some embodiments, at a first time, a user (e.g., 704) of a device is biometrically authenticated based on a first biometric feature of that user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints). In some embodiments, a first biometric feature of a user (e.g., 704) is periodically detected (e.g., detecting details related to the presence of eyes, iris features, pupils, and / or eye glints), and each criterion is based on the periodic detection of the first biometric feature of the user (e.g., 704) (e.g., the continuity criterion depends on the detected details related to the presence of eyes, iris features, pupils, and / or eye glints). Utilizing separate biometric features for both initial authentication and biometric continuity verification provides improved security / privacy by reducing the sensor types required for authentication.

[0209] In some embodiments, biometrically authenticating a user (e.g., 704) of the device at a first time includes obtaining sensor measurements at a plurality of intermediate times between the first time and the second time (e.g., detecting details related to the presence of eyes, iris features, pupils, and / or eye glints) based on a first biometric feature of the user (e.g., 704), where the first level (e.g., full or high fidelity) iris feature verification is performed including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints, and the obtained sensor measurements include at least one sensor measurement indicating that the first biometric feature of the user (e.g., 704) was not detected at a respective intermediate time of the plurality of intermediate times (e.g., the sensor measurement indicates the absence of the presence of eyes, iris features, pupils, and / or eye glints (e.g., based on the user's (e.g., 704's) eyes being closed). Utilizing a separate biometric feature for initial authentication that is hidden from view when biometric continuity verification is performed based on another feature provides improved security / privacy by ensuring that continuity verification can occur despite the feature being hidden.

[0210] In some embodiments, biometrically authenticating a user (e.g., 704) of the device at a first time includes based on a first biometric characteristic of the user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints), and periodically obtaining sensor measurements of a second biometric characteristic of the user (e.g., 704) at a plurality of intermediate times between the first time and the second time (e.g., detecting the eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and / or cheeks of the user (e.g., 704)), where the obtained sensor measurements indicate that the first biometric characteristic of the user (e.g., 704) was not detected at the respective intermediate times. The first biometric feature includes at least one sensor measurement (e.g., the sensor measurement does not indicate the presence of eyes, iris features, pupils, and / or eye glints (e.g., based on the user's (e.g., 704) eyes being closed)), the acquired sensor measurements include at least one sensor measurement indicating that a second biometric feature of the user (e.g., 704) was detected at the respective intermediate time (e.g., the sensor measurement indicates the presence of the user's (e.g., 704) eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, cheeks, while the presence of eyes, iris features, pupils, and / or eye glints is not detected), and the first biometric feature is different from the second biometric feature (e.g., iris feature verification is performed for initial authentication, but facial features are monitored for continuity). Utilizing a separate biometric feature for initial authentication that is hidden from view when biometric continuity verification is performed based on another feature that is visible creates an efficient and uninterrupted user experience for biometric authentication, encouraging users to use biometric authentication, thus providing a more secure experience, thereby resulting in improved security / privacy.

[0211] In some embodiments, each criterion (e.g., continuity criterion) between the first and second times includes a continuity criterion that is met when a biometric feature (e.g., an eye of a user (e.g., 704) and / or a fingerprint of a user (e.g., 704)) is repeatedly (e.g., continuously or periodically) detected between the first and second times (e.g., without determining that the biometric feature has not been detected between the first and second times). Obtaining multiple sensor measurements at multiple intermediate times between the first and second times provides improved security / privacy by verifying that the same user has been using the device since the time of initial authentication.

[0212] In some embodiments, biometrically authenticating a user (e.g., 704) of the device using a biometric sensor to perform a first type of biometric authentication includes providing (e.g., displaying via a display generating component (e.g., 120 and / or 700a)) an authentication guidance user interface (e.g., 724) to the user (e.g., 704) including instructions to perform the first type of biometric authentication, the instructions requesting the user (e.g., 704) to provide a particular movement of a biometric feature (e.g., instructions to gaze at a particular portion (e.g., 726) of the authentication guidance user interface, rotate the face (for facial authentication) along a particular path, and / or move a finger along a particular path). Displaying an authentication guidance user interface to facilitate user authentication provides improved security / privacy by reducing the chance of a failed authentication attempt.

[0213] In some embodiments, the secure operation includes providing payment information (e.g., as shown in FIG. 7N) (e.g., providing payment information to purchase items and / or services (e.g., providing credit card information to an e-commerce website to purchase shoes)). In some embodiments, the electronic device (e.g., 700) participates in a transaction. For example, upon detecting that input at a hardware user interface element (e.g., 722 and / or 758) has been received and other criteria have been met, a secure element in the computer system (e.g., 700) releases the payment information so that an application on the computer system (e.g., 700) can access the information (e.g., send the information to a server associated with the application). In some embodiments, the secure element is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms. In some embodiments, the secure element provides (or reveals) the payment information (e.g., an account number and / or a transaction-specific dynamic security code). In some embodiments, the secure element provides (or reveals) payment information in response to the device (e.g., 700) receiving authorization such as user authentication (e.g., fingerprint authentication, passcode authentication, detecting a double press of a hardware button (e.g., 722 and / or 758) by providing authentication credentials to the device (e.g., 700) when the device (e.g., 700) is in an unlocked state and optionally when the device (e.g., 700) is continuously on the user's (e.g., 704) wrist, with the continued presence of the device (e.g., 700) on the user's (e.g., 704) wrist being determined by periodically checking whether the device (e.g., 700) is in contact with the user's (e.g., 704) skin).In some embodiments, the secure element provides (or releases) the payment information in response to a continuity criterion (e.g., a second level (e.g., continuity) iris feature verification is performed) and / or a gaze criterion (e.g., gaze is directed at the user interface and / or the target of the user's gaze is the user interface) being met. Enabling user authorization of a payment operation using biometric continuity verification provides improved security / privacy by ensuring that the same user is using the device from the time of initial authentication.

[0214] In some embodiments, the secure operation includes providing access credentials to access an application (e.g., an application associated with a third party service (e.g., providing facial recognition data to access a stock trading application or a password storage application)). In some embodiments, the secure operation includes providing access credentials to access an application (e.g., an application associated with a third party service (e.g., providing facial recognition data to access a stock trading application or a password storage application)). Enabling user permission for application authentication using biometric continuity verification provides improved security / privacy by ensuring that the same user is using the device from the time of initial authentication.

[0215] In some embodiments, the secure operation includes autofilling user credentials (e.g., username and / or password information to access secure information (e.g., providing login information to access a banking website (e.g., 706 and / or 708)). Enabling user permission for the autofill operation using biometric continuity verification provides improved security / privacy by ensuring that the same user is using the device from the time of initial authentication.

[0216] In some embodiments, receiving a request to perform a secure operation includes detecting a physical input to activate a hardware button (e.g., 722 and / or 758) (e.g., a user (e.g., 704) engaging in a double press of a hardware button to initiate a payment operation and / or a user (e.g., 704) engaging in a double click of a hardware button to initiate an autofill operation). In some embodiments, the hardware button (e.g., 722 and / or 758) when activated using various inputs performs various respective functions such as navigating to a home user interface, powering off the device, displaying a system user interface for powering off the device, and / or activating a voice assistant. In some embodiments, the physical input includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a press followed by a rotation, and / or multiple presses in succession (e.g., double press, triple press). Utilizing a hardware element for authorization provides improved security / privacy by reducing the likelihood of erroneous authorization and / or preventing unauthorized input from malicious software.

[0217] In some embodiments, a system user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is provided to a user (e.g., 704) (e.g., displayed via a display generation component (e.g., 120 and / or 700a)), which includes an affordance associated with a secure operation (e.g., an affordance labeled "Payment" or an affordance labeled "Autofill"), where receiving a request to perform the secure operation includes detecting activation of an affordance (e.g., 710 and / or 750) associated with the secure operation (e.g., a user (e.g., 704) activates the affordance with gaze, gaze and a hardware button, or a hardware button only). In some embodiments, the user gazes at the affordance (e.g., 710 and / or 750), and the affordance (e.g., 710 and / or 750) is activated pursuant to a determination that the user has gazed at the affordance for a threshold period of time. In some embodiments, a user (e.g., 704) activates an affordance (e.g., 710 and / or 750) by activating a hardware button (e.g., 722 and / or 758) (e.g., a double press of a hardware button and / or a long press of a hardware button). In some embodiments, a user (e.g., 704) activates an affordance (e.g., 710 and / or 750) by activating a hardware button (e.g., 722 and / or 758) (e.g., a double press of a hardware button and / or a long press of a hardware button), and the affordance (e.g., 710 and / or 750) is activated pursuant to a determination that the user (e.g., 704) is gazing at the affordance (e.g., 710 and / or 750) while the user (e.g., 704) activates the hardware button.In some embodiments, a user (e.g., 704) activates an affordance (e.g., 710 and / or 750) by controlling a secondary device (e.g., a controller) to select an affordance (e.g., 710 and / or 750) (e.g., by moving a cursor over the affordance while activating a hardware button and / or by using a pointing device while activating a hardware button (e.g., 722 and / or 758)). Leveraging affordances within the system user interface for authorization provides improved security / privacy when the device is utilized with one or more alternative input devices.

[0218] In some embodiments, receiving a request to perform a secure operation includes detecting a user activation of an affordance (e.g., 710 and / or 750) within a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., the user (e.g., 704) activates an affordance labeled “Payment Authorization” or the user (e.g., 704) activates an affordance labeled “Autofill Authorization”) while providing (e.g., displaying via a display generation component (e.g., 120 and / or 700a)) a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) that includes information regarding the secure operation (e.g., 706, 708, 710, 716a, 716b, 744a, and / or 744b) to a user (e.g., 704). Displaying information about the secure operation when requesting permission provides improved security / privacy by ensuring that the user has an opportunity to review information related to the secure operation before allowing the secure operation to proceed.

[0219] In some embodiments, each criterion includes an eyes-open criterion that is met when one or more eyes (e.g., one and / or both eyes) of a user (e.g., 704) do not remain continuously closed for more than a threshold time period (e.g., non-zero) (e.g., the continuity criterion is not met when the user's (e.g., 704) eyes are closed for more than a threshold time period), and an eyes-open criterion that is not met when one or more eyes (e.g., one and / or both eyes) of a user (e.g., 704) remain continuously closed for more than a threshold time period (e.g., non-zero) (e.g., the continuity criterion is not met when the user's (e.g., 704) eyes are closed for more than a threshold time period). Determining that bio-continuity is not met when the eyes are detected as closed provides improved security / privacy by ensuring informed consent regarding the operation.

[0220] In some embodiments, each criterion is based on biometric features that are available for analysis regardless of whether the user's (e.g., 704) eyes are closed (e.g., regardless of the duration the user's (e.g., 704) eyes are closed) (e.g., the continuity criterion may be met even when the user's (e.g., 704) eyes are closed for an extended period of time). Determining that biometric continuity is met when the eyes are detected as closed provides improved security / privacy by improving authentication when features used for initial authentication are unavailable for continuity verification.

[0221] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchanged, substituted, and / or added between these methods. For example, the determination in method 900 of whether a user has met particular (e.g., persistence) criteria is optionally used as part of method 800 to determine whether to activate user permissions. For the sake of brevity, those details will not be repeated here.

[0222] Additional description regarding FIGS. 7A-7N is provided below with reference to the method 1000 described with respect to FIGS. 7A-7N.

[0223] FIG. 10 is a flow diagram of an exemplary method 1000 of continuity of authentication for secure operation, according to some embodiments. In some embodiments, the method 1000 is executed in a computer system (e.g., computer system 101 of FIG. 1 (e.g., a smartphone, a smartwatch, a tablet, and / or a wearable device)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a display controller, a touch-sensitive display system, a display (e.g., embedded and / or connected), a 3D display, a transparent display, a projector, and / or a head-up display) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and / or one or more input devices. In some embodiments, the method 1000 is governed by a plurality of instructions that are stored in a non-transitory (or transitory) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 of FIG. 1). Some operations of the method 1000 are optionally combined and / or the order of some operations is optionally changed.

[0224] As described below, the method 1000 provides an intuitive way to authorize secure operations through an accessibility interface. The method reduces the cognitive load on the user while performing secure operations through the accessibility interface, thereby creating a more efficient human-machine interface. In the case of battery-operated computing devices, faster and more efficient authorization of secure operations through the accessibility interface conserves power and extends the time between battery charges.

[0225] A request is received (1002) via one or more input devices (e.g., 125 and / or 150) for display of a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) associated with performing a secure operation (e.g., a request to perform an autofill procedure and / or a request to perform a payment transaction). In response to a request to display a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) related to performing a secure operation, a first user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is displayed (1004) via the display generation component (e.g., 120 and / or 700a) (e.g., an interface displayed in response to receiving a request to perform an autofill procedure and / or an interface displayed in response to receiving a request to perform a payment transaction) that includes respective prompts (e.g., 720, 734, and / or 756) for providing additional input (e.g., login details, account details, and / or information related to a pending purchase) to authorize the device to perform the secure operation.

[0226] In accordance with a determination that the computer system (e.g., 700) is operating in a first mode (e.g., a standard mode in which accessibility features are not enabled and / or a mode that allows secure operation using a physical input mechanism (e.g., 722 and / or 758)), the separate prompt is a first prompt (e.g., 722, 734 and / or 758) to provide a physical input (e.g., 720 and / or 756) (1006) (e.g., the first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) prompts the user (e.g., 704) to provide a physical input (e.g., 722 and / or 758) (e.g., the physical input mechanism (e.g., 7. In one embodiment, the user (e.g., 704) may provide a physical input (e.g., 722 and / or 758) using a depressible button (e.g., 722 and / or 758), press, and / or successive presses of a button (e.g., 722 and / or 758) (e.g., the prompts (e.g., 720, 734, and / or 756) indicate that if the user (e.g., 704) intends to authorize an autofill procedure, the user (e.g., 704) should provide a physical input (e.g., 722 and / or 758) and / or the prompts (e.g., 720, 734, and / or 756) indicate that if the user (e.g., 704) intends to authorize a purchase, the user (e.g., 704) should provide a physical input (e.g., 722 and / or 758)).

[0227] Pursuant to a determination that the computer system (e.g., 700) is operating in a second mode different from the first mode (e.g., an accessibility mode and / or a mode in which input at a physical input mechanism (e.g., 722 and / or 758) is not required to permit secure operation), the separate prompt (e.g., 720, 734 and / or 756) is a second prompt for displaying a second user interface (e.g., 742) (e.g., the first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) does not prompt the user (e.g., 704) that the user should provide a physical input (e.g., 722 and / or 758) if the user (e.g., 704) intends to permit the autofill procedure, and / or ... (e.g., 704) that the user should provide a physical input (e.g., 722 and / or 758) if the user (e.g., 704) intends to authorize the purchase), in which case the second user interface (e.g., 742) enables (1008) user authorization of secure operation without using a physical input (e.g., 722 and / or 758) (e.g., enabling authorization of secure operation using a first level (e.g., full or high fidelity) iris feature verification (e.g., the first level of iris feature verification occurs while the user (e.g., 704) activates an affordance (e.g., 744), and / or the first level of iris feature verification occurs while the user (e.g., 704) gazes at an affordance (e.g., 744), and / or enabling authorization of secure operation using a first level (e.g., full or high fidelity) fingerprint verification). Being able to permit secure operation without physical input provides improved security / privacy when the device is utilized with one or more alternative input devices.

[0228] In some embodiments, a first distinct affordance (e.g., 740 and / or 750) is provided in a second prompt (e.g., 736 and / or 738) (e.g., the affordance includes an indication (e.g., 734) that authorization of the secure action is to be performed using a secondary input type that is different from the primary input type (e.g., “Autofill” is displayed next to “Confirm with Assistive Touch” or an affordance for “Pay” is provided next to “Confirm with Assistive Touch” (e.g., displayed via a display generation component (e.g., 120 and / or 700a))) and then a user selection of an affordance (e.g., 740 and / or 750) is received (e.g., a user selects an affordance using a secondary input type that is different from the primary input type (e.g., “Autofill” is displayed next to “Confirm with Assistive Touch” or an affordance for “Pay” is provided next to “Confirm with Assistive Touch” (e.g., displayed via a display generation component (e.g., 120 and / or 700a))). (e.g., 704) selects an “autofill” affordance (e.g., 740) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input, or a user (e.g., 704) selects a “pay” affordance (e.g., 750) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input.) In some embodiments, the user (e.g., 704) gazes at an affordance (e.g., 740 and / or 750), and the affordance (e.g., 740 and / or 750) is activated pursuant to a determination that the user (e.g., 704) has gazed at the affordance (e.g., 740 and / or 750) for a threshold period of time.In some embodiments, a user (e.g., 704) activates an affordance (e.g., 740 and / or 750) by controlling an adaptive accessory (e.g., a switch control) to select an affordance (e.g., 740 and / or 750) (e.g., the user (e.g., 704) taps the switch control a first time to initiate the display of a sliding vertical line that slides back and forth across the display (e.g., 700a), the user (e.g., 704) taps the switch control a second time when the vertical line appears over the affordance (e.g., 740 and / or 750), and in response to the second tap, a horizontal line is displayed that slides up and down across the display (e.g., 700a), and the user (e.g., 704) taps the switch control a third time when the horizontal line appears over the affordance (e.g., 740 and / or 750)). In some embodiments, a user (e.g., 704) controls an adaptive accessory to select an affordance (e.g., 740 and / or 750), and an affordance (e.g., 740 and / or 750) is activated pursuant to a determination that the user (e.g., 704) gazes at an affordance (e.g., 740 and / or 750) while the user (e.g., 704) selects the affordance (e.g., 740 and / or 750). An adaptive accessory is an accessory that allows a user (e.g., 704) to provide input via alternative means compared to a typical usage scenario of the device; one example of an adaptive accessory is a switch control that provides a limited number of selection states (e.g., between 1 and 5) used to select among options provided by a device (e.g., 700).One example of a switch control is a device that can be activated by pressing, pulling, blinking, squeezing, and / or sucking / blowing through a straw, allowing a user with cognitive or physical accessibility needs to interact with a device (e.g., 700) when the user (e.g., 704) is unable to easily interact with the device (e.g., 700) using standard input methods (e.g., using a touchscreen, physical buttons / knobs, and / or air gestures).

[0229] In some embodiments, pursuant to receiving a user selection of an affordance, a second user interface (e.g., 742) is displayed, where the second user interface (e.g., 742) includes a representation of a plurality of functions (e.g., displaying an accessibility menu (e.g., 742) including a plurality of affordances representing device functionality), including a first distinct function of the plurality of functions that is invoked in response to a secondary input type that is different from the primary input type (e.g., the accessibility menu (e.g., 742) includes options such as “Home,” “Control Center,” “Devices,” “Notifications,” or “Custom”) and a second distinct function of the plurality of functions (e.g., 744) that is invoked in response to a tertiary input type that is different from the primary and secondary input types (e.g., the accessibility menu (e.g., 742) includes a new option “Allow Autofill” or a new option “Payments”). In some embodiments, the accessibility menu (e.g., 742) includes a grid of affordances and an outline surrounding one of the affordances (e.g., 744) that periodically (e.g., every 2 seconds or every 5 seconds) moves the outline to surround another affordance (e.g., 744) and allows the user (e.g., 704) to provide input (e.g., input to a switch control and / or gaze input directed at the surrounded affordance) to select the currently outlined affordance (e.g., 744). Displaying an accessibility interface with alternative input options provides security / privacy improvements when the device is utilized with one or more alternative input devices.

[0230] In some embodiments, the secondary input type simulates input of the primary input type (e.g., invoking "home" in response to a swipe up from a button on the screen), and the tertiary input type simulates input of the primary input type (e.g., in response to a custom input (e.g., long press) via an accessibility menu (e.g., 742)). Enabling simulated input in place of traditional input types provides improved security / privacy when the device is utilized with one or more alternative input devices.

[0231] In some embodiments, while the computer system (e.g., 700) is operating in the second mode, affordances (e.g., 740 and / or 750) associated with the respective prompts are displayed (e.g., displaying an “Autofill” affordance with the text “Confirm with assistive touch” or displaying a “Pay” affordance with the text “Confirm with assistive touch”) and pursuant to receiving user input associated with the respective prompts (e.g., a user (e.g., 704) uses gaze, gaze and adaptive accessory input, and / or selects the “Autofill” affordance (e.g., 740) or a user (e.g., 704) uses gaze, gaze and adaptive accessory input, and / or selects the “Pay” affordance (e.g., 750)) and a second user interface (e.g., 742) is displayed to prompt the second user to enter a user input. The user interface (e.g., 742) includes an affordance (e.g., 744) for allowing secure operation (e.g., displaying an accessibility menu (e.g., 742) including a new option “Allow Autofill” or a new option “Payment”), where the affordance (e.g., 744) for allowing secure operation is invoked in response to a secondary input type different from the primary input type (e.g., the option “Allow Autofill” is invoked in response to a custom input (e.g., long press) via the accessibility menu (e.g., 742) instead of the primary input type (e.g., double pressing the side buttons (e.g., 722 and / or 758)), or the option “Payment” is invoked in response to a custom input (e.g., long press) via the accessibility menu (e.g., 742) instead of the primary input type (e.g., double pressing the side buttons (e.g., 722 and / or 758)). Displaying an accessibility interface including an affordance for allowing secure operation provides security / privacy improvements when the device is utilized with one or more alternative input devices.

[0232] In some embodiments, details related to performing a secure operation (e.g., an interface includes details for performing an autofill procedure (e.g., username and password fields) or an interface includes details for performing a payment transaction (e.g., product name, product description, product price)) are displayed within a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) related to performing a secure operation. In some embodiments, while the computer system (e.g., 700) is operating in a second mode (e.g., an accessibility mode and / or a mode in which input via a physical input mechanism (e.g., 722 and / or 758) is not required to permit a secure operation), an affordance (e.g., 740 and / or 750) related to a separate prompt is displayed (e.g., displaying an affordance for "autofill" with the text "confirm with assistive touch" or displaying an affordance for "pay" with the text "confirm with assistive touch").In some embodiments, in response to receiving user input associated with the respective prompt (e.g., a user (e.g., 704) selects an “autofill” affordance (e.g., 740) using gaze, gaze plus adaptive accessory input, and / or an adaptive accessory input, or a user (e.g., 704) selects a “pay” affordance (e.g., 750) using gaze, gaze and adaptive accessory input, and / or an adaptive accessory input), and determining that the user (e.g., 704) is gazed at a user interface associated with performing a secure operation while the user input associated with the respective prompt is received (e.g., a user (e.g., 704) selects an “autofill” affordance (e.g., 740) while gazed at an interface including details for performing an autofill procedure, or a user (e.g., 704) selects a “pay” affordance (e.g., 750) while gazed at an interface including details for performing a payment transaction), A second user interface (e.g., 742) is displayed, where the second user (e.g., 742) interface includes an affordance (e.g., 744) for allowing the secure action (e.g., displaying an accessibility menu (e.g., 742) including, for example, a new option “Allow Autofill” or a new option “Payment”), where the affordance (e.g., 744) for allowing the secure action is displayed in response to a secondary input type different from the primary input type (e.g., for example, the option “Allow Autofill” is invoked in response to a custom input (e.g., long press) via the accessibility menu (e.g., 742) (instead of the primary input type (e.g., double press of the side buttons (e.g., 722 and / or 758))), or the option “Payment” is invoked in response to a custom input (e.g., long press) via the accessibility menu (e.g., 742) instead of the primary input type (e.g., double press of the side buttons (e.g., 722 and / or 758)).Displaying an accessibility interface in response to a combination of gaze and affordance activation provides security / privacy improvements when the device is utilized with one or more alternative input devices.

[0233] In some embodiments, while a computer system (e.g., 700) is operating in a first mode (e.g., a standard mode without accessibility features enabled and / or a mode that allows secure operation using a physical input mechanism), a user activation of a physical input (e.g., 722 and / or 758) is received (e.g., a user (e.g., 704) performs a predefined activation pattern of a hardware input affordance, such as a double click of a hardware button (e.g., 722 and / or 758)) while a first prompt (e.g., 720, 734, and / or 756) is displayed). In some embodiments, the predefined activation pattern includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a rotation followed by a press, and / or multiple presses in succession (e.g., a double press or a triple press). In some embodiments, in response to receiving a user activation of a physical input (e.g., 722 and / or 758) while the first prompt (e.g., 720, 734, and / or 756) is displayed, a secure operation is permitted (e.g., permitting an autofill procedure in response to activation of a hardware button (e.g., 722 and / or 758) or permitting a payment procedure in response to activation of a hardware button (e.g., 722 and / or 758)) pursuant to determining that a user authentication criterion is met (e.g., the user (e.g., 704) is gazing at the prompt (e.g., 720, 734, and / or 756)). In some embodiments, pursuant to a determination that user authentication criteria has not been met (e.g., the user (e.g., 704) has not gazed upon a prompt (e.g., 720, 734, and / or 756)), permission for secure operation is withheld (e.g., preventing an autofill procedure in response to a double-click of a hardware button (e.g., 722 and / or 758) or preventing a payment procedure in response to a double-click of a hardware button (e.g., 722 and / or 758)).Utilizing accessible and inaccessible modes of operation provides improved security / privacy when the device is utilized with one or more alternative input devices.

[0234] In some embodiments, the authentication criteria include criteria that are met when a user (e.g., 704) provides a specific movement of a biometric feature (e.g., gazes at a specific portion of an authentication guidance user interface (e.g., 724), provides a facial rotation along a specific path, and / or moves a finger along a specific path). Requiring a specific movement of a biometric feature while a physical input mechanism is activated provides improved security / privacy by ensuring informed consent for secure operations.

[0235] In some embodiments, input is received from a user (e.g., 704) to allow secure operation (e.g., while in standard mode, a user (e.g., 704) performs activation of a hardware button (e.g., 722 and / or 758)) (e.g., single press, long press, single press followed by long press, press and rotate, press and rotate, multiple presses in succession (e.g., double press, triple press), or while in accessibility mode, an option “Allow Autofill” is invoked in response to custom input via an accessibility menu (e.g., 742) or in response to a command from the accessibility menu). In response to receiving the input to authorize secure operation, a user authentication is performed (e.g., a first level (e.g., full or high fidelity) iris verification is performed, a second level (e.g., continuous) iris verification is performed, or a passcode verification is performed), and in response to a determination that the user (e.g., 704) is authenticated, the secure operation is authorized (e.g., determining if authentication continuity has been true since the last full or high fidelity biometric authentication, performing a full or high fidelity biometric authentication, and / or performing a passcode authentication). In some embodiments, in response to a determination that the user (e.g., 704) is not authenticated, the secure operation is not authorized (e.g., the device ceases to authorize the secure operation). Requiring user authentication in addition to user authorization for secure operation provides improved security / privacy by adding an additional layer of verification specific to the user requesting authorization.

[0236] In some embodiments, performing the user authentication includes performing a second type of biometric authentication (e.g., a second level (e.g., tracking eye information for continuity) iris feature verification is performed) (e.g., performing includes repeating the authentication every 2 frames (e.g., a frame includes a still image or set of images obtained from a sensor (e.g., biometric sensor) of the device), every 3 frames, every 4 frames, every 5 frames, every 6 frames, every 7 frames, every 8 frames, every 9 frames, or every 10 frames), where the second type of biometric authentication includes receiving an input from a user (e.g., 704) including the first individual biometric information (e.g., obtaining eye information from the current frame) and comparing the received input to a previously received input including the second individual biometric information (e.g., comparing eye information from the current frame to eye information from the previous frame). Utilizing stored authentication information in addition to user authorization for secure operation provides improved security / privacy by verifying that the same user has been using the device since the time of the initial authentication.

[0237] In some embodiments, performing user authentication includes performing a first type of biometric authentication, where the first type of biometric authentication includes detecting the presence of a first biometric feature type (e.g., a first level (e.g., full or high fidelity) verification based on eye features, or a first level (e.g., full or high fidelity) verification based on facial features). Requiring full or high fidelity biometric authentication in addition to user authorization for secure operations provides improved security / privacy by verifying that an authorized user is requesting the secure operation.

[0238] In some embodiments, performing user authentication includes providing a prompt (e.g., 728) requesting a passcode from the user (e.g., 704); receiving a passcode from the user (e.g., 704) in response to the prompt (e.g., 728) (e.g., receiving a plurality of numeric inputs (e.g., 0-9)); determining that the user (e.g., 704) is authenticated in accordance with a determination that the received passcode matches a stored passcode; and allowing a secure operation in accordance with a determination that the user (e.g., 704) is authenticated (e.g., the user interface is not grayed out, a button is marked as active, or the user interface provides text (e.g., "please proceed with the request") indicating that the user (e.g., 704) can request to perform a secure operation). In some embodiments, in accordance with a determination that the received passcode does not match the stored passcode, a determination is made that the user (e.g., 704) is not authenticated and the secure operation is not allowed. Requiring passcode authentication in addition to user authorization for secure operation provides improved security / privacy by providing an additional method for verifying the identity of the user.

[0239] In some embodiments, an input for authorizing secure operation is detected, and in response to detecting the input for authorizing secure operation, a determination is made as to whether the input is a physical input for authorizing secure operation (e.g., 722 and / or 758) (e.g., an activation procedure of a physical button (e.g., 722 and / or 758) for authorizing autofill, or an activation procedure of a physical button (e.g., 722 and / or 758) for authorizing payment), in some embodiments the activation procedure includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a press followed by a rotation, or multiple presses in succession (e.g., double press, triple press). In some embodiments, authentication information is retrieved, and the authentication information is associated with a second type of biometric authentication that is repeatedly performed (e.g., retrieving continuity information related to a previously performed second level (e.g., tracking eye information for continuity) iris feature verification). In some embodiments, a determination is made as to whether the input is associated with a second user interface (e.g., 742) in response to determining that the input is not a physical input for authorizing secure operation. In some embodiments, secure operation is permitted pursuant to a determination that the user (e.g., 704) is authenticated based on the second type of biometric authentication (e.g., permitting autofill if a second level (e.g., tracking eye information for continuity) of iris feature verification is successful, or permitting payment if a second level (e.g., tracking eye information for continuity) of iris feature verification is successful). In some embodiments, secure operation is not permitted pursuant to a determination that the user (e.g., 704) is not authenticated based on the second type of biometric authentication (e.g., tracked eye information indicates that the user's (e.g., 704) eyes were closed for a period of time (e.g., 5 seconds, 7 seconds, or 10 seconds) during interaction with the device).

[0240] In some embodiments, if an input associated with the second user interface (e.g., 742) enables user authorization of a secure operation without using a physical input (e.g., the detected input is associated with the option “autofill” or the new option “pay”), a first type of biometric authentication is performed in accordance with a determination that the input is an input associated with the second user interface (e.g., 742) (e.g., detecting the input via an accessibility menu), and in accordance with a determination that the user (e.g., 704) is authenticated based on the first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed), the secure operation is permitted (e.g., permitting autofill if the first level (e.g., full or high fidelity) iris feature verification is successful, or permitting payment if the first level (e.g., full or high fidelity) iris feature verification is successful). In some embodiments, the secure operation is not permitted in accordance with a determination that the user (e.g., 704) is not authenticated based on the first type of biometric authentication. In some embodiments, following a determination that the input is not an input associated with the second user interface (e.g., 742), a determination is made whether the input is an additional input not associated with a physical input (e.g., 722 and / or 758) for authorizing a secure operation. Requiring full or high-fidelity biometric authentication when the device operates in an accessible mode provides improved security / privacy by verifying that an authorized user is requesting a secure operation.

[0241] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchanged, substituted, and / or added between these methods. For example, the determination in method 900 that a user has met particular (e.g., persistence) criteria is optionally used to allow secure operation via an accessibility interface as part of method 1000. For the sake of brevity, those details will not be repeated here.

[0242] The above has been described in terms of specific embodiments for purposes of illustration. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular uses contemplated.

[0243] As mentioned above, one aspect of the present technology is the collection and use of data available from various sources to improve authorization of secure operations. This disclosure contemplates that in some cases, this collected data may include personal information data that uniquely identifies a particular person or that can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, addresses, data or records regarding the user's (e.g., 704) health or fitness level (e.g., vital sign measurements, medication information, exercise information), birth date, or any other identifying or personal information.

[0244] This disclosure recognizes that the use of such personal information data in the present technology may be for the benefit of the user. For example, the personal information data may be used to permit secure operation. Additionally, other uses of personal information data that benefit the user are also contemplated by this disclosure. For example, health and fitness data may be used to provide insight into the overall wellness of the user, or may be used as proactive feedback to individuals using the technology in pursuit of wellness goals.

[0245] This disclosure contemplates that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will adhere to robust privacy policies and / or privacy practices. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for keeping personal information data confidential and secure. Such policies should be easily accessible by users and should be updated as data collection and / or use changes. Personal information from users should be collected for the entity's lawful and legitimate use and should not be shared or sold except for those lawful uses. Furthermore, such collection / sharing should be carried out after informing and obtaining consent from the user. Moreover, such entities should consider taking all necessary measures to protect and secure access to such personal information data and ensure that others who have access to the personal information data adhere to their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to attest to their adherence to widely accepted privacy policies and practices. Moreover, policies and practices should be adapted to the specific types of personal information data being collected and / or accessed, and should conform to applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, collection of or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). Meanwhile, health data in other countries may be subject to other regulations and policies and should be addressed accordingly. Therefore, different privacy practices should be maintained with respect to different types of personal data in each country.

[0246] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of allowing secure operations, the present technology may be configured to allow a user to select to "opt-in" or "opt-out" of participating in the collection of personal information data during registration for the service or at any time thereafter. In another example, a user may choose not to provide data for customization of secure operations such as payments and autofill. In yet another example, a user may choose to limit the length of time that data is maintained or completely prohibit the deployment of customized services to facilitate secure operations such as payments or autofill. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications regarding access or use of personal information. For example, a user may be notified upon download of an app that will access the user's personal information data, and then again immediately before the personal information data is accessed by the app.

[0247] Moreover, it is the intent of this disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintended or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting it when it is no longer needed. In addition, where applicable in certain health-related applications, anonymization of data can be used to protect user privacy. De-identification can be facilitated by removing certain identifiers (e.g., date of birth, etc.) where appropriate, controlling the amount or specificity of data stored (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0248] Thus, while this disclosure broadly encompasses the use of personal information data to realize one or more of the various disclosed embodiments, this disclosure also contemplates that the various embodiments may be practiced without requiring access to such personal information data. That is, the various embodiments of the technology are not rendered inoperable by the absence of all or a portion of such personal information data. For example, allowing secure operation may be facilitated based on non-personal information data or a minimal amount of personal information, such as content requested by a device associated with the user, other non-personal information available to the service, or publicly available information.

Claims

1. A computer system in communication with one or more input devices and a display generating component, comprising: detecting, via the one or more input devices, a change in the user's current viewpoint while a three-dimensional environment including a virtual user interface object that includes information regarding a secure operation is visible via the display generation component; In response to detecting the change in the viewpoint of the user, enabling user authorization of the secure operation using the virtual user interface object in accordance with a determination that at least a threshold amount of the virtual user interface object is visible from the viewpoint of the user and the user is authorized to perform the secure operation; withdrawing from enabling user permission of the secure operation using the virtual user interface object in response to a determination that fewer than the threshold amount of the virtual user interface object is visible from the viewpoint of the user; A method comprising:

2. receiving an input from the user to authorize performance of the secure operation; and in response to receiving the input to authorize performance of the secure operation, initiating execution of the secure operation in accordance with the determination that a set of one or more execution criteria are satisfied, the set including a visibility criterion that is satisfied when at least the threshold amount of the virtual user interface objects are visible from the viewpoint of the user; aborting initiation of execution of the secure operation in accordance with the determination that the set of one or more execution criteria is not met; and The method of claim 1 , comprising:

3. The method of claim 2 , wherein the set of one or more performance criteria includes a user authentication criterion that is satisfied when the user is authenticated.

4. detecting biometric information from the user; comparing the biometric information with enrolled biometric information, and determining whether the user authentication criteria is met is based on the comparison of the biometric information with the enrolled biometric information. The method according to claim 3.

5. 5. The method of claim 3 or 4, wherein determining that the user authentication criteria is met is made in response to receiving the input to authorize performance of the secure operation.

6. 6. The method of claim 3, wherein determining whether the user authentication criteria are met comprises retrieving stored authentication information, the stored authentication information being indicative of one or more previously performed biometric authentications.

7. The determination of whether the user authentication standard is satisfied is performing a first type of biometric authentication; and repeatedly performing a second type of biometric authentication after performing the first type of biometric authentication, the second type of biometric authentication comprising: receiving an input from the user including a first individual biometric information; and comparing the received input to a previously received input including a second individual biometric information.

7. The method according to any one of claims 3 to 6.

8. performing the first type of biometric authentication includes detecting a presence of a first biometric feature type; performing the second type of biometric authentication includes detecting a presence of a second biometric feature type, and the first biometric feature type and the second biometric feature type correspond to a same biometric feature type. The method according to claim 7.

9. The method of claim 3 , wherein the user authentication criterion is met when the user is authenticated based on an eye biometric.

10. The method of claim 3 , wherein the user authentication criterion is met when a passcode input matches a registered passcode.

11. providing a prompt to enable authentication using a passcode in response to a determination that the user authentication criteria is not met.

11. The method according to any one of claims 3 to 10.

12. The method of claim 1 , wherein the user authorization of the secure operation using the virtual user interface object includes user activation of a hardware user interface element.

13. The method of claim 1 , wherein the secure action is a payment.

14. The method of claim 1 , wherein the secure action comprises auto-filling user credentials.

15. varying an amount of the virtual user interface object that is visible to the user in response to detecting the change in the viewpoint of the user; 15. The method of any one of claims 1 to 14, comprising:

16. The method of claim 15 , wherein the amount of the virtual user interface object that is visible changes based on occlusion by a physical object.

17. The method of claim 15 , wherein the amount of the virtual user interface object that is visible varies based on occlusion by a virtual object.

18. The method of claim 1 , wherein an amount of the virtual user interface object that is visible changes based on the virtual user interface object moving out of the field of view of the user.

19. 19. The method of claim 1, wherein an amount of the virtual user interface object that is visible changes based on the virtual user interface object moving beyond a threshold distance from the center of the user's field of view.

20. The method of claim 1 , wherein an amount of the virtual user interface object that is visible varies based on occlusion by a physical object.

21. 21. The method of claim 1, wherein the determining that the user is authorized to perform the secure operation includes determining whether a gaze criterion is met when the user gazes at the virtual user interface object.

22. 22. The method of claim 1, wherein the determining that the user is authorized to perform the secure operation comprises determining whether an eye criterion is met when the user's eyes are open.

23. modifying an appearance of the virtual user interface object to indicate to the user that the user may request performance of the secure operation in accordance with the determination that at least a threshold amount of the virtual user interface object is visible from the viewpoint of the user; modifying the appearance of the virtual user interface objects to indicate to the user that the user cannot request performance of the secure operation in accordance with the determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint; 23. The method of any one of claims 1 to 22, comprising:

24. the three-dimensional environment includes a virtual object associated with the secure operation, the method comprising: pasting the virtual user interface object onto the virtual object associated with the secure action, where moving the virtual object causes the pasted virtual user interface object to move with the virtual object.

24. The method of any one of claims 1 to 23.

25. 25. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and one or more input devices, the one or more programs including instructions for performing the method of any one of claims 1 to 24.

26. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs including instructions for performing the method of any one of claims 1 to 24.

27. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: A computer system comprising means for carrying out the method according to any one of claims 1 to 24.

28. 1. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and one or more input devices, the one or more programs comprising: detecting a change in the user's current viewpoint via the one or more input devices while a three-dimensional environment including a virtual user interface object that includes information regarding a secure operation is visible via the display generation component; In response to detecting the change in the viewpoint of the user, enabling user authorization of the secure operation using the virtual user interface object in accordance with a determination that at least a threshold amount of the virtual user interface object is visible from the viewpoint of the user and the user is authorized to perform the secure operation; instructions for ceasing to enable user authorization of the secure operation using the virtual user interface objects in response to a determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint. A non-transitory computer-readable storage medium.

29. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising: detecting a change in the user's current viewpoint via the one or more input devices while a three-dimensional environment including a virtual user interface object that includes information regarding a secure operation is visible via the display generation component; In response to detecting the change in the viewpoint of the user, enabling user authorization of the secure operation using the virtual user interface object in accordance with a determination that at least a threshold amount of the virtual user interface object is visible from the viewpoint of the user and the user is authorized to perform the secure operation; instructions for ceasing to enable user authorization of the secure operation using the virtual user interface objects in response to a determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint. Computer system.

30. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: means for detecting, via the one or more input devices, a change in the user's current viewpoint while a three-dimensional environment is visible via the display generation component, the three-dimensional environment including a virtual user interface object that includes information regarding a secure operation; In response to detecting the change in the viewpoint of the user, means for validating user authorization of the secure operation using the virtual user interface object in accordance with a determination that at least a threshold amount of the virtual user interface object is visible from the viewpoint of the user and the user is authorized to perform the secure operation; means for withdrawing from enabling user permission of the secure operation using the virtual user interface object in accordance with a determination that fewer than the threshold amount of the virtual user interface object is visible from the user's viewpoint. Computer system.

31. 25. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs including instructions for performing the method of any one of claims 1 to 24.

32. 1. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising: detecting a change in the user's current viewpoint via the one or more input devices while a three-dimensional environment including a virtual user interface object that includes information regarding a secure operation is visible via the display generation component; In response to detecting the change in the viewpoint of the user, enabling user authorization of the secure operation using the virtual user interface object in accordance with a determination that at least a threshold amount of the virtual user interface object is visible from the viewpoint of the user and the user is authorized to perform the secure operation; instructions for ceasing to enable user authorization of the secure operation using the virtual user interface objects in response to a determination that fewer than the threshold amount of the virtual user interface objects are visible from the user's viewpoint. Computer program products.

33. 1. A computer system in communication with one or more input devices, a display generating component, and a biometric sensor, comprising: biometrically authenticating a user of the device at a first time using the biometric sensor to perform a first type of biometric authentication; receiving a request to perform a secure operation at a second time after the first time; in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first and second times based on sensor measurements taken at a plurality of intermediate times between the first and second times, including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, the respective criteria detecting that the same user used the device between the first and second times; and withdrawing from performing the secure operation in response to a determination that a respective criterion was not satisfied by the user of the device between the first time and the second time. method.

34. in response to receiving the request to perform the secure operation, biometrically authenticating the user of the device using the biometric sensor to perform the first type of biometric authentication in accordance with a determination that a respective criterion was not satisfied by the user of the device between the first time and the second time.

34. The method of claim 33.

35. 35. The method of claim 33 or 34, wherein the first type of biometric authentication is based on an eye biometric feature.

36. 36. The method of any one of claims 33 to 35, wherein the first type of biometric authentication is based on facial biometric features and the second type of biometric authentication is based on facial biometric features.

37. Fulfilling each criterion by the user of the device includes repeatedly performing a second type of biometric authentication using one or more sensors of the computer system, the second type of biometric authentication comprising: receiving an input from the user including a first individual biometric information; and comparing the received input to a previously received input including a second individual biometric information.

37. The method of any one of claims 33 to 36.

38. 38. The method of any one of claims 33 to 37, wherein the one or more sensors used to perform the second type of biometric authentication includes the biometric sensor used to perform the first type of biometric authentication.

39. 39. The method of any one of claims 33 to 38, wherein the first type of biometric authentication is based on a first biometric characteristic and the second type of biometric authentication is based on a second type of biometric characteristic, the first biometric characteristic being different from the second biometric characteristic.

40. 39. The method of any one of claims 33 to 38, wherein the first type of biometric authentication is based on a first biometric characteristic and the second type of biometric authentication is based on the first type of biometric characteristic.

41. biometrically authenticating the user of the device at the first time based on a first biometric characteristic of the user; 41. The method of claim 33, comprising obtaining sensor measurements at the plurality of intermediate times between the first time and the second time, the obtained sensor measurements including at least one sensor measurement indicating that the first biometric characteristic of the user was not detected at a respective one of the plurality of intermediate times.

42. biometrically authenticating the user of the device at the first time based on a first biometric characteristic of the user; periodically obtaining sensor measurements of a second biometric characteristic of the user at the plurality of intermediate times between the first time and the second time; the acquired sensor measurements include at least one sensor measurement indicating that the first biometric characteristic of the user was not detected at a discrete intermediate time; the acquired sensor measurements include at least one sensor measurement indicating that the second biometric characteristic of the user was detected at the respective intermediate times; the first biometric characteristic is different from the second biometric characteristic; 42. The method of claim 41.

43. 43. The method of any one of claims 33 to 42, wherein the respective criterion between the first and second times includes a continuity criterion that is satisfied when the biometric feature is repeatedly detected between the first and second times.

44. biometrically authenticating the user of the device using the biometric sensor to perform a first type of biometric authentication; providing the user with an authentication guidance user interface including instructions for performing the first type of biometric authentication, the instructions requiring the user to provide a specific movement of the biometric feature.

44. The method of any one of claims 33 to 43.

45. 45. The method of any one of claims 33 to 44, wherein the secure action includes providing payment information.

46. 46. ​​The method of any one of claims 33 to 45, wherein the secure operation includes providing an access credential to access an application.

47. 47. The method of any one of claims 33 to 46, wherein the secure action includes auto-filling user credentials.

48. 48. The method of any one of claims 33 to 47, wherein receiving the request to perform the secure operation comprises detecting a physical input to activate a hardware button.

49. providing the user with a system user interface including an affordance associated with the secure action, wherein receiving the request to perform the secure action includes detecting activation of the affordance associated with the secure action.

49. The method of any one of claims 33 to 48.

50. receiving the request to perform the secure operation; detecting user activation of an affordance within a virtual user interface object while providing the virtual user interface object to the user, the affordance including information about a secure operation.

50. The method of any one of claims 33 to 49.

51. 51. The method of any one of claims 33 to 50, wherein the respective criteria include an eye open criterion that is met when one or more eyes of the user do not remain continuously closed for more than a threshold period of time, and the eye open criterion is not met when one or more eyes of the user remain continuously closed for more than the threshold period of time.

52. 51. A method according to any one of claims 33 to 50, wherein the respective criteria are based on biometric features that are available for analysis regardless of whether the user's eyes are closed or not.

53. 53. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component, a biosensor, and one or more input devices, the one or more programs including instructions for performing the method of any one of claims 33 to 52.

54. 1. A computer system in communication with a display generating component, a biosensor, and one or more input devices, the computer system comprising: one or more processors; and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs including instructions for performing the method of any one of claims 33 to 52.

55. 1. A computer system in communication with a display generating component, a biosensor, and one or more input devices, the computer system comprising:

53. A computer system comprising means for carrying out the method of any one of claims 33 to 52.

56. 1. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component, a biometric sensor, and one or more input devices, the one or more programs comprising: biometrically authenticating a user of the device at a first time using the biometric sensor to perform a first type of biometric authentication; receiving a request to perform a secure operation at a second time after the first time; in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first and second times based on sensor measurements taken at a plurality of intermediate times between the first and second times, including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, the respective criteria detecting that a same user used the device between the first and second times; instructions for ceasing to perform the secure operation in response to a determination that a respective criterion has not been satisfied by the user of the device between the first time and the second time; A non-transitory computer-readable storage medium.

57. 1. A computer system in communication with a display generating component, a biosensor, and one or more input devices, the computer system comprising: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising: biometrically authenticating a user of the device at a first time using the biometric sensor to perform a first type of biometric authentication; receiving a request to perform a secure operation at a second time after the first time; in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first and second times based on sensor measurements taken at a plurality of intermediate times between the first and second times, including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, the respective criteria detecting that a same user used the device between the first and second times; instructions for ceasing to perform the secure operation in response to a determination that a respective criterion has not been satisfied by the user of the device between the first time and the second time; Computer system.

58. 1. A computer system in communication with a display generating component, a biosensor, and one or more input devices, the computer system comprising: means for biometrically authenticating a user of the device at a first time using the biometric sensor to perform a first type of biometric authentication; means for receiving, at a second time, a request to perform a secure operation after the first time; in response to receiving the request to perform the secure operation, means for performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied, between the first and second times, respective criteria based on sensor measurements taken at a plurality of intermediate times between the first and second times, including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, the respective criteria detecting that the same user used the device between the first and second times; and means for withdrawing from performing the secure operation in response to a determination that a respective criterion has not been satisfied by the user of the device between the first time and the second time. Computer system.

59. 53. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component, a biosensor, and one or more input devices, the one or more programs including instructions for performing the method of any one of claims 33 to 52.

60. 1. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component, a biometric sensor, and one or more input devices, the one or more programs comprising: biometrically authenticating a user of the device at a first time using the biometric sensor to perform a first type of biometric authentication; receiving a request to perform a secure operation at a second time after the first time; in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first and second times based on sensor measurements taken at a plurality of intermediate times between the first and second times, including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, the respective criteria detecting that a same user used the device between the first and second times; instructions for ceasing to perform the secure operation in accordance with a determination that the user of the device has not satisfied a respective criterion between the first time and the second time. Computer program products.

61. A computer system in communication with one or more input devices and a display generating component, comprising: receiving a request via the one or more input devices for display of a user interface associated with performing a secure operation; in response to the request for display of the user interface associated with performing the secure operation, displaying via the display generation component a first user interface including a respective prompt for providing additional input for authorizing the device to perform the secure operation; pursuant to determining that the computer system is operating in a first mode, the separate prompt is a first prompt for providing a physical input to authorize the secure operation; pursuant to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt being a second prompt that displays a second user interface, the second user interface enabling user authorization of the secure operation without using the physical input. method.

62. providing a first distinct affordance within the second prompt; and receiving a user selection of the affordance; displaying the second user interface including a representation of a plurality of functions in accordance with receiving the user selection of the affordance, the plurality of functions including: a first distinct function of the plurality of functions that is invoked in response to a secondary input type that is different from the primary input type; a second separate function among the plurality of functions that is invoked in response to a tertiary input type different from the primary input type and the secondary input type; 62. The method of claim 61.

63. 63. The method of claim 62, wherein the secondary input type simulates an input of the primary input type, and the tertiary input type simulates an input of the primary input type.

64. while said computer system is operating in said second mode; Displaying an affordance associated with the individual prompt; and displaying the second user interface in accordance with receiving a user input associated with the respective prompt, the second user interface including an affordance for permitting the secure action, the affordance for permitting the secure action being invoked in response to a secondary input type different from a primary input type.

64. The method of any one of claims 61 to 63.

65. displaying details associated with the performance of the secure action in the user interface associated with the performing of the secure action; while said computer system is operating in said second mode; Displaying an affordance associated with the individual prompt; and receiving a user input associated with the individual prompt; displaying the second user interface in accordance with a determination that the user is gazing at the user interface associated with performing the secure action while the user input associated with the respective prompt is received; the second user interface includes an affordance for permitting the secure action; the affordance for permitting the secure action is displayed in response to a secondary input type that is different from a primary input type.

65. The method of claims 61 to 64.

66. While the computer system is operating in the first mode, receiving a user activation of the physical input while the first prompt is displayed; in response to receiving a user activation of the physical input while the first prompt is displayed; permitting the secure operation in accordance with a determination that a user authentication criterion is satisfied; and and withdrawing from permitting the secure operation in response to a determination that a user authentication criterion has not been met.

66. The method of claims 61 to 65.

67. 67. The method of claim 66, wherein the authentication criteria include criteria that are met when the user provides a particular movement of a biometric characteristic.

68. receiving an input from the user to authorize the secure operation; in response to receiving the input to permit the secure operation, performing user authentication; permitting the secure operation in accordance with a determination that the user is authenticated; 68. The method of claim 61 to 67, comprising:

69. performing user authentication includes performing a second type of biometric authentication, the second type of biometric authentication comprising: receiving an input from the user including a first individual biometric information; and comparing the received input to a previously received input including a second individual biometric information.

69. The method of claim 68.

70. 70. The method of claim 68 or 69, wherein performing user authentication comprises performing a first type of biometric authentication, the first type of biometric authentication comprising detecting the presence of a first biometric feature type.

71. Performing user authentication is providing a prompt requesting a passcode from the user; receiving a passcode from the user in response to the prompt; upon determining that the received passcode and the stored passcode match, determining that the user is authenticated; and permitting the secure operation in accordance with a determination that the user is authenticated.

71. The method of any one of claims 68 to 70.

72. detecting an input for enabling the secure operation; in response to detecting the input to permit the secure operation, in response to determining that the input is a physical input for permitting the secure operation, permitting the secure operation in accordance with a determination that the user is authenticated based on a second type of biometric authentication; in response to determining that the input is an input associated with the second user interface, the input being associated with the second user interface that enables user authorization of the secure operation without using the physical input; performing a first type of biometric authentication; and authorizing the secure operation in accordance with a determination that the user is authenticated based on the first type of biometric authentication.

72. The method of any one of claims 61 to 71.

73. 73. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and one or more input devices, the one or more programs including instructions for performing the method of any one of claims 61 to 72.

74. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs including instructions for performing the method of any one of claims 61 to 72.

75. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising:

73. A computer system comprising means for carrying out the method of any one of claims 61 to 72.

76. 1. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component and one or more input devices, the one or more programs comprising: receiving a request via the one or more input devices for display of a user interface associated with performing a secure operation; and instructions for displaying, in response to the request for display of the user interface associated with performing the secure operation, via the display generation component, a first user interface including a respective prompt for providing additional input for authorizing the device to perform the secure operation; pursuant to determining that the computer system is operating in a first mode, the separate prompt is a first prompt for providing a physical input to authorize the secure operation; a first input for enabling user authorization of the secure operation without using the physical input; and a second prompt for displaying a second user interface in response to a determination that the computer system is operating in a second mode different from the first mode.

77. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising: receiving a request via the one or more input devices for display of a user interface associated with performing a secure operation; and instructions for displaying, in response to the request for display of the user interface associated with performing the secure operation, via the display generation component, a first user interface including a respective prompt for providing additional input for authorizing the device to perform the secure operation; pursuant to determining that the computer system is operating in a first mode, the separate prompt is a first prompt for providing a physical input to authorize the secure operation; pursuant to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt being a second prompt that displays a second user interface, the second user interface enabling user authorization of the secure operation without using the physical input. Computer system.

78. 1. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: means for receiving, via the one or more input devices, a request for display of a user interface associated with performing a secure operation; means for displaying, in response to the request for display of the user interface associated with performing the secure operation, via the display generation component, a first user interface including a respective prompt for providing additional input for authorizing the device to perform the secure operation; pursuant to determining that the computer system is operating in a first mode, the separate prompt being a first prompt for providing a physical input to authorize the secure operation; in response to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt being a second prompt that displays a second user interface, the second user interface enabling user authorization of the secure operation without using the physical input; Computer system.

79. 73. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs including instructions for performing the method of any one of claims 61 to 72.

80. 1. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising: receiving a request via the one or more input devices for display of a user interface associated with performing a secure operation; and instructions for displaying, in response to the request for display of the user interface associated with performing the secure operation, via the display generation component, a first user interface including a respective prompt for providing additional input for authorizing the device to perform the secure operation; pursuant to determining that the computer system is operating in a first mode, the separate prompt is a first prompt for providing a physical input to authorize the secure operation; pursuant to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt being a second prompt that displays a second user interface, the second user interface enabling user authorization of the secure operation without using the physical input. Computer program products.

Images