DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR ALLOWING SECURE OPERATION - Patent application

Abstract

In some examples, a change in a user's current viewpoint is detected while the three-dimensional environment is visible. In some examples, user permission for a secure operation using the virtual user interface object is enabled in accordance with a determination that at least a threshold amount of the object is visible from the user's viewpoint. In some examples, at a first time, a user is biometrically authenticated to perform a first type of biometric authentication. In some examples, at a second time, a request to perform a secure operation is received. In some examples, after receiving the request, the secure operation is performed without performing the first type of biometric authentication in accordance with a determination that the user has met the respective criteria between the first and second times.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application Ser. No. 18 / 167,767, filed February 10, 2023, entitled "DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR AUTHORIZING A SECURE OPERATION," and U.S. Provisional Application No. 63 / 314,900, filed February 28, 2022, entitled "DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR AUTHORIZING A SECURE OPERATION," the entire contents of each of which are incorporated herein by reference in their entirety for all purposes. [Technical Field]

[0002] The present disclosure generally relates to computer systems in communication with display generation components, one or more input devices, and (optionally) biometric sensors, that provide computer-generated experiences, including, but not limited to, electronic devices that provide virtual reality and mixed reality experiences via a display. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary extended reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays for computer systems and other electronic computing devices are used to interact with the virtual / extended reality environment. Exemplary virtual elements include virtual objects such as digital images, video, text, icons, and control elements such as buttons and other graphics. Summary of the Invention

[0004] Methods and interfaces that allow secure operation in environments that include some virtual elements (e.g., augmented reality, mixed reality, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide solutions for automatically filling out various text fields, facilitating payment for items or other services, and providing access to third-party applications (e.g., social networking, gaming, business services, etc.) are complex, cumbersome, and error-prone, creating significant cognitive load for users and detracting from their experience in the virtual / augmented reality environment. In addition, these methods take longer than necessary, thereby wasting computer system energy. This latter consideration is particularly important in battery-operated devices.

[0005] Therefore, there is a need for a computer system having improved methods and interfaces for authorizing secure operations to make interactions with the computer system more efficient and intuitive for users. Such methods and interfaces optionally complement or replace conventional methods for authorizing secure operations. Such methods and interfaces reduce the number, extent, and / or type of inputs from a user by helping the user understand the connection between a provided input and a device response to that input, thereby creating a more efficient human-machine interface.

[0006] The above-mentioned drawbacks and other problems associated with user interfaces of computer systems are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a wristwatch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touchscreen display"). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generating components, the output devices including one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing a plurality of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on the touch-sensitive surface, the movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through interaction optionally include image editing, drawing, presentation, word processing, spreadsheet creation, gameplay, making phone calls, video conferencing, sending emails, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note-taking, and / or digital video playback. The executable instructions for performing those functions optionally include primary computer-readable storage media and / or non-primary computer-readable storage media, or other computer program products configured to be executed by one or more processors.

[0007] Electronic devices with improved methods and interfaces for enabling secure operation are needed. Such methods and interfaces may complement or replace conventional methods for enabling secure operation. Such methods and interfaces reduce the number, extent, and / or type of user input, resulting in a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces conserve power and extend the interval between battery charges. Furthermore, these methods and interfaces help reduce the number of repetitions of user input by automatically filling in or otherwise inputting sensitive information such as passwords or payment information. Such methods and interfaces also reduce processing power and display usage by reducing the amount of time the user spends interacting with the device when enabling secure operation.

[0008] According to some embodiments, a method is described that is performed in a computer system in communication with one or more input devices and a display generation component. The method includes detecting a change in a user's current viewpoint via the one or more input devices while a three-dimensional environment is visible via the display generation component, the three-dimensional environment including virtual user interface objects that include information related to a secure operation. In response to detecting the change in the user's viewpoint, enabling user permission for the secure operation using the virtual user interface objects in accordance with a determination that at least a threshold amount of the virtual user interface objects are visible from the user's viewpoint and that the user is authorized to perform the secure operation, and withdrawing enabling user permission for the secure operation using the virtual user interface objects in accordance with a determination that fewer than the threshold amount of virtual user interface objects are visible from the user's viewpoint.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a computer system in communication with one or more input devices and a display generation component, the one or more programs including instructions for detecting a change in a user's current viewpoint via the one or more input devices while a three-dimensional environment including virtual user interface objects that include information related to a secure operation is visible via the display generation component, and in response to detecting the change in the user's viewpoint, enabling user permission for the secure operation using the virtual user interface objects in accordance with a determination that at least a threshold amount of the virtual user interface objects are visible from the user's viewpoint and that the user is authorized to perform the secure operation, and withdrawing enabling user permission for the secure operation using the virtual user interface objects in accordance with a determination that fewer than the threshold amount of virtual user interface objects are visible from the user's viewpoint.

[0010] According to some embodiments, a computer system is described that is in communication with one or more input devices and a display generation component. The computer system includes one or more processors and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for detecting a change in a user's current viewpoint via the one or more input devices while a three-dimensional environment is visible via the display generation component, the three-dimensional environment including virtual user interface objects that include information related to a secure operation, and in response to detecting the change in the user's viewpoint, enabling user permission for the secure operation using the virtual user interface objects in accordance with a determination that at least a threshold amount of the virtual user interface objects are visible from the user's viewpoint and that the user is authorized to perform the secure operation, and withdrawing enabling user permission for the secure operation using the virtual user interface objects in accordance with a determination that fewer than the threshold amount of virtual user interface objects are visible from the user's viewpoint.

[0011] According to several embodiments, a computer system is described. This computer system communicates with one or more input devices and display generation components. This computer system is a three-dimensional environment, and while the three-dimensional environment, which includes virtual user interface objects containing information about secure operations, is visible via the display generation components, it detects a change in the user's current viewpoint via one or more input devices, and in response to detecting a change in the user's viewpoint, it includes means for enabling user permission to perform secure operations using virtual user interface objects according to a determination that at least a threshold amount of virtual user interface objects is visible from the user's viewpoint and the user is permitted to perform secure operations, and means for deactivating user permission to perform secure operations using virtual user interface objects according to a determination that the amount of virtual user interface objects visible from the user's viewpoint is less than a threshold amount.

[0012] A method is described according to several embodiments. The method is performed in a computer system communicating with one or more input devices, display generating components, and biosensors. The method includes: biometric authentication of a device user using a biosensor to perform a first type of biometric authentication at a first time; receiving a request to perform a secure operation at a second time following the first time; and, in response to receiving the request to perform a secure operation, performing a secure operation without performing a first type of biometric authentication after receiving the request to perform a secure operation, based on a determination that the device user met the criteria between the first and second time, which are based on a plurality of sensor measurements obtained at a plurality of intermediate time points between the first and second time points, including sensor measurements obtained at a first intermediate time point and sensor measurements obtained at a second intermediate time point, and which determine that the same user was using the device between the first and second time points; and ceasing to perform the secure operation based on a determination that the continuity criteria were not met between the first and second time points.

[0013] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with one or more input devices, a display generating component, and a biometric sensor, the one or more programs including instructions for: biometrically authenticating a user of the device using the biometric sensor to perform a first type of biometric authentication at a first time; receiving a request to perform a secure operation at a second time after the first time; in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first time and the second time based on sensor measurements obtained at multiple intermediate times between the first time and the second time, the sensor measurements including a sensor measurement obtained at a first intermediate time and a sensor measurement obtained at a second intermediate time, the sensor measurements detecting that the same user used the device between the first time and the second time; and aborting performance of the secure operation in accordance with a determination that a continuity criterion was not satisfied between the first time and the second time.

[0014] According to some embodiments, a computer system is described, the computer system being in communication with one or more input devices, a display generating component, and a biometric sensor. The computer system includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: biometrically authenticating a user of a device using a biometric sensor to perform a first type of biometric authentication at a first time; receiving a request to perform a secure operation at a second time after the first time; and, in response to receiving the request to perform the secure operation, performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in accordance with a determination that a user of the device satisfied respective criteria between the first and second times based on sensor measurements taken at multiple intermediate times between the first and second times, the sensor measurements including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, the sensor measurements detecting that the same user used the device between the first and second times; and aborting performance of the secure operation in accordance with a determination that a continuity criterion was not satisfied between the first and second times.

[0015] According to some embodiments, a computer system is described, in communication with one or more input devices and a display generation component. The computer system includes: means for biometrically authenticating a user of the device at a first time using a biometric sensor to perform a first type of biometric authentication; means for receiving a request to perform a secure operation at a second time after the first time; means for performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation in response to receiving the request to perform the secure operation, in accordance with a determination that a user of the device satisfied respective criteria between the first and second times based on sensor measurements taken at multiple intermediate times between the first and second times, the sensor measurements including a sensor measurement taken at a first intermediate time and a sensor measurement taken at a second intermediate time, the sensor measurements detecting that the same user used the device between the first and second times; and means for canceling performance of the secure operation in accordance with a determination that a continuity criterion was not satisfied between the first and second times.

[0016] A method is described according to several embodiments. The method is performed in a computer system communicating with one or more input devices and a display generating component. The method includes receiving a request via one or more input devices for the display of a user interface relating to performing a secure operation, and, in response to the request for the display of a user interface relating to performing a secure operation, displaying a first user interface via a display generating component that includes a separate prompt for providing additional input to allow the device to perform a secure operation, wherein, according to a determination that the computer system is operating in a first mode, the separate prompt is a first prompt that provides physical input to allow a secure operation, and according to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt that displays a second user interface which enables user authorization of a secure operation without using physical input.

[0017] According to some embodiments, a non-transitory computer-readable storage medium is described that stores one or more programs configured to be executed by one or more processors of a computer system in communication with one or more input devices and a display generating component, the one or more programs including instructions for receiving, via the one or more input devices, a request for displaying a user interface associated with performing a secure operation, and, in response to the request for displaying the user interface associated with performing the secure operation, displaying, via the display generating component, a first user interface including a separate prompt for providing additional input for authorizing the device to perform the secure operation, wherein, in accordance with a determination that the computer system is operating in a first mode, the separate prompt is a first prompt for providing physical input for authorizing the secure operation, and in accordance with a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt for displaying a second user interface, the second user interface enabling user authorization of the secure operation without using physical input.

[0018] According to some embodiments, a computer system is described that is in communication with one or more input devices and a display generation component. The computer system includes one or more processors and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for receiving, via the one or more input devices, a request for displaying a user interface associated with performing a secure operation, and, in response to the request for displaying the user interface associated with performing the secure operation, displaying, via the display generation component, a first user interface that includes a separate prompt for providing additional input for authorizing the device to perform the secure operation, wherein, in response to a determination that the computer system is operating in a first mode, the separate prompt is a first prompt that provides physical input for authorizing the secure operation, and in response to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt that displays a second user interface that enables user authorization of the secure operation without using physical input.

[0019] According to some embodiments, a computer system is described, the computer system being in communication with one or more input devices and a display generation component. The computer system includes: means for receiving, via the one or more input devices, a request for displaying a user interface associated with performing a secure operation; and means for, in response to the request for displaying the user interface associated with performing the secure operation, displaying, via the display generation component, a first user interface including a separate prompt for providing additional input for authorizing the device to perform the secure operation, wherein, in accordance with a determination that the computer system is operating in a first mode, the separate prompt is a first prompt for providing physical input for authorizing the secure operation, and in accordance with a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt for displaying a second user interface, the second user interface enabling user authorization of the secure operation without using physical input.

[0020] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and explanation, and not to define or limit the subject matter of the present invention. [Brief explanation of the drawings]

[0021] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0022] [Figure 1]FIG. 1 is a block diagram illustrating an operating environment for a computer system for providing an XR experience, according to some embodiments.

[0023] [Figure 2] FIG. 1 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience, according to some embodiments.

[0024] [Figure 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a user with visual components of an XR experience, according to some embodiments.

[0025] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0026] [Figure 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0027] [Figure 6A] FIG. 1 is a flow diagram illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0028] [Figure 6B] FIG. 1 illustrates exemplary devices connected via one or more communication channels, according to some embodiments.

[0029] [Figure 7A] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7B]FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7C] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7D] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7E] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7F] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7G] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7H] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7I] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7J] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7K] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7L] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7M] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments. [Figure 7N] FIG. 1 illustrates an example technique for permitting secure operation, according to some embodiments.

[0030] [Figure 8] FIG. 1 is a flow diagram of a method for facilitating user consent for secure operation, according to various embodiments.

[0031] [Figure 9]FIG. 1 is a flow diagram of a method of authentication continuity for secure operation, according to various embodiments.

[0032] [Figure 10] FIG. 1 is a flow diagram of a method for allowing secure operation via an accessibility interface, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0033] The present disclosure relates to a user interface that provides an extended reality (XR) experience to a user, according to some embodiments.

[0034] FIGS. 1-6B provide an illustration of an example computer system for providing an XR experience to a user. FIGS. 7A-7N illustrate example techniques for authorizing secure operation, according to some embodiments. FIG. 8 is a flow diagram of a method for facilitating informed consent for secure operation, according to various embodiments. The process of FIG. 8 is described using the user interfaces of FIGS. 7A-7N. FIG. 9 is a flow diagram of a method of authentication continuity for secure operation, according to various embodiments. The process of FIG. 9 is described using the user interfaces of FIGS. 7A-7N. FIG. 10 is a flow diagram of a method for authorizing secure operation via an accessibility interface, according to various embodiments. The process of FIG. 10 is described using the user interfaces of FIGS. 7A-7N.

[0035] The processes described below enhance device operability and streamline the user-device interface (e.g., by helping users provide appropriate inputs when operating / interacting with the device and reducing user errors) through various techniques, including providing improved visual feedback to the user, reducing the number of inputs required to perform an action, adding control options without cluttering the user interface with additional displayed controls, performing an action without requiring further user input when a set of conditions is met, improving privacy and / or security, and / or other techniques. These techniques also reduce power usage and improve the device's battery life by allowing users to use the device more quickly and efficiently.

[0036] Furthermore, for methods described herein in which one or more steps are conditioned on one or more conditions being satisfied, it should be understood that the described method can be repeated in multiple iterations, such that over the course of the iterations, all of the conditions on which the method steps are conditioned are satisfied in different iterations of the method. For example, if a method requires performing a first step if a condition is satisfied and a second step if the condition is not satisfied, one skilled in the art will understand that the steps recited in the claim are repeated in a particular order until the conditions are satisfied and then no longer satisfied. Thus, a method described with one or more steps that depend on one or more conditions being satisfied can be rewritten as a method that is repeated until each condition recited in the method is satisfied. However, this is not required for system or computer-readable medium claims in which the system or computer-readable medium includes instructions that perform a conditional action based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency is met without explicitly repeating the method steps until all conditions on which the method steps are conditioned are satisfied. Those skilled in the art will also understand that, as with methods having conditional steps, the system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0037] In some embodiments, as shown in Figure 1, the XR experience is provided to the user via an operating environment 100 which includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or remote server), display generation components 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a velocity sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a home appliance, a wearable device, etc.). In some embodiments, one or more of input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with display generation component 120 (e.g., within a head-mounted or handheld device).

[0038] When describing an XR experience, various terms are used to specifically refer to several related but distinct environments that the user can perceive and / or interact with (for example, using inputs detected by the computer system 101, which causes the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to the various inputs provided to the computer system 101 that generates the XR experience). The following is a subset of these terms.

[0039] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through their senses, such as sight, touch, hearing, taste, and smell.

[0040] Extended reality: In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and / or interact with through electronic systems. In XR, a subset of a person's body motions or representations thereof are tracked, and one or more properties of one or more virtual objects simulated within the XR environment are adjusted accordingly to behave according to at least one law of physics. For example, an XR system may detect a person's head rotation and adjust the graphical content and sound field presented to the person accordingly, in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustment of a property(ies) of a virtual object(ies) in an XR environment may occur in response to a representation of body motion (e.g., a voice command). A person may sense and / or interact with an XR object using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person may sense and / or interact with only audio objects.

[0041] Examples of XR include virtual reality and mixed reality.

[0042] Virtual reality: A virtual reality (VR) environment refers to a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can perceive and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can perceive and / or interact with virtual objects in a VR environment through a simulation of their presence within the computer-generated environment and / or through a simulation of a subset of their physical movement within the computer-generated environment.

[0043] Mixed Reality: A mixed reality (MR) environment is a simulated environment designed to incorporate sensory input from or its representation from a physical environment, in addition to including computer-generated sensory input (e.g., virtual objects), in contrast to a virtual reality (VR) environment designed to rely entirely on computer-generated sensory input. On a virtual continuum, a mixed reality environment is any location between, but not including, the complete physical environment at one end and the virtual reality environment at the other end. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Also, some electronic systems for presenting an MR environment may track location and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical articles or their representations from the physical environment). For example, the system may take motion into account so that a virtual tree appears stationary relative to the physical ground.

[0044] Examples of mixed reality include extended reality and augmented virtuality.

[0045] Extended Reality: An extended reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, whereby a person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment that are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. The person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, video of a physical environment shown on an opaque display is referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may include a projection system that projects virtual objects, e.g., as holograms, into the physical environment or onto a physical surface, such that a person using the system perceives the virtual objects superimposed on the physical environment. An extended reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) other than the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be distorted by graphically modifying (e.g., enlarging) a portion thereof, such that the modified portion becomes a non-photorealistic, altered version that represents the originally captured image.As a further example, the representation of the physical environment may be altered by graphically removing or obscuring portions of it.

[0046] Augmented Virtuality: An augmented virtuality (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0047] Perspective-Locked Virtual Object: A virtual object is perspective-locked when the computer system displays the virtual object in the same location and / or position within the user's perspective, even as the user's perspective shifts (e.g., changes). In embodiments in which the computer system is a head-mounted device, the user's perspective is locked to the forward-facing orientation of the user's head (e.g., the user's perspective is at least a portion of the user's field of view when the user is looking straight ahead). Thus, the user's perspective remains fixed even as the user's line of sight moves without moving the user's head. In embodiments in which the computer system has a display generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's perspective is an extended reality view being presented to the user on the display generating component of the computer system. For example, a perspective-locked virtual object displayed in the upper left corner of the user's perspective when the user's perspective is in a first orientation (e.g., the user's head is facing north) will continue to be displayed in the upper left corner of the user's perspective even if the user's perspective changes to a second orientation (e.g., the user's head is facing west). In other words, the location and / or position at which a viewpoint-locked virtual object is displayed in a user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments in which the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object."

[0048] Environment-Locked Virtual Object: A virtual object is environment-locked (or "world-locked") when a computer system displays the virtual object at a location and / or position within a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) locations and / or objects within a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint shifts, the locations and / or objects within the environment relative to the user's viewpoint change, resulting in the environment-locked virtual object appearing at a different location and / or position within the user's viewpoint. For example, an environment-locked virtual object locked to a tree directly in front of the user will appear centered within the user's viewpoint. If the user's viewpoint shifts to the right (e.g., the user's head is turned to the right) and the tree becomes more left-leaning within the user's viewpoint (e.g., the position of the tree within the user's viewpoint shifts), the environment-locked virtual object locked to the tree will appear more left-leaning within the user's viewpoint. In other words, the location and / or position at which the environment-locked virtual object appears within the user's viewpoint depends on the position and / or orientation of the location and / or object in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a coordinate system fixed to a fixed location and / or object in the physical environment) to determine a position at which to display an environment-locked virtual object in the user's viewpoint. The environment-locked virtual object can be locked to a stationary portion of the environment (e.g., a floor, wall, table, or other stationary object) or can be locked to a moving portion of the environment (e.g., a vehicle, an animal, a person, or a representation of a part of the user's body that moves independent of the user's viewpoint, such as the user's hand, wrist, arm, or leg), so that the virtual object moves as the viewpoint or part of the environment moves in order to maintain a fixed relationship between the virtual object and the part of the environment.

[0049] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits delayed-following behavior, which reduces or delays the motion of the environment-locked or viewpoint-locked virtual object relative to movement of a reference point that the virtual object is following. In some embodiments, when exhibiting delayed-following behavior, the computer system intentionally delays movement of the virtual object when it detects movement of a reference point that the virtual object is following (e.g., a part of the environment, the viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 and 300 cm from the viewpoint). For example, when the reference point (e.g., a part of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but at a second speed that is slower than the first speed (e.g., until the reference point stops or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits delayed-following behavior, the device ignores small amounts of movement of the reference point (e.g., ignores movement of the reference point that is less than a threshold amount, such as movement between 0 and 5 degrees or movement between 0 and 50 cm). For example, when the reference point (e.g., a portion of the environment or a viewpoint to which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment different from the reference point to which the virtual object is locked), and when the reference point (e.g., a portion of the environment or a viewpoint to which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases beyond a threshold (e.g., a “delayed following” threshold) as the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point.In some embodiments, a virtual object maintaining a substantially fixed position relative to a reference point includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the reference point in one or more dimensions (e.g., above / below, left / right, and / or forward / backward relative to the position of the reference point).

[0050] Hardware: There are many different types of electronic systems that enable a person to perceive and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (e.g., contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing sounds of the physical environment. A head-mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface. In some embodiments, the controller 110 is configured to manage and adjust the XR experience for the user.In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. Controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, controller 110 is a computing device that is local or remote to scene 105 (e.g., the physical environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., an HMD, a display, a projector, a touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of one or more of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195, or shares the same physical housing or support structure as one or more of the foregoing.

[0051] In some embodiments, display generation component 120 is configured to provide an XR experience (e.g., at least a visual component of an XR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0052] According to some embodiments, the display generation component 120 provides an XR experience to the user while the user is virtually and / or physically present in the scene 105.

[0053] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on their head, their hand, etc.). Thus, display generating component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, display generating component 120 surrounds the user's field of view. In some embodiments, display generating component 120 is a handheld device (e.g., a smartphone or tablet) configured to present XR content, where the user holds the device with a display pointed toward the user's field of view and a camera pointed toward scene 105. In some embodiments, the handheld device is optionally located within a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, display generating component 120 is an XR chamber, housing, or room configured to present XR content without the user wearing or holding display generating component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a tripod-mounted device) can also be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD in which the interactions occur in the space in front of the HMD and the XR content responses are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD in which the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0054] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will understand from this disclosure that for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0055] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), central processing unit (CPU), processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global Mobile Communication System (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZiGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0056] In some embodiments, one or more communication buses 204 include circuits for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices 206 include at least one of the following: a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0057] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and an XR experience module 240:

[0058] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for respective groups of one or more users). To that end, in various embodiments, XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0059] 1 , and optionally one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data acquisition unit 241 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0060] In some embodiments, tracking unit 242 is configured to map scene 105 and track the position / location of at least display generation component 120 relative to scene 105 of FIG. 1 and, optionally, relative to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, tracking unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, tracking unit 242 includes hand tracking unit 244 and / or eye tracking unit 243. In some embodiments, hand tracking unit 244 is configured to track the position / location of one or more parts of a user's hand and / or the motion of one or more parts of a user's hand relative to scene 105 of FIG. 1, relative to display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. Hand tracking unit 244 is described in more detail below with respect to FIG. 4. In some embodiments, eye tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)), or relative to XR content displayed via display generation component 120. Eye tracking unit 243 is described in more detail below with respect to FIG. 5.

[0061] In some embodiments, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120 and, optionally, by one or more of output devices 155 and / or peripheral devices 195. To that end, in various embodiments, coordination unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0062] In some embodiments, data dissemination unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data dissemination unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0063] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 can be located within separate computing devices.

[0064] Furthermore, Figure 2 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, in various embodiments, some functional modules shown separately in Figure 2 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary depending on implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0065] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the embodiments disclosed herein. To that end, by way of non-limiting example, in some embodiments, the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0066] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0067] In some embodiments, the one or more XR displays 312 are configured to provide an XR experience to a user. In some embodiments, the one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emissive element display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, the one or more XR displays 312 correspond to waveguide displays, such as diffractive, reflective, polarized, holographic, etc. For example, the display generation component 120 (e.g., an HMD) includes a single XR display. In another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, the one or more XR displays 312 are capable of presenting mixed reality (MR) or virtual reality (VR) content. In some embodiments, the one or more XR displays 312 are capable of presenting mixed reality (MR) or virtual reality (VR) content.

[0068] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hands and optionally, at least a portion of the user's arms (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene that the user would view if a display generation component 120 (e.g., an HMD) were not present (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., complementary metal-oxide-semiconductor (CMOS) image sensors or charge-coupled device (CCD) image sensors), one or more infrared (IR) cameras, one or more event-based cameras, and / or similar.

[0069] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-temporary computer-readable storage medium. In some embodiments, memory 320, or the non-temporary computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 330 and XR presentation module 340.

[0070] The operating system 330 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to a user via one or more XR displays 312. To that end, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0071] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0072] In some embodiments, the XR presentation unit 344 is configured to present XR content via one or more XR displays 312. To that end, in various embodiments, the XR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0073] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate an extended reality) based on the media content data. To that end, in various embodiments, the XR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0074] In some embodiments, data dissemination unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data dissemination unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0075] Although the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 may be located in separate computing devices.

[0076] Furthermore, Figure 3 is intended more to illustrate the functionality of various features that may be present in particular implementations, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in Figure 3 can be implemented within a single module, and various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0077] Figure 4 is a schematic diagram of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 (Figure 1) is controlled by the hand tracking unit 244 (Figure 2) to track the location / position of one or more parts of the user's hand and / or the motion of one or more parts of the user's hand relative to the scene 105 of Figure 1 (e.g., relative to a part of the physical environment surrounding the user, relative to the display generation component 120, or relative to a part of the user (e.g., the user's face, eyes, or head), and / or the user's hand, in a coordinate system defined therein). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0078] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for differentiation of the fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or all of the body, and can have either zoom capabilities or a dedicated sensor with high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0079] In some embodiments, image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to controller 110, which extracts high-level information from the map data. This high-level information is typically provided via an application program interface (API) to an application running on the controller, which drives display generation component 120 accordingly. For example, a user can interact with software running on controller 110 by moving their hand 406 and changing the posture of their hand.

[0080] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spots of the pattern. This approach is advantageous in that it does not require the user to hold or wear any type of beacon, sensor, or other marker. This provides depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, the image sensor 404 is assumed to define a set of orthogonal x, y, and z axes such that the depth coordinate of a point in the scene corresponds to the z-component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurement, based on single or multiple cameras or other types of sensors.

[0081] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors with patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D locations of the user's wrist joints and fingertips.

[0082] The software can also analyze the trajectory of the hand and / or fingers across multiple frames in a sequence to identify gestures. The pose estimation function described herein may be interleaved with the motion tracking function so that patch-based pose estimation is performed only once every two frames (or more), and tracking is used to detect changes in pose that occur over the remaining frames. Pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify an image presented on the display generation component 120, or perform other functions, depending on the pose and / or gesture information.

[0083] In some embodiments, the gesture includes an air gesture. An air gesture is a gesture detected by the user without (or independently of) touching an input element that is part of a device (e.g., a computer system 101, one or more input devices 125, and / or a hand tracking device 140), and is based on the detected motion of a part of the user's body in the air (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs), including the motion of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), the motion of the user's body relative to another part of the user's body (e.g., the movement of the user's hand relative to the user's shoulder, the movement of one of the user's hands relative to the user's other hand, and / or the movement of the user's fingers relative to another finger or part of the user's hand), and / or the absolute motion of a part of the user's body (e.g., a tap gesture including the movement of the hand in a predetermined position by a predetermined amount and / or speed, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body).

[0084] In some embodiments, input gestures used in various examples and embodiments described herein include air gestures performed by movement of a user's finger(s) relative to other finger(s) or part(s) of the user's hand to interact with an XR environment (e.g., a virtual or mixed reality environment), according to some embodiments. In some embodiments, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and is based on detected motion of a part of the user's body, including motion of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), motion of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one of the user's hands, and / or movement of the user's finger relative to another finger or part of the user's hand), and / or absolute motion of a part of the user's body (e.g., a tap gesture that includes movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture that includes rotation of a part of the user's body a predetermined speed or amount).

[0085] In some embodiments where the input gesture is an air gesture (e.g., in the absence of physical contact with an input device that provides a computer system with information about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touchscreen or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., in the case of direct input, as described below). Thus, in implementations that include air gestures, the input gesture is detected attention (e.g., gaze) to a user interface element in combination with (e.g., simultaneous with) movement of the user's finger(s) and / or hand to perform pinch and / or tap input, as described in more detail below.

[0086] In some embodiments, an input gesture directed at a user interface object is performed directly or indirectly with respect to the user interface object. For example, user input is performed directly at a user interface object in concert with the user's hand performing the input gesture at a position corresponding to the user interface object's position in the three-dimensional environment (e.g., determined based on the user's current viewpoint). In some embodiments, the input gesture is performed indirectly at a user interface object in concert with the user performing the input gesture while detecting the user's attention (e.g., gaze) toward the user interface object while the user's hand position is not at a position corresponding to the user interface object's position in the three-dimensional environment. For example, in the case of a direct input gesture, the user can direct the user's input to a user interface object by initiating the gesture at or near a position corresponding to the user interface object's displayed position (e.g., within a distance of 0.5 cm, 1 cm, 5 cm, or 0-5 cm, measured from an outer edge of the option or a central portion of the option). For indirect input gestures, a user can direct their input to a user interface object by paying attention to the user interface object (e.g., by gazing at the user interface object), and while paying attention to the option, the user initiates an input gesture (e.g., at any position detectable by the computer system) (e.g., at a position that does not correspond to the displayed position of the user interface object).

[0087] In some embodiments, input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch inputs and tap inputs for interacting with a virtual or mixed reality environment, according to some embodiments. For example, pinch inputs and tap inputs, as described below, are performed as air gestures.

[0088] In some embodiments, the pinch input is part of an air gesture, including one or more of a pinch gesture, a long pinch gesture, a pinch-and-drag gesture, or a double pinch gesture. For example, a pinch gesture that is an air gesture includes moving two or more fingers of a hand to contact each other, i.e., optionally with a short break (e.g., within 0-1 second) after contact with each other. A long pinch gesture that is an air gesture includes moving two or more fingers of a hand to contact each other for at least a threshold amount of time (e.g., at least 1 second) before detecting a break in contact with each other. For example, a long pinch gesture includes a user holding a pinch gesture (e.g., when two or more fingers are in contact), and the long pinch gesture continues until a break in contact between the two or more fingers is detected. In some embodiments, a double pinch gesture that is an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected immediately in succession (e.g., within a predetermined period of time) after each other. For example, a user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaking contact between two or more fingers), and performs a second pinch input within a predetermined period of time (e.g., within 1 second or 2 seconds) after releasing the first pinch input.

[0089] In some embodiments, an air gesture, a pinch-and-drag gesture, includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in relation to (e.g., after) a drag input that changes the user's hand position from a first position (e.g., a drag initiation position) to a second position (e.g., a resistance termination position). In some embodiments, the user maintains the pinch gesture while performing the drag input and releases the pinch gesture (e.g., spreading two or more fingers) to terminate the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together and touches them to each other, and then moves the same hand to a second position in the air with a drag gesture). In some embodiments, the pinch input is performed by the user's first hand and the drag input is performed by the user's second hand (e.g., the user's second hand moves from the first position to the second position in the air while the user continues the pinch input with the user's first hand). In some embodiments, an input gesture that is an air gesture includes an input (e.g., a pinch input and / or a tap input) performed using both of the user's hands. For example, an input gesture includes two (e.g., or more) pinch inputs performed in relation to each other (e.g., simultaneously or within a predetermined period of time). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) performed using the user's first hand, and a second pinch input performed using the other hand (e.g., a second hand of the user's hands) in relation to performing a pinch input using the first hand. In some embodiments, movement between the user's hands (e.g., to increase and / or decrease the distance or relative orientation between the user's hands).

[0090] In some embodiments, a tap input (e.g., directed toward a user interface element) performed as an air gesture includes movement of a user's finger(s) toward the user interface element, movement of a user's hand toward a user interface element, optionally with the user's finger(s) extended toward the user interface element, a downward motion of a user's finger (e.g., mimicking a mouse click motion or a tap on a touchscreen), or other predefined movement of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture, moving the finger or hand away from the user's viewpoint and / or toward the object that is the target of the tap input followed by an end of the movement. In some embodiments, an end of the movement is detected based on a change in movement characteristics of the finger or hand performing the tap gesture (e.g., an end of movement away from the user's viewpoint and / or toward the object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the direction of acceleration of the movement of the finger or hand).

[0091] In some embodiments, the user's attention is determined to be directed to a portion of the three-dimensional environment based on detecting a gaze directed to the portion of the three-dimensional environment (optionally, without requiring other conditions). In some embodiments, the device determines that the user's attention is directed to the portion of the three-dimensional environment based on detecting a gaze directed to the portion of the three-dimensional environment with one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., dwell time) while the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment, and / or requiring the gaze to be directed to the portion of the three-dimensional environment, and if one of the additional conditions is not met, the device determines that the user's attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until one or more additional conditions are met).

[0092] In some embodiments, the detection of a ready state configuration of the user or a part of the user is detected by the computer system. The detection of a ready state configuration of the hand is used by the computer system as an indicator that the user is likely to be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of a hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape where the thumb and one or more fingers are extended and spaced apart, ready to perform a pinch or grab gesture, or a pre-tap shape where one or more fingers are extended and the palm is facing away from the user), whether the hand is in a predetermined position relative to the user's line of sight (e.g., below the user's head, above the user's waist, or extended at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular way (e.g., moved towards the area in front of the user above the user's waist, below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interaction element of the user interface is responsive to attentional (e.g., gaze) input.

[0093] In scenarios where input is described with reference to air gestures, it should be understood that similar gestures can also be detected using a hardware input device attached to or held by one or more of the user's hands, where the position of the hardware input device in space may be tracked using optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers, and / or one or more inertial measurement units, and where the position and / or movement of the hardware input device is substituted for the position and / or movement of the one or more hands in the corresponding air gesture(s). In scenarios where input is described with reference to air gestures, it should be understood that similar gestures can also be detected using a hardware input device attached to or held by one or more of the user's hands, in which case user input can be detected using controls included in the hardware input device, such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, one or more hand or finger covers, and / or other hardware input device controls that can detect the position or change in position of parts of the hands and / or fingers relative to each other, relative to the user's body, and / or relative to the user's physical environment, in which case user input using controls included in the hardware input device is used instead of hand and / or finger gestures such as an air tap or air pinch in the corresponding air gesture(s). For example, a selection input described as occurring with an air tap or air pinch input can alternatively be detected with a button press, a tap on a touch-sensitive surface, a press on a pressure-sensitive surface, or other hardware input.As another example, movement input described as being made by an air pinch and drag may alternatively be detected based on interaction with a hardware input control, such as a press and hold of a button, a touch on a touch-sensitive surface, a press on a pressure-sensitive surface, or based on hardware input followed by the movement of another hardware input device in space (e.g., accompanying the hand with which the hardware input device is associated). Similarly, two-handed input, including the movement of both hands relative to one another, may be made using one air gesture and one hardware input device held in the hand not making the air gesture, two hardware input devices held in separate hands, or two air gestures made by separate hands, using various combinations of air gestures and / or input detected by one or more of the hardware input devices described above.

[0094] In some embodiments, the software may be downloaded electronically to the controller 110, for example, over a network, or instead, it may be provided on a tangible non-temporary medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively or additionally, some or all of the computer's described functions may be implemented in dedicated hardware such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in Figure 4, for example, as a separate unit from the image sensor 404, some or all of the controller's processing functions may be associated with the image sensor 404 by a suitable microprocessor and software, or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand-tracking device), or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or by any other suitable computerized device such as a game console or media player. The sensing function of the image sensor 404 may likewise be integrated into a computer or other computerized device that is controlled by the sensor output.

[0095] FIG. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels having respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing gray levels as depth increases. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics may include, for example, the overall size, shape, and frame-to-frame motion of the depth map sequence.

[0096] 4 also schematically illustrates a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to some embodiments. In FIG. 4, the hand skeleton 414 is overlaid on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand where it connects to the wrist, etc.), and optionally the wrist or arm connected to the hand, are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0097] FIG. 5 shows an exemplary embodiment of eye tracking device 130 ( FIG. 1 ). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 243 ( FIG. 2 ) to track the position and movement of a user's gaze relative to scene 105 or relative to XR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed in a wearable frame, the head-mounted device includes both components for generating XR content for viewing by the user and components for tracking the user's gaze relative to the XR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if the display generation component is a handheld device or an XR chamber, eye tracking device 130 is optionally a device separate from the handheld device or the XR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, head-mounted eye tracking device 130 is optionally used in conjunction with head-mounted or non-head-mounted display generating components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally used in combination with head-mounted display generating components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally part of non-head-mounted display generating components.

[0098] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display that allows the user to view the physical environment directly and display virtual objects on the transparent or translucent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, allowing an individual using the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0099] As shown in FIG. 5 , in some embodiments, eye tracking device 130 (e.g., gaze tracking device) includes at least one eye tracking camera (e.g., an infrared (IR) camera or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera may be aimed at the user's eyes to receive reflected IR or NIR light from the light source directly from the eyes, or alternatively, may be aimed at a “hot” mirror positioned between the user's eyes and a display panel that reflects the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. Eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to controller 110. In some embodiments, the user's eyes are tracked separately by their respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a separate eye-tracking camera and lighting source.

[0100] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the eye tracking device's parameters for the particular operating environment 100, such as the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility before delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic or manual calibration process. The user-specific calibration process may include estimation of a particular user's eye parameters, such as pupil location, central visual location, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters for the eye tracking device 130 have been determined, images captured by the eye tracking camera can be processed using a glint-assisted method to determine the user's current visual axis and point of gaze relative to the display.

[0101] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and a gaze tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, a projector, etc.) and may be directed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown at the top of FIG. 5), or may be directed at the user's eye(s) 592 to receive reflected IR or NIR light from the eye(s) 592 (e.g., as shown at the bottom of FIG. 5).

[0102] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as in processing the frames 562 for display. The controller 110 optionally estimates the user's point of gaze on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a glint-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the user's current looking direction.

[0103] Some possible use cases of the user's current gaze direction are described below, but are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller may position or move virtual content within a view based at least in part on the user's current gaze direction. As another example, the controller may display particular virtual content within a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the XR experience and focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface within the environment the user is currently viewing on the display 510. As another exemplary use case, eyepiece 520 may be a focusable lens, and eye-tracking information is used by the controller to adjust the focus of eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. Controller 110 can utilize the eye-tracking information to orient and focus eyepiece 520 so that close objects the user is looking at appear at the correct distance.

[0104] In some embodiments, the eye-tracking device is part of a head-mounted device mounted on a wearable housing, which includes a display (e.g., display 510), two eyepieces (e.g., one or more eyepieces 520), an eye-tracking camera (e.g., one or more eye-tracking cameras 540), and a light source (e.g., a light source 530 (e.g., an IR LED or NIR LED)). The light source emits light (e.g., IR light or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in Figure 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0105] In some embodiments, the display 510 emits light within the visible light range and does not emit light within the IR or NIR range, thus not introducing noise into the eye-tracking system. Note that the location and angle of the eye-tracking camera(s) 540 are given as examples and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is positioned on each side of the user's face. In some embodiments, two or more NIR cameras 540 can be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0106] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0107] FIG. 6A illustrates a glint-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a glint-assisted gaze tracking system (e.g., eye tracking device 130 as shown in FIGS. 1 and 5). The glint-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the glint-assisted gaze tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted gaze tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0108] As shown in FIG. 6A, an eye-tracking camera may capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system may continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0109] At 610, if the tracking status is "yes" for the currently captured image, the method proceeds to element 640. If the tracking status is "no" at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eyes.

[0110] At 640, proceeding from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from previous frames. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to verify that the tracking or detection results are reliable. For example, the results may be checked to determine whether the pupils and a sufficient number of glints to perform gaze estimation have been successfully tracked or detected in the current frame. At 650, if the results are not reliable, the tracking state is set to "no" at element 660, and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to "yes" (if not already "yes"), and the pupil and glint information is passed to element 680 to estimate the user's point of gaze.

[0111] 6A is intended to serve as an example of eye-tracking technology that may be used in particular implementations. As will be recognized by those skilled in the art, other eye-tracking technologies, now existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with an XR experience according to various embodiments.

[0112] 6B illustrates exemplary devices connected via one or more communication channels to participate in a transaction, according to some embodiments. One or more exemplary electronic devices (e.g., devices 602 and 604) are optionally configured to detect input (e.g., specific user input, NFC field) and optionally transmit payment information (e.g., using NFC). One or more electronic devices optionally include NFC hardware and are configured to be NFC-enabled.

[0113] The electronic device (e.g., devices 602 and 604) is optionally configured to store payment account information associated with each of one or more payment accounts. The payment account information includes, for example, one or more of a person's or company's name, a billing address, a login, a password, an account number, an expiration date, a security code, a telephone number, a bank associated with the payment account (e.g., an issuing bank), and a card network identifier. In some embodiments, the payment account information includes an image, such as a photograph of a payment card (e.g., taken by and / or received at the device). In some embodiments, the electronic device receives user input including at least some payment account information (e.g., receives a credit, debit, account, or gift card number and expiration date entered by a user). In some embodiments, the electronic device detects at least some payment account information from an image (e.g., a payment card captured by a camera sensor of the device). In some embodiments, the electronic device receives at least some payment account information from another device (e.g., another user device or a server). In some embodiments, the electronic device receives payment account information from a server associated with another service (e.g., an app for renting or selling audio and / or video files) to which the user or user device or account for the identified payment account data has previously made a purchase.

[0114] In some embodiments, a payment account is added to an electronic device (e.g., devices 602 and 604) such that the payment account information is securely stored on the electronic device. In some embodiments, after a user initiates such a process, the electronic device transmits information for the payment account to a transaction coordination server, which then communicates with a server operated by the payment network for the account (e.g., a payment server) to ensure the validity of the information. The electronic device is optionally configured to receive a script from the server that enables the electronic device to program the payment information for the account onto the secure element.

[0115] In some embodiments, communication between electronic devices 602 and 604 facilitates a transaction (e.g., a general or specific transaction). For example, a first electronic device (e.g., 602) can act as a provisioning or management device and send notifications of new or updated payment account data (e.g., information about new accounts, updated information about existing accounts, and / or alerts for existing accounts) to a second electronic device (e.g., 604). In another example, a first electronic device (e.g., 602) can send data to a second electronic device reflecting information about a payment transaction facilitated at the first electronic device. The information optionally includes one or more of the payment amount, the account used, the time of purchase, and whether the default account has changed. The second device (e.g., 604) optionally uses such information to update the default payment account (e.g., based on a learning algorithm or explicit user input).

[0116] The electronic devices (e.g., 602, 604) are configured to communicate with each other over any of a variety of networks. For example, the devices communicate using a Bluetooth connection 616 (e.g., including a conventional Bluetooth connection or a Bluetooth low energy connection) or a WiFi network 614. Communications between user devices are optionally coordinated to reduce the likelihood of inappropriate sharing of information between devices. For example, communications involving payment information require the communicating devices to be paired (e.g., associated with each other through explicit user interaction) or associated with the same user account.

[0117] In some embodiments, the electronic devices (e.g., 602, 604) are optionally used to communicate with a point-of-sale (POS) payment terminal 606 that is NFC-enabled. This communication optionally occurs using various communication channels and / or technologies. In some embodiments, the electronic devices (e.g., 602, 604) communicate with the payment terminal 606 using an NFC channel 618. In some embodiments, the payment terminal 606 communicates with the electronic devices (e.g., 602, 604) using a peer-to-peer NFC mode. The electronic devices (e.g., 602, 604) are optionally configured to transmit a signal to the payment terminal 606 that includes payment information for a payment account (e.g., a default account or an account selected for a particular transaction).

[0118] In some embodiments, proceeding with the transaction includes transmitting a signal including payment information for an account, such as a payment account. In some embodiments, proceeding with the transaction includes reconfiguring the electronic device (e.g., 602, 604) to respond as a contactless payment card, such as an NFC-enabled contactless payment card, and then transmitting account credentials via NFC to a payment terminal 606, etc. In some embodiments, after transmitting the account credentials via NFC, the electronic device reconfigures itself to not respond as a contactless payment card (e.g., requiring authorization before reconfiguring itself to respond as a contactless payment card via NFC again).

[0119] In some embodiments, signal generation and / or transmission is controlled by a secure element within the electronic device (e.g., 602, 604). The secure element optionally requires a specific user input before disclosing the payment information. For example, the secure element optionally requires detecting that the electronic device is being worn, a button press, a passcode entry, a touch, one or more option selections (e.g., received during interaction with an application), a fingerprint signature, a voice or audio command, and / or a gesture or movement (e.g., rotation or acceleration). In some embodiments, if a communication channel (e.g., an NFC communication channel) is established with another device (e.g., payment terminal 606) within a defined period of time from detecting the input, the secure element discloses the payment information to be sent to the other device (e.g., payment terminal 606). In some embodiments, the secure element is a hardware component that controls the disclosure of the secure information. In some embodiments, the secure element is a software component that controls the disclosure of the secure information.

[0120] In some embodiments, the protocols involved in the transaction depend, for example, on the device type. For example, the conditions for generating and / or transmitting payment information may differ between a wearable device (e.g., device 604) and a telephone (e.g., device 602). For example, the conditions for generation and / or transmission on a wearable device include detecting that a button has been pressed (e.g., after security verification), while the conditions on a telephone do not require a button press but instead require the detection of a specific interaction with an application. In some embodiments, the conditions for transmitting and / or disclosing payment information include receiving a specific input on each of multiple devices. For example, disclosing payment information may optionally require the detection of a fingerprint and / or passcode on one device (e.g., device 602) and the detection of a machine input (e.g., a button press) on another device (e.g., device 604).

[0121] The payment terminal 606 optionally generates a signal to send to the payment server 612 using payment information to determine whether the payment is authorized. The payment server 612 optionally includes any device or system configured to receive payment information related to a payment account and determine whether the proposed purchase is authorized. In some embodiments, the payment server 612 includes a server of the issuing bank. The payment terminal 606 communicates with the payment server 612 directly or indirectly through one or more other devices or systems (e.g., a server of the acquiring bank and / or a server of the card network).

[0122] The payment server 612 optionally uses at least a portion of the payment information to identify a user account from a database of user accounts (e.g., 608). For example, each user account includes payment information. An account is optionally located by locating the account that has specific payment information matching the payment information from the POS communication. In some embodiments, payment is rejected if the provided payment information is inconsistent (e.g., the expiration date does not correspond to a credit, debit, or gift card number) or if there is no account with payment information matching the payment information from the POS communication.

[0123] In some embodiments, the data for a user account further identifies one or more limits (e.g., credit limit), current or previous balance, previous transaction date, location, and / or amount, account status (e.g., active or frozen), and / or authorization instruction. In some embodiments, a payment server (e.g., 612) uses such data to determine whether to authorize the payment. For example, the payment server may reject the payment if the purchase amount added to the current balance exceeds the account limit, if the account is frozen, if the amount of previous transactions exceeds a threshold, or if the number or frequency of previous transactions exceeds a threshold.

[0124] In some embodiments, the payment server 612 responds to the POS payment terminal 606 with an indication as to whether the proposed purchase was authorized or denied. In some embodiments, the POS payment terminal 606 sends a signal to the electronic device (e.g., 602, 604) to identify the outcome. For example, the POS payment terminal 606 sends a receipt to the electronic device (e.g., 602, 604) when the purchase is authorized (e.g., via a transaction coordination server that manages the transaction app on the user device). In some cases, the POS payment terminal 606 presents an output (e.g., a visual or audio output) indicating the outcome. Payment can be sent to the merchant as part of the authorization process or can be sent at a later time.

[0125] In some embodiments, the electronic device (e.g., 602, 604) participates in a transaction that is completed without the intervention of the POS payment terminal 606. For example, upon detecting that machine input has been received, a secure element within the electronic device (e.g., 602, 604) exposes payment information so that an application on the electronic device can access the information (e.g., transmit the information to a server associated with the application).

[0126] In some embodiments, the electronic device (e.g., 602, 604) is in a locked state or an unlocked state. In the locked state, the electronic device is powered on and operational, but is prevented from performing a predefined set of operations in response to user input. The predefined set of operations may include navigation between user interfaces, activation or deactivation of a predefined set of features, and activation or deactivation of specific applications. The locked state may be used to prevent unintended or unauthorized use of some functionality of the electronic device or activation or deactivation of some features on the electronic device. In the unlocked state, the electronic device 602 is powered on and operational, and is not prevented from performing at least a portion of the predefined set of operations that cannot be performed in the locked state.

[0127] When a device is in a locked state, the device is said to be locked. In some embodiments, a device in a locked state may respond to a limited set of user inputs, including inputs corresponding to attempts to transition the device from an unlocked state or inputs corresponding to powering off the device.

[0128] In some embodiments, the secure element is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms. In some embodiments, the secure element provides (or reveals) payment information (e.g., an account number and / or a transaction-specific dynamic security code). In some embodiments, the secure element provides (or reveals) payment information in response to the device receiving authorization, such as user authentication (e.g., fingerprint authentication, passcode authentication, detecting a double press of a hardware button by providing authentication credentials to the device when the device is in an unlocked state and, optionally, when the device is continuously on the user's wrist since the device was unlocked, etc., where the continued presence of the device on the user's wrist is determined by periodically checking that the device is in contact with the user's skin). For example, the device detects a fingerprint with a fingerprint sensor (e.g., a fingerprint sensor integrated into a button) on the device. The device determines whether the fingerprint matches a registered fingerprint. In response to determining that the fingerprint matches the registered fingerprint, the secure element provides (or reveals) payment information. If the system determines that this fingerprint does not match any registered fingerprints, it will suspend the provision (or disclosure) of payment information.

[0129] In this disclosure, various input methods are described with respect to interaction with a computer system. Where one example is provided using one input device or input method and another example is provided using a different input device or input method, it should be understood that each example may be compatible with, and optionally utilize, the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. Where one example is provided using one output device or output method and another example is provided using a different output device or output method, it should be understood that each example may be compatible with, and optionally utilize, the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. Where one example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with, and optionally utilize, the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User Interface and Related Processes

[0130] Attention is now directed to embodiments of a user interface (UI) and associated processes that may be implemented in a computer system, such as a portable multifunction device or a head-mounted device, in communication with a display generation component, one or more input devices, and (optionally) a biometric sensor.

[0131] Figures 7A-7N illustrate examples of authorizing secure operation. Figure 8 is a flow diagram of an example method 800 for facilitating user consent for secure operation. Figure 9 is a flow diagram of a method 900 of continuity of authentication for secure operation. Figure 10 is a flow diagram of an example method 1000 for authorizing secure operation via an accessibility interface. The user interfaces of Figures 7A-7N are used to illustrate processes described below, including the processes of Figures 8, 9, and 10.

[0132] FIG. 7A illustrates an electronic device 700 including a display 700a. In FIG. 7A, a user 704 is interacting with the electronic device 700. The electronic device 700 may correspond to a tablet device, a wearable device (e.g., a head-mounted display), a smartphone, and / or a smartwatch. Additionally, the display 700a may include or be coupled to display generation components (e.g., a display controller, a touch-sensitive display system, a display (e.g., built-in and / or connected), a 3D display, a transparent display, a projector, and / or a head-up display). The electronic device 700 displays one or more interface objects on the display 700a, such as a display login user interface 702. In some embodiments, the login user interface 702 corresponds to a virtual interface object in the XR environment. The login user interface 702 optionally includes various display fields, such as a username field 706 and a password field 708. Login user interface 702 also optionally includes affordances 710 (e.g., “autofill”) for initiating a secure action process, such as authorization of various displayed fields. In some embodiments, the secure action corresponds to a payment, such as providing credit card information to an e-commerce website to purchase shoes (as described in more detail with respect to FIG. 7N ), or providing payment information to a third-party service to purchase an item. In some embodiments, the secure action corresponds to providing access credentials to access an application. For example, the secure action provides access credentials to an application associated with a third-party service (e.g., a stock trading application) that requires user authentication before accessing that application. In some embodiments, electronic device 700 allows user 704 to manually enter information into fields 706 and / or 708 (e.g., using a displayed keyboard and / or voice commands).

[0133] 7A , the electronic device 700 detects a selection of an affordance 710 by a user 704 to initiate an autofill function that automatically fills in information in fields 706 and / or 708. For example, the display 700a is a touch-sensitive display, such that the electronic device 700 detects touch input (e.g., a tap or a tap-and-hold) on the affordance 710. In some embodiments, when the device 700 is implemented in an XR environment, the electronic device 700 optionally detects the selection of the affordance 710 using a controller communicatively coupled to the electronic device 700. Additionally, the electronic device 700 detects the selection of the affordance 710 using a gaze, for example, by detecting that the user 704 is looking at the affordance 710 for a predetermined period of time and / or (optionally) while detecting one or more gestures. For example, electronic device 700 optionally corresponds to a head-mounted display, and is adapted to detect when user 704 performs one or more gestures, such as air gestures, to activate affordance 710. Electronic device 700 optionally displays additional / other elements on display 700a, such as an additional user interface (e.g., an email application), representations of other users (e.g., via a video call, an avatar of another user in an XR environment), and / or representations of a physical environment, such as the physical environment of user 704.

[0134] 7B , in response to activation of affordance 710, electronic device 700 displays authorization interface 712. In some embodiments, authorization interface 712 corresponds to a displayed virtual interface object in the XR environment. Permission interface 712 includes search function 714 and one or more credential options 716a and 716b. Credential option 716a is associated with (e.g., used to select) information, such as, for example, a username "janeapples@mail.com" to log in to website "abc.com." In some embodiments, electronic device 700 detects that a credential option, such as credential option 716b (shown in FIG. 7B ), has been selected. Credential option 716b is associated with information, such as, for example, a username "janeapples@mail.com" to log in to website "xyz.com." In some embodiments, the credential option is associated with a separate password, which may be displayed as part of authorization interface 712 or may not be displayed to protect privacy. In some embodiments, the electronic device 700 receives information (e.g., text and / or voice) via the search functionality 714 and, in response, displays one or more search result options corresponding to the received information for use in the autofill functionality.

[0135] In some embodiments, various requirements (a set of one or more criteria) must be met for the electronic device 700 to enable user authorization of a secure operation, including a visibility criterion and / or a user authentication criterion. For example, the visibility criterion is met when a threshold amount of one or more interface objects, such as the login user interface 702 and / or the authorization interface 712, is visible from the user's 704 perspective (e.g., the threshold amount may correspond to any portion of an interface object, a non-zero amount of an interface object, or an entire interface object). Additionally, the user authentication criterion is met when the electronic device 700 receives a request to perform a secure operation. For example, the electronic device 700 detects the user's 704 selection of a displayed individual credential option, such as credential option 716b (as shown in FIG. 7B ). Once the credential option is selected, the user 704 may provide an input (e.g., a double press of a hardware button) that authorizes the secure operation to be performed, as described in more detail with respect to FIG. 7D . Pursuant to determining that input permitting performance of a secure operation has been received, electronic device 700, provided that the set of one or more criteria is met, causes the individual credential option to be used to automatically fill in displayed fields, such as username field 706 and password field 708, with individual credential information. In Figure 7B, electronic device 700 receives a selection of credential option 716b.

[0136] In Figure 7C, upon receiving the selection of credential option 716b, the electronic device 700 updates the authorization interface as shown in Figure 7C. In Figure 7C, the electronic device 700 detects a user gaze direction 718 related to (e.g., corresponding to) the user 704's gaze direction. Typically, the user gaze direction 718 is not part of the displayed user interface of the electronic device 700, but is provided to aid in illustrating this technique. While the user 704 is interacting with the device 700, the electronic device 700 detects a change in the user 704's viewpoint (e.g., a change from a first viewpoint to a second viewpoint different from the first viewpoint) based on the change in the user gaze direction 718. In some embodiments, the user gaze direction is detected within the context of the XR environment (e.g., via a head-mounted display). The user gaze direction 718 optionally indicates that the user 704's gaze is not directed towards the login user interface 702 or the authorization interface 712. Therefore, the electronic device 700 determines that the login user interface 702 and the authorization interface 712 are sufficiently invisible from the user 704's viewpoint. For example, the user's line of sight direction 718 may be directed towards a portion of the display 700a that is not close to the location where the login user interface 702 and the authorization interface 712 are displayed. In some embodiments, portions of the login user interface 702 or the authorization interface 712 that are within a predetermined distance (e.g., a predetermined number of pixels and / or a predetermined length) from the center of the user's line of sight direction 718 are determined to be visible from the user 704's viewpoint. For example, 10% of the authorization interface 712 may be determined to be within a predetermined distance from the center of the user's line of sight direction 718, and 5% of the login user interface 702 may be determined to be within a predetermined distance from the center of the user's line of sight direction 718. In this example, the electronic device 700 determines that the amount of the login user interface 702 and / or the authorization interface 712 that is visible from the user's viewpoint is less than a threshold amount (e.g., a 75% threshold amount or a 90% threshold amount).

[0137] In Figure 7C, the electronic device 700 disables and / or deactivates user authorization for the autofill function based on a determination that the amount of the login user interface 702 and / or authorization interface 712 visible from the user's perspective is below a threshold. In some embodiments, the electronic device 700 modifies the appearance of one or more interface objects to indicate that the user cannot request the execution of a secure action (e.g., by graying out the authorization interface 712). For example, in response to detecting the selection of credential option 716b, the electronic device 700 does not automatically fill in the username field 706 and password field 708 while the login user interface 702 and authorization interface 712 are not sufficiently visible from the user 704's perspective (e.g., based at least on the user's line of sight direction 718).

[0138] 7D , the electronic device 700 determines that the user 704 is authorized to perform a secure operation, and therefore, user permission for the autofill function is enabled. In some examples, user permission for the secure operation is enabled conditioned on determining that the user 704 is authorized to perform the secure operation. For example, the electronic device 700 determines that the user 704 is authorized to perform the secure operation based on gaze criteria or other biometric criteria. The gaze criteria is optionally met when the user 704 gazes at a respective interface object, such as the login user interface 702 and / or the permission interface 712. Optionally, the login user interface 702 and / or the permission interface 712 are displayed in a central portion of the display 700a. In some embodiments, the electronic device 700 determines that the user 704 is not authorized to perform a secure operation when the user is not gazing at the login user interface 702 and / or the authorization interface 712 (e.g., the user's eyes are closed or the user is looking toward a corner of the display 700a (away from the interfaces 702 and / or 712)). In some embodiments, user authorization for a secure operation is enabled conditioned on a determination that the user has or provides a specific movement of a biometric characteristic. For example, the specific movement may include the user gazing at a specific portion of the login user interface 702 (e.g., affordance 710) or a specific portion of the authorization interface 712 (e.g., a specific credential option), the user providing a specific facial rotation, and / or the user moving a finger along a specific path.

[0139] Typically, the electronic device 700 operates in various modes. For example, the electronic device 700 may optionally operate in a first mode, such as a standard mode in which accessibility features are not enabled, and / or a mode in which secure operation is permitted using a default physical input mechanism, such as a hardware button 722 (as described with respect to Figure 7D). In some embodiments, the device 700 may optionally operate in a second mode, such as an accessibility mode, and / or a mode in which input via a physical input mechanism is not required to permit secure operation (as described in more detail with respect to Figures 7L to 7M). Returning to Figure 7D, the electronic device 700 enables user permission for the autofill function, based on the determination that at least a threshold amount of the login user interface 702 and / or the permission interface 712 is visible from the user 704's viewpoint. For example, the user's line of sight 718 may be directed towards the permission interface 712. Therefore, it is determined that the user's line of sight is the authorization interface 712, and that 95% or 100% of the authorization interface 712 is visible from the user 704's viewpoint. As a result, the electronic device 700 determines that at least a threshold amount (e.g., a 75% threshold amount or a 90% threshold amount) of the login user interface 702 and / or the authorization interface 712 is visible from the user's viewpoint. In some embodiments, the electronic device 700 then enables user authorization for secure operation and indicates to the user that user authorization for secure operation is enabled via an indication 720 that includes the text instruction “Double-click to authorize”. Specifically, the indication 720 notifies the user that the autofill function is authorized by double-pressing a hardware button, such as a hardware button 722. In some embodiments, user authorization for the autofill function remains enabled for a certain period (e.g., until 1 second, until 3 seconds, etc.) after it is determined that the login user interface 702 and / or authorization interface 712 are visible from the user's perspective, and optionally is disabled after that period.In some embodiments, device 700 visually indicates that the viewpoint includes the authorization interface 712 and therefore its individual authorization is enabled (for example, by de-graying out the authorization interface 712 and / or buttons).

[0140] If the autofill function is enabled, user 704 may proceed to enable secure operation. Specifically, the electronic device 700 may optionally modify the appearance of one or more interface objects and / or display one or more additional interface objects to indicate to the user that the user can request the execution of secure operation. For example, if the electronic device 700 is implemented in an XR environment, the electronic device 700 may display indications such as commands for performing one or more air gestures (e.g., instructions to move one or more hands or fingers in a specific motion) and / or commands for performing one or more inputs on a controller adapted to the XR environment.

[0141] 7D , the electronic device 700 receives user authorization for the autofill feature via a double-click of the hardware button 722. In response to receiving user authorization for the autofill feature, a determination is optionally made as to whether a set of one or more execution criteria is met. For example, visibility may be required to be met before, during, and / or after receiving a user authorization request to perform a secure operation. In some embodiments, the one or more execution criteria optionally also include a requirement that at least a threshold amount of the login user interface 702 and / or the authorization interface 712 be visible from the perspective of the user 704 after the user 704 requests the secure operation to be performed. In some embodiments, the set of one or more execution criteria includes a user authentication criterion that is met when the user is authenticated (e.g., via a password and / or biometric authentication). 7F, the criteria optionally include a positive indication of a first level (e.g., full or high fidelity) iris feature verification, a positive indication of continued (e.g., second level) iris feature verification from the time the user 704 begins interacting with the device 700, and / or a positive indication of proper passcode authentication. As a result, upon receiving a request to perform a secure operation, the electronic device 700 begins performing the secure operation pursuant to a determination that one or more sets of execution criteria have been met.

[0142] 7E , the electronic device 700 provides the user 704 with an indication that the user's authorization of the autofill feature was successful. Specifically, the electronic device 700 modifies (or otherwise replaces with an additional interface) the authorization interface 712 to include a positive indicator that the user's authorization of the autofill feature was successful (e.g., a "check" mark next to the text "Done"). In addition, the electronic device 700 updates the login user interface 702 by updating the username field 706 to include the username of the username corresponding to the selected credential option and updating the password field 708 to include obscured characters that conceal the password corresponding to the selected credential option, indicating successful user authorization of the autofill feature. In some embodiments, the electronic device 700 does not obscure the password corresponding to the selected credential option and updates the password field 708 to include readable characters.

[0143] FIG. 7F illustrates a user authentication process. Specifically, continuing from FIG. 7D, in response to receiving a request to perform a secure operation, a determination is made as to whether the user 704 meets user authentication criteria to perform the secure operation. In some embodiments, a determination is made that the set of execution criteria is not met. Typically, a first type of biometric authentication is performed at a first time (e.g., the first time of device interaction), such that the electronic device 700 biometrically authenticates the user 704 using one or more biometric sensors of the device 700. The first time may correspond to the first time the user activates the electronic device 700 and / or the first time the user wears the electronic device 700 on a part of the user's body. In embodiments in which the device 700 corresponds to a head-mounted display, the first time may correspond to the time the user 704 secures / wears the electronic device 700 on the user's 704's head. The first type of biometric authentication may include a first level (e.g., full or high-fidelity) of iris feature verification. The first level of iris feature verification includes a retinal scan to obtain one or more images and / or measurements of at least one eye of the user 704. In some embodiments, different users have different iris features. The one or more images and / or measurements are then compared to one or more stored images and / or measurements corresponding to the user's enrolled retinal scan to determine a match and / or similarity between the obtained image(s) / measurements and the enrolled image(s) / measurements. If the obtained image(s) / measurements and the enrolled image(s) match with a sufficient degree of similarity (e.g., 95% or 99% similarity), the electronic device 700 determines that the user 704 is authenticated. In some embodiments, the first type of biometric authentication optionally includes fingerprint verification, facial recognition verification, and / or voiceprint verification (in addition to or instead of iris feature verification).

[0144] The user authentication criteria used to determine whether to perform secure operation are optionally based on sensor measurements taken at multiple intermediate time points during a specific time frame, specifically after the initial time when user 704 began interacting with the electronic device 700 and / or after user 704 has worn the electronic device 700. The sensor measurements are optionally based on measurements taken to detect whether the same user (e.g., user 704) is using the device 700 throughout the entire duration of the interaction. For example, in an embodiment where the electronic device 700 corresponds to a head-mounted display, optionally the electronic device 700 authenticates user 704 via a first type of biometric authentication (e.g., full or high fidelity) when user 704 attaches / wears the electronic device 700 on user 704's head. If authentication via the first type of biometric authentication is successful, the electronic device 700 repeatedly takes sensor measurements using a second type of biometric authentication. In some embodiments, the first type of biometric authentication corresponds to high-fidelity biometric verification, so that the electronic device 700 analyzes characteristics related to the entire eye or a portion of the user 704's eye, such as iris features. In some embodiments, the second type of biometric authentication corresponds to one or more sensor measurements that directly or indirectly confirm whether a biometric feature has been continuously present since high-fidelity biometric verification was performed for that biometric feature. The second type of biometric authentication optionally analyzes the same (or similar) amount of biometric features as the first type of biometric authentication, or optionally analyzes fewer biometric features compared to the first type of biometric authentication. In some embodiments, the second type of biometric authentication is used to verify that a user authenticated using the first type of biometric authentication has remained a user of the electronic device 700 since the first type of biometric authentication was performed (e.g., without interruption of the continuity of use of that biometric feature), so in some embodiments, the second type of biometric authentication may be referred to as continuity verification.

[0145] In some embodiments, both the first type and the second type of biometric authentication are enabled while the electronic device 700 is operating in a first mode (e.g., a standard mode in which accessibility features are not enabled and / or a mode that allows secure operation using a default physical input mechanism). In some embodiments, the first type of biometric authentication is enabled and the second type of biometric authentication is disabled while the electronic device 700 is operating in a second mode (e.g., an accessibility mode and / or a mode in which input via a physical input mechanism is not required to allow secure operation). Thus, in some embodiments, while operating in the second mode (as described in more detail with respect to FIGS. 7L and 7M ), the first type of biometric authentication (e.g., full or high-fidelity) is optionally required in response to receiving a request to perform a secure operation.

[0146] In some embodiments, at least one (e.g., the same or different) biometric sensor is used to perform both the first type of biometric authentication and the second type of biometric authentication (e.g., a retinal scanner, a depth camera, and / or a proximity sensor).

[0147] For example, the second type of biometric authentication optionally focuses on detecting features and characteristics such as pupils, eye glint, and iris features (e.g., shape, size, and / or color pattern) of user 704. Specifically, electronic device 700 captures information about the appearance of the user's eyes and records information about the detected pupils, eye glint, and / or iris features. Electronic device 700 again captures information about the appearance of the user's eyes at a subsequent time (e.g., 1 second later, 2 seconds later, etc.) and stores information about the detected pupils, eye glint, and / or iris features. A comparison is made between the information recorded at the first time and the information recorded at the second time to determine whether the same user (e.g., user 704) was using the device at both times. For example, if a first detected pupil, eye glint, and / or iris feature matches (e.g., exactly and / or with sufficient confidence) a second detected pupil, eye glint, and / or iris feature, the electronic device 700 determines that the same user used the device at both times, and therefore the respective criteria remain satisfied. This process may repeat until a particular time, such as when the user provides a request to perform a secure operation. If, at this particular time, a determination has been made (since the first type of biometric authentication) that the same user has not used the device at any given time (e.g., the first detected information did not match the second detected information), the respective criteria are determined to be not satisfied, and as a result, performance of the secure operation is aborted (e.g., until further authentication is performed). For example, if sensor measurements indicate that the presence of an eye, iris feature, pupil, and / or eye glint was not detected, the respective criteria would be determined to be not satisfied. The lack of presence of eye features may be based on the user's eyes being closed or otherwise not being detectable by the sensor (e.g., an improperly mounted head-mounted display). For example, in embodiments in which electronic device 700 represents a head-mounted display, the user may have removed the head-mounted display.In embodiments in which electronic device 700 corresponds to a tablet or smartphone device, user 704 may be holding device 700 such that the eyes of user 704 are not within the field of view of the sensor (e.g., electronic device 700 is placed upside down on a surface). In other words, each criterion optionally includes an "eyes open" criterion that is met when at least one eye of user 704 does not remain continuously closed for more than a threshold period (e.g., a non-zero period, such as 1 second or 2 seconds).

[0148] In some embodiments, the first type of biometric authentication is based on different biometric features than the second type of biometric authentication. Specifically, the first type of biometric authentication optionally corresponds to full iris feature verification, such that the electronic device 700 analyzes characteristics related to the entire eye or portions of the user's 704 eyes, such as pupils, eye glint, and / or iris features. The second type of biometric authentication may correspond to partial facial recognition verification. Partial facial recognition optionally includes analyzing features related to portions of the face surrounding (and excluding) the eyes, such as the user's 704 eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose. For example, the electronic device 700 captures the appearance of the portion of the user's 704 face surrounding the eyes. At a later time (e.g., one or two seconds later), the electronic device 700 again captures the appearance of the same portion of the user's 704 face surrounding the eyes. The electronic device 699 compares the information recorded at the first time with the information recorded at the second time to determine whether the same user (e.g., user 704) used the device at both times. For example, if the first detected eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose match (e.g., perfectly or with sufficient confidence) the second detected eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose, it is determined that the same user used the device at both times, and thus the respective criteria remain satisfied. The process may repeat until a specific time, such as when the electronic device 700 receives a request from the user to perform a secure operation. If, at this specific time, it is determined that the same user did not use the device from the time of the first type of authentication (e.g., the first detected information did not match the second detected information), it is determined that the respective criteria have not been satisfied, and as a result, the electronic device 700 refrains from performing the secure operation (e.g., until further authentication is performed). In other words, each criterion may optionally include non-eye criteria that can be met even if the user 704's eyes are undetectable (e.g., undetectable over a threshold period of 1, 5, 10, 15, or 60 seconds).

[0149] In some embodiments, using a first biometric feature for a first type of biometric authentication and another biometric feature for a second type of biometric authentication improves authentication. For example, analyzing eye features for the first type of biometric authentication and facial features for the second type of biometric authentication improves authentication when eye features are not always available for analysis. Specifically, the user 704 is initially authenticated based on eye features, but facial features (e.g., eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose) are used for the second type of biometric authentication. In this case, even in situations where the user's 704 eyes cannot be detected by one or more sensors of the electronic device 700, the electronic device 700 can still detect the user's facial features and perform the second type of authentication to allow secure operations to be performed.

[0150] Returning to FIG. 7F , in response to receiving a request to perform a secure operation, electronic device 700 determines whether user 704 meets the respective authentication criteria to perform the secure operation. In some embodiments, user 704 was biometrically authenticated at a first time, and the request to perform the secure operation is received at a second time (after the first time). In this case, if a determination is made that user 704 did not meet the respective criteria between the first and second times, authentication guidance user interface 724 is provided to the user to biometrically authenticate user 704 using the first type of biometric authentication. In other words, if electronic device 700 cannot verify that the same user has been using the device since the initial first type of biometric authentication (or can verify that the same user has not been using the device since the initial first type of biometric authentication), electronic device 700 attempts to verify the user's identity again using the first type of biometric authentication. Failure to meet the respective authentication criteria can be the result of various factors. For example, the user's 704 eyes may be undetectable for a period of time between the first time and the second time. In some embodiments, the initial authentication may be based on another type of biometric authentication (e.g., fingerprint authentication) or non-biometric authentication (e.g., passcode entry). Certain initial authentications, such as fingerprint authentication or passcode authentication, optionally require the user 704 to provide a specific first type of biometric authentication (e.g., full or high-fidelity iris feature verification) to grant secure operation because the respective authentication criteria are not automatically met. In some embodiments, the authentication guidance user interface 724 includes instructions for performing the first type of biometric authentication, such as instructions requesting the user to provide specific movements of individual biometric features. For example, the authentication guidance user interface 724 may include an initial prompt such as "Biometric authentication required," with an option for the user to "continue" and another option for the user to "cancel." In FIG. 7F , the electronic device 700 detects that the "continue" option has been activated.

[0151] 7G , in response to detecting activation of the “Continue” option, electronic device 700 initiates a first type of biometric authentication. When the first type of biometric authentication corresponds to iris feature verification, electronic device 700 instructs the user to look at various portions of the display as the biometric authentication begins (e.g., via a “Look Here” prompt). During the first type of biometric authentication, electronic device 700 optionally analyzes gaze 726 corresponding to user 704 to obtain characteristics associated with the user's 704 eye or portions of the user's 704 eye, such as iris features, pupil, and / or glint. When the first type of biometric authentication corresponds to face verification, electronic device 700 instructs user 704 to rotate their face at a particular angle. In some embodiments, when the first type of biometric authentication corresponds to fingerprint verification, electronic device 700 instructs the user to move their finger along a particular path and / or touch a touch-sensitive contact of device 700 or another device in a particular manner.

[0152] 7H illustrates a passcode entry prompt 728. In some embodiments, the user authentication criteria are met when the passcode entry matches a registered passcode entered via the passcode entry prompt 728. For example, the electronic device 700 is optionally configured such that user authentication (or user authentication to perform secure operations) occurs via a passcode or password (e.g., the user has disabled iris feature verification, face verification, and / or other types of biometric authentication). Alternatively, following a determination that the user authentication criteria are not met based on a failed first type of biometric authentication (e.g., the device 700 was unable to successfully complete the first type of biometric authentication), the electronic device 700 displays the passcode entry prompt 728 to provide an additional attempt to meet the user authentication criteria. In some embodiments, a determination may be made that the initial first type of biometric authentication was successful (e.g., the user successfully passed the biometric authentication the first time they accessed the device), but the second type of biometric authentication to detect continuity may have failed (e.g., the user's 704 eyes were undetectable for a period of time after the initial time). In this case, the electronic device 700 attempts to authenticate the user by displaying a passcode entry prompt 728 to the user.

[0153] In some embodiments, if continuity verification fails, the electronic device 700 first attempts to authenticate the user by means other than passcode verification before attempting passcode verification. For example, the electronic device 700 attempts a first type of biometric authentication, such as full or high-fidelity iris feature verification (e.g., as shown in FIG. 7G ), or another type of full or high-fidelity biometric authentication, such as fingerprint authentication and / or facial recognition authentication. If full or high-fidelity biometric authentication fails, passcode verification is optionally used as the default option. In some embodiments, the passcode entry prompt 728 includes a passcode entry portion and an affordance for rejecting the passcode entry. If the user selects the option to reject the passcode entry, the user's permission for secure operation is withdrawn.

[0154] FIG. 7I illustrates a change in the viewpoint of the user 704, following FIG. 7D. Typically, the visible amount of an interface object may change based on a change in the viewpoint of the user 704. In some embodiments, at least a portion of the login user interface 702 and the permission interface 712 is invisible to the user 704 based on, for example, portions of the login interface and portions of the permission interface not being fully displayed on the display 700a. In some embodiments, when the electronic device 700 is implemented in an XR environment, certain user motions optionally cause interface objects to become invisible or partially invisible. For example, the user turns their head in a particular direction while wearing the device 700 (e.g., a head-mounted display). The login user interface 702 and the permission interface 712 are optionally environment-locked within the environment, so that the interfaces remain in the same general location within the environment regardless of the user's head movements or other motions. Thus, at least a portion of the interface is optionally invisible to the user 704 based on the interface being toward the side of the user's 704's field of view (or behind the user in the XR environment). Thus, based on the user's 704's movement, a visible amount of the login user interface 702 and the permission interface 712 may be reduced. As a result, a determination is made that less than a threshold amount of the login user interface 702 and / or the permission interface 712 is visible from the user's 704's perspective. Based on this determination, the electronic device 700 modifies the appearance of the permission interface 712, for example, to indicate that the user 704 cannot request performance of the secure operation.

[0155] In FIG. 7J , an exemplary environment is depicted on a display 700a, including a background region 730. In some embodiments, the environment is implemented in the context of an XR environment, such that the electronic device 700 corresponds to a head-mounted display. The background region 730 optionally corresponds to a representation of the user's 704's physical environment, a virtual environment, or a combination of the physical and virtual environments. The user 704 may interact with various interface objects, virtual objects, or other objects depicted in the environment. For example, the electronic device 700 displays a messaging interface 732, through which the user 704 can send and receive messages. In addition, the electronic device 700 also displays a login user interface 702 and an authorization interface 712. While the electronic device 700 detects the user 704 interacting with an interface, the interface may be fully visible to the user 704, such that the interface is not occluded by objects or other elements in the environment. Electronic device 700 optionally detects that user 704 is currently looking at permission interface 712, as depicted by user gaze direction 718. User permission for secure operation is optionally enabled, as described herein. Thus, the appearance (e.g., not grayed out) of permission interface 712 and indication 734 notifies the user that the user can authorize the autofill function, for example, by performing the function indicated by indication 734 (e.g., double-pressing button 722).

[0156] Figure 7K shows an exemplary environment including a change in the user 704's viewpoint. In some embodiments, the visibility of objects depicted in the environment changes based on occlusion by other objects in the environment. Such occlusion occurs based on the movement of the object itself (which is occluded or occluding) or a change in the user 704's viewpoint. For example, in relation to Figure 7J, the electronic device 700 detects that the user 704 has turned left in the environment. Specifically, if the environment is implemented in the context of an XR environment, the electronic device 700 detects that the user 704 has turned their head to the left while wearing the electronic device 700 (e.g., a head-mounted display). Various objects in the environment may be viewpoint-locked or environment-locked. Specifically, the messaging interface 732 may be viewpoint-locked so that the messaging interface 732 appears at the same location and / or position in the user 704's viewpoint even when the user 704's viewpoint changes. Alternatively, the login user interface 702 and authorization interface 712 may be environment-locked, so that they appear in a position within the user's viewpoint that is based on their location within the XR environment (for example, in front of a sofa representation). Therefore, when user 704 adjusts their viewpoint to the left (relative to Figure 7J), the messaging interface 732 appears to move over the login user interface 702 and authorization interface 712. In other words, the login user interface 702 and authorization interface 712 remain in the same overall location relative to the background of the XR environment from the user's viewpoint, and as a result of the user's viewpoint shift, the messaging interface 732 is overlaid on top of the login user interface 702 and authorization interface 712. When the overlay by the messaging interface 732 occurs based on the change in viewpoint, the visibility of the login user interface 702 and authorization interface 712 is reduced.The electronic device 700 detects the user's gaze direction 718 as remaining directed toward the permission interface 712, but a determination is made that less than a threshold amount of the login user interface 702 and / or the permission interface 712 is visible from the user's perspective, and therefore the appearance of the permission interface 712 is adjusted (e.g., grayed out) to reflect that the user's permission for secure operation has been disabled.

[0157] In FIG. 7L, electronic device 700 displays an interface for authorizing secure operation while electronic device 700 is operating in a second mode different from the first mode (described with respect to FIGS. 7A-7K). In some embodiments, electronic device 700 operates in the second mode, such as an accessibility mode and / or a mode in which input via a physical input mechanism is not required to authorize secure operation. While in the second mode, electronic device 700 displays login user interface 736. Login user interface 736 corresponds to a login interface that includes functionality to facilitate interaction for users with accessibility requirements (e.g., limited fine motor skills, vision, and / or hearing abilities). Electronic device 700 also optionally displays authorization interface 738 (e.g., an “autofill” function) that facilitates the process for secure operation, such as authorizing information into certain displayed fields.

[0158] In some embodiments, the login user interface 736 and the authorization interface 738 correspond to displayed virtual interface objects within the XR environment. The authorization interface 738 includes a search function 742 and one or more credential options 744a and 744b. In some embodiments, the credential options are associated with a username and / or password. In some embodiments, the electronic device 700 receives a search query for additional credential options via the search function 742 and provides any populated result options based on the search. While in the second mode, the user 704 may interact with the authorization interface 738 (e.g., select one of the credential options 744a or 744b or search for additional credential options via the search function 742) using an input or combination of inputs, such as gaze input, controller input, and / or voice input, that are compatible with accessibility features. In some embodiments, the electronic device 700 detects the selection of a credential option (e.g., credential option 744a). After the credential option is selected, user 704 may activate affordance 740 using an input or combination of inputs, such as gaze input, controller input, and / or voice input, that are compatible with accessibility features. Specifically, affordance 740 may be displayed with the text "Confirm with Assistive Touch" to indicate to the user that the device is operating in the second mode, and thus no physical input mechanism is required to allow the autofill operation. In FIG. 7L, electronic device 700 detects selection of affordance 740.

[0159] In FIG. 7M , in response to detecting selection of affordance 740, electronic device 700 displays assistive input interface 742 including representations of multiple functions for facilitating various tasks. In some embodiments, user 704 must gaze at affordance 740 while activating affordance 740 to trigger selection of affordance 740. For example, user 704 gazes at affordance 704 and activates a special input while gazing at affordance 740 (described in more detail in paragraphs

[0272] -

[0273] ). Additionally, in response to user selection of affordance 740, electronic device 700 also modifies the appearance of authorization interface 738 to indicate that a credential option was selected by the user. In some embodiments, the position or location of authorization interface 738 is adjusted (e.g., in an XR environment, authorization interface 738 is moved to the side of the user's field of view).

[0160] The multiple functions included in the assistive input interface 742 include a variety of options. For example, the various options include an option to navigate to the main screen (e.g., a “Home” option), an option to display device notifications (e.g., “Notifications”), and / or an option to display a settings menu (e.g., “Control Menu”). The assistive input interface 742 includes a secure operation permission affordance 744. The multiple functions can be optionally invoked using a specific input type, which may be pre-configured by the electronic device 700 or a specific user. For example, while in the first mode, the option to navigate to the main screen is executed in response to the electronic device 700 receiving a primary input type (e.g., a “swipe” input at a specific location), while while in the second mode, the option to navigate to the main screen is executed in response to the electronic device 700 receiving an accessibility-based input type (e.g., the user looking at the “Home” icon in combination with a long press of the controller). In addition, while in the first mode, enabling secure operation is performed in response to the electronic device 700 receiving a primary input type such as a double press of a specific hardware button, while while in the second mode, enabling secure operation is performed by accessibility-based input types such as activating a secure operation enablement affordance (for example, the user gazing at the secure operation enablement affordance 744 in combination with a long press of the controller).

[0161] In some embodiments, in response to receiving input authorizing secure operation, electronic device 700 performs user authentication. For example, a first type of biometric authentication is performed, whereby user 704 is biometrically authenticated using one or more biometric sensors of device 700. The first type of biometric authentication optionally includes a first level (e.g., full or high-fidelity) of iris feature verification. Specifically, while operating in the second mode, second type of biometric authentication (e.g., passive iris feature verification, in which electronic device 700 repeatedly analyzes specific characteristics associated with user 704's entire eye or a portion of user 704's eye while user 704 interacts with device 700) is disabled. Thus, once input authorizing secure operation is received, first type of biometric authentication is required. In particular, (referring back to FIG. 7F ), electronic device 700 provides an authentication guidance user interface to the user to biometrically authenticate user 704 using the first type of biometric authentication. The authentication guidance user interface includes instructions for performing a first type of biometric authentication, such as instructions requesting the user to provide a particular movement of an individual biometric feature. For example, the authentication guidance user interface includes an initial prompt such as "Biometric authentication required," with an option for the user to "continue" and another option for the user to "cancel."

[0162] When the electronic device 700 detects that the option to “continue” has been activated, the electronic device 700 begins a first type of biometric authentication. If the first type of biometric authentication corresponds to iris feature verification, the electronic device 700 optionally instructs the user to look at various portions of the display as the biometric authentication begins (e.g., via a “look here” prompt). During the first type of biometric authentication, the electronic device 700 analyzes a gaze corresponding to the user 704 to obtain characteristics associated with the user's 704 eye or portions of the user's 704 eye, such as iris features, pupil, and / or glint. If the first type of biometric authentication corresponds to face verification, the electronic device 700 instructs the user 704 to rotate their face by a particular angle. In some embodiments, if the first type of biometric authentication corresponds to fingerprint verification, the electronic device 700 instructs the user 704 to move their finger along a particular path and / or touch their finger in a particular manner to a touch-sensitive contact of the device 700 or another device. In some embodiments, the first type of biometric authentication is based on another type of biometric authentication (eg, voice authentication) or is based on a non-biometric authentication (eg, passcode entry).

[0163] 7N , an additional exemplary user interface for authorizing a secure action is displayed on display 700a of device 700. Specifically, FIG. 7N illustrates a secure action related to payment. In some embodiments, the secure action related to payment is implemented in the context of an XR environment, such that device 700 corresponds to a head-mounted display. Electronic device 700 receives user input and, in response, navigates to a displayed e-commerce website via web browser 748 to purchase an item (e.g., shoes). In some embodiments, when electronic device 700 is operating in a first mode (e.g., a standard mode without accessibility features enabled and / or a mode that allows secure action using a default physical input mechanism), electronic device 700 detects selection of affordance 750 to initiate a payment function. For example, because display 700a is a touch-sensitive display, user 704 presses affordance 750 with a finger. In some embodiments, when device 700 is implemented in an XR environment, electronic device 700 detects selection of affordance 750 using a controller communicatively coupled to electronic device 700. Additionally, the electronic device 700 detects selection of the affordance 750 using gaze, for example, by detecting that the user 704 is looking at the affordance 750 for a predetermined period of time and / or receiving one or more gestures. In some embodiments, because the electronic device 700 corresponds to a head-mounted display, the electronic device 700 detects that the user 704 is making one or more gestures to activate the affordance 750, such as an air gesture. When the electronic device 700 detects activation of the affordance 750, the electronic device 700 displays a payment details interface 752. The payment details interface 752 includes details regarding information for completing an order to purchase the item displayed in the associated web browser 748. For example, the payment details interface 752 includes credit card billing information, shipping information, and / or pricing information.

[0164] In some embodiments, the electronic device 700 makes user permission for a payment operation conditional upon determining that the user 704 is authorized to perform the payment operation. For example, the user is determined to be authorized to perform a secure operation based on a gaze criterion related to the user's gaze direction 754. The gaze criterion is met when the user 704 gazes at an interface object, such as the web browser 748 and / or the payment details interface 752. For example, the electronic device 700 optionally displays the web browser 748 and / or the payment details interface 752 in a central portion of the display 700a. In some embodiments, the electronic device 700 determines that the user is not authorized to perform a secure operation when the user is not gazed at the web browser 748 and / or the payment details interface 752 (e.g., the user's eyes are closed or the user is looking at a corner of the display 700a (away from the interfaces 748 and 752)). In some embodiments, the user permission for a secure operation is conditional upon determining that the user 704 has or provides a specific movement of a biometric characteristic. For example, the particular movement may include the user gazing at a particular portion of the web browser 748 (e.g., gazing at affordance 750) or at the payment details interface 752 (e.g., the displayed price), the user providing a particular facial rotation, or the user moving a finger along a particular path.

[0165] Once the payment function authorization is enabled, the user proceeds to authorize the payment function. Specifically, the electronic device 700 modifies the appearance of one or more interface objects and / or displays one or more additional interface objects to indicate to the user that the user can request to perform a payment action. Indication 756 includes the instruction “Double-click to authorize,” which may be displayed adjacent to a hardware button 758. In some embodiments, if the electronic device 700 is implemented in an XR environment, the electronic device 700 displays indications such as instructions for performing one or more air gestures (e.g., instructions for moving one or more hands or fingers in a specific motion) and / or instructions for performing one or more inputs on an XR-compatible controller.

[0166] In some embodiments, upon receiving a request to perform a secure operation, the electronic device 700 determines whether user 704 meets the user authentication criteria for performing the payment operation, as discussed with respect to Figures 7F to 7H. If the user does not meet the user authentication criteria, user authorization for the payment operation is invalidated. Therefore, the user authentication process is performed, as discussed with respect to Figures 7F to 7H. If the user does not meet the user authentication criteria, the electronic device 700 initiates the payment transaction and provides feedback to the user regarding the payment transaction.

[0167] Additional information regarding Figures 7A to 7N is provided below with reference to the method 800 described with respect to Figures 7A to 7N.

[0168] Figure 8 is a flowchart of exemplary method 800 for facilitating user consent for secure operation, according to several embodiments. In some embodiments, method 800 is executed in a computer system (e.g., computer system 101 in Figure 1 (e.g., a smartphone, smartwatch, tablet, and / or wearable device)) which includes display generating components (e.g., display generating components 120 in Figures 1, 3, and 4 (e.g., display controller, touch-sensitive display system, display (e.g., embedded and / or connected), 3D display, transparent display, projector, and / or head-up display)) (e.g., head-up display, display, touchscreen, projector, etc.) and / or one or more input devices. In some embodiments, method 800 is governed by a plurality of instructions, which are stored in a non-temporary (or temporary) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 in Figure 1). Some operations of method 800 are optionally combined and / or the order of some operations is optionally changed.

[0169] As will be described later, Method 800 provides an intuitive method for obtaining user consent for secure operation. This method reduces the cognitive burden on the user while performing secure operation, thereby creating a more efficient human-machine interface. In the case of battery-powered computing devices, streamlining user consent for secure operation to make it faster and more efficient saves power and extends the time between battery charges.

[0170] Detect (802) a change in a user's (e.g., user 704) current viewpoint (e.g., from a first viewpoint to a second viewpoint different from the first viewpoint) via one or more input devices (e.g., 125 and / or 150) (e.g., receiving a request to autofill input fields (e.g., with a password and / or email information) and / or detecting activation of a payment affordance) while a three-dimensional environment (e.g., 730) including virtual user interface objects (e.g., 702, 712, 736, 738, 748 and / or 752) including information regarding secure operations (e.g., 706, 708, 710, 716a, 716b, 744a, and / or 744b) is visible via a display generation component (e.g., 120 and / or 700a).

[0171] In response to detecting a change in a user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), enabling user authorization (804) for a secure action using the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., enabling an autofill authorization option to automatically fill in fields in a second user interface (e.g., a form user interface) and / or enabling an activatable purchase authorization option) in accordance with determining that at least a threshold amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are visible from the user's viewpoint (e.g., 718, 746, and / or 754) (e.g., the gaze is directed at the user interface and / or the target of the user's gaze is the user interface) and that the user (e.g., 704) is authorized to perform the secure action). In some embodiments, the computer system (e.g., 700) visually indicates that a viewpoint (e.g., 718, 746, and / or 754) includes a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and thus a particular permission option is enabled (e.g., by not graying out a portion of the second user interface (e.g., 702, 712, 736, 738, 748, and / or 752) or a button (e.g., 710)). In some embodiments, activatable purchase permission options include a single press, a long press, a single press followed by a long press, a press followed by a turn, a turn followed by a press, and / or multiple consecutive presses (e.g., a double press, a triple press).

[0172] If the number of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) visible from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) is less than a threshold amount (e.g., the line of sight (e.g., 718, 746, and / or 754) is not directed towards that user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the object of the user's line of sight In accordance with the determination that it is not 02, 712, 736, 738, 748, and / or 752, user permission for secure actions using virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) is revoked (806) (e.g., disabling the autofill permission option to automatically fill in fields in a second user interface (e.g., a form user interface), and / or disabling the activation of a purchase permission option (e.g., the double-press procedure for a physical button)). In some embodiments, upon determining that a viewpoint (e.g., 718, 746, and / or 754) does not include a user interface (e.g., 702, 712, 736, 738, 748, and / or 752), the computer system (e.g., 700) visually indicates that the viewpoint (e.g., 718, 746, and / or 754) does not include a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and therefore the individual permission options are not enabled (e.g., by graying out parts of the second user interface (e.g., 702, 712, 736, 738, 748, and / or 752), such as a button (e.g., 710). Enabling user permission for secure actions when individual user interfaces are visible improves security and privacy by ensuring that users have the opportunity to review information related to the secure action before authorizing its progress.

[0173] In some embodiments, an input to authorize the performance of a secure operation (e.g., double-clicking a hardware button (e.g., 722 and / or 758) and / or activating a particular affordance) is received from a user (e.g., 704). In some embodiments, the authorizing input includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a rotation followed by a press, or multiple presses in succession (e.g., a double press, a triple press). In some embodiments, in response to receiving input authorizing performance of a secure operation, performance of the secure operation is initiated (e.g., completing fields in a second user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., a form user interface) and / or making a payment) pursuant to determining that one or more sets of execution criteria are met (e.g., the gaze (e.g., 718, 746, and / or 754) is directed toward a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the user's gaze (e.g., 718, 746, and / or 754) is directed toward a user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). Initiate a new transaction), where the set of one or more execution criteria includes a visibility criterion, and the visibility criterion is met when at least a threshold amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., the threshold amount corresponds to any portion of the interface objects (e.g., 702, 712, 736, 738, 748, and / or 752), a non-zero amount of the interface objects (e.g., 702, 712, 736, 738, 748, and / or 752), or the entirety of the interface objects (e.g., 702, 712, 736, 738, 748, and / or 752)) is visible from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754).In some embodiments, the execution of a secure action is not initiated based on the determination that one or more sets of execution criteria are not met (for example, that the user's gaze (e.g., 718, 746, and / or 754) is not directed towards a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the target of the user's gaze (e.g., 718, 746, and / or 754) is not a user interface (e.g., 702, 712, 736, 738, 748, and / or 752)) (e.g., that the user refrains from filling in fields (e.g., 706 and / or 708) of a second user interface (e.g., a form user interface) and / or refrains from initiating a payment transaction). Initiating secure actions when a threshold amount is visible within individual user interfaces improves security and privacy by ensuring that users have the opportunity to review information related to the secure action before authorizing its progress.

[0174] In some embodiments, a set of one or more execution criteria includes user authentication criteria that are met when a user (e.g., 704) is authenticated (e.g., the criteria include positive indicators of continuous iris feature verification from the time the user (e.g., 704) interacted with the device (e.g., 700), first level (e.g., full or high fidelity) iris feature verification, and / or positive indicators of proper passcode authentication). Requiring user authentication in addition to user authorization for secure operation results in improved security / privacy by adding an additional verification layer specific to the user requesting authorization.

[0175] In some embodiments, biometric information is detected from the user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature validation is performed and / or a second level (e.g., tracking eye information (e.g., detected glint information and / or detected pupil information) for continuity) iris feature validation is performed). In some embodiments, detecting biometric information includes receiving input (e.g., one or more images of the user's face) from the user (e.g., 704) in response to a prompt (e.g., 724) to gaze at a specific part (e.g., 726) of the screen (e.g., 700a). In some embodiments, detecting biometric information includes passively detecting biometric information while the user (e.g., 704) is using the device (e.g., 700) (e.g., wearing a head-mounted display). In some embodiments, detecting biometric information includes detecting biometric information while a hardware button (e.g., 722 and / or 758) is activated (e.g., during a single press, long press, long press after a single press, rotation after a press, press after rotation, or during multiple consecutive presses (e.g., double press, triple press)), or in response to the activation of a hardware button (e.g., 722 and / or 758). In some examples, the biometric information is compared with registered biometric information (e.g., comparing an acquired iris scan with a stored iris pattern and / or comparing the current eye information with eye information from a previous frame), and the determination of whether the user authentication criteria are met is based on the comparison of the biometric information with the registered biometric information (e.g., if the comparison meets a sufficient similarity threshold (e.g., 95% similarity), the user authentication criteria are met). In some embodiments, a first level (e.g., full or high fidelity) iris feature validation is performed, and a second level (e.g., tracking eye information for continuity) iris feature validation is performed at a later time (e.g., one hour later) to link the first level of iris feature validation to the execution criteria for initiating the execution of secure operation.Requiring biometric authentication in addition to user authorization for secure operation provides improved security / privacy by adding a biometric verification layer specific to the user requesting authorization.

[0176] In some embodiments, the determination that the user authentication criteria have been met occurs in response to receiving input that allows the secure operation to be performed (e.g., in response to activation of a hardware button (e.g., 722 and / or 758) (e.g., single press, long press, single press followed by long press, single press followed by rotation, rotation followed by press, multiple consecutive presses (e.g., double press, triple press)), verification that the user authentication criteria have been met). Requiring user authentication at the time of a user request to perform a secure operation provides improved security / privacy by verifying the identity of the user requesting the operation.

[0177] In some embodiments, determining whether user authentication criteria are met includes retrieving stored authentication information indicative of one or more previously performed biometric authentications (e.g., retrieving continuity results indicative of a positive result (e.g., iris feature continuity and / or facial continuity is maintained) or a negative result (e.g., iris feature continuity and / or facial continuity is not maintained)). Utilizing stored authentication information in addition to user permissions for secure operations creates an efficient and uninterrupted user experience for biometric authentication, encourages users to use biometric authentication, and therefore provides improved security / privacy by providing a more secure experience.

[0178] In some embodiments, the user authentication criterion is to perform a first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed), and after performing the first type of biometric authentication (e.g., after successfully performing the first level (e.g., full or high fidelity) iris feature verification), a second type of biometric authentication (e.g., a second level (e.g., tracking eye information for continuity) iris feature verification is performed), and to receive an input containing first individual biometric information from the user (e.g., 704) (e.g., obtaining eye information from the current frame), and to process the received input into a second individual Determining whether a second type of biometric authentication is performed repeatedly, including comparing it with previously received inputs that include biometric information (e.g., comparing eye information from the current frame with eye information from a previous frame), and performing it every two frames (e.g., every three frames, every four frames, every five frames, every six frames, every seven frames, every eight frames, every nine frames, or every five or one-tenth frame). Obtaining multiple sensor measurements at multiple intermediate time points between the first and second time points improves security / privacy by verifying that the same user has been using the device since the initial authentication.

[0179] In some embodiments, performing the first type of biometric authentication includes detecting the presence of a first type of biometric feature (e.g., first level (e.g., full or high fidelity) verification based on eye features, or first level (e.g., full or high fidelity) verification based on facial features), and performing the second type of biometric authentication includes detecting a second biometric feature type (e.g., second level verification based on eye features (e.g., tracking eye information for continuity) or second level verification based on facial features (tracking eyebrows, eyelids, forehead, and / or nose for continuity)), where the first biometric feature type and the second biometric feature type correspond to the same biometric feature type (e.g., first level verification and / or second level verification based on eye features, or first level verification and / or second level verification based on facial features). For secure operation, continuously verifying user presence based on the same biometric characteristics as the initial authentication, in addition to user authorization, creates an efficient and uninterrupted user experience for biometric authentication, encourages users to use biometric authentication, and therefore provides security / privacy improvements by providing a more secure experience.

[0180] In some embodiments, the user authentication criteria is met when the user (e.g., 704) is authenticated based on an eye biometric (e.g., detecting a first pattern based on a plurality of features corresponding to the user's (e.g., 704) eye (e.g., a pattern formed based on eye fibers, pits, grooves, coronas, and / or rings)). In some embodiments, the detected first pattern is compared to a reference pattern, where the reference pattern is based on a plurality of features corresponding to the user's eye (e.g., a stored pattern based on full or high-fidelity iris feature verification, or a stored pattern based on partial iris feature verification for continuity). Requiring eye biometric authentication in addition to user authorization for secure operation provides improved security / privacy by adding a biometric verification layer specific to the user requesting authorization.

[0181] In some embodiments, the user authentication criterion is met when a passcode input matches a registered passcode (e.g., receiving multiple digit inputs (e.g., 0-9) and comparing them to a previously registered passcode (e.g., a passcode containing digits 0-9)). Requiring passcode authentication in addition to user authorization for secure operation results in improved security / privacy by providing an additional method for verifying the user's identity.

[0182] In some embodiments, following a determination that user authentication criteria are not met (e.g., the device cannot obtain a sufficient iris feature scan, the result of a first-level iris feature verification indicates that the iris feature scan does not match a reference scan, or the user refuses to provide iris feature verification), a prompt (e.g., 728) (e.g., a button that the user activates to invoke a passcode input interface) is provided (e.g., displayed via a display generating component (e.g., 120 and / or 700a)). In some embodiments, the prompt (e.g., 704) includes a passcode input portion and an affordance for refusing passcode input. Requiring passcode authentication after biometric authentication failure results in improved security / privacy by providing a backup option for verifying the user's identity.

[0183] In some embodiments, user authorization for a secure operation (e.g., initiating a payment operation and / or initiating an autofill operation) using a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) includes user activation (e.g., a single press, a long press, a single press followed by a long press, a rotation and press of the hardware user interface element, a rotation and press, two, three, or more consecutive presses) of a hardware user interface element (e.g., 722 and / or 758). Utilizing a hardware element for authorization provides improved security / privacy by reducing the likelihood of erroneous authorization and / or preventing unauthorized input from malicious software.

[0184] In some embodiments, the secure action is a payment (e.g., providing payment information to a third party service to purchase an item, such as providing credit card information to an e-commerce website to purchase shoes). Enabling user permission for the payment action when a separate user interface is visible provides improved security / privacy by ensuring that the user has an opportunity to review information related to the secure action before allowing the secure action to proceed.

[0185] In some embodiments, the secure operation includes autofilling user credentials (e.g., providing login information (e.g., 706 and / or 708) (e.g., username and / or password) to access secure information (e.g., a banking website, a stock trading application, and / or user profile information)). Enabling user permission for the autofill operation when a separate user interface is visible provides improved security / privacy by ensuring that the user has an opportunity to review information related to the secure operation before allowing the secure operation to proceed.

[0186] In some embodiments, in response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), the amount of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) visible to the user (e.g., 704) changes (e.g., as a result of a viewpoint shift (e.g., looking left or right), the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are partially (e.g., as shown in Figures 7I and 7K) or completely removed from the field of view, or, as a result of a viewpoint shift, the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) become visible after being completely removed from the field of view). Enabling user permission for secure behavior based on changes in the user's viewpoint results in improved security / privacy by verifying whether the user is still viewing details related to secure behavior.

[0187] In some embodiments, the amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) that are visible varies based on occlusion by physical (e.g., real) objects. In some embodiments, the visible amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) from a user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) is detected, and the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes physical objects represented in the three-dimensional environment. In some embodiments, a user (e.g., 704) views a physical object (e.g., a computer monitor) through an additional display, or a user (e.g., 704) views a representation of a physical object reproduced on an opaque display, where the physical object is closer to the user (e.g., 704) than the representations of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752). In some embodiments, the reduction in the visibility is detected in response to detecting a change in a user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), where the detected change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes detecting physical objects appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) with respect to the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754). In some embodiments, the reduction in the visibility is based on the detected physical objects appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., fewer virtual interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are displayed than were displayed before the viewpoint change).In some embodiments, a physical object may move in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., a physical balloon floats in front of the user's (e.g., 704) central field of view). In some embodiments, the modified user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) may include a physical object close to the user (e.g., 704) that obstructs the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., the user (e.g., 704) turns towards a tree one foot in front of the user (e.g., 704)). Where it is stated that a physical object is obscuring a virtual object (e.g., 702, 712, 736, 738, 748, and / or 752), this means, optionally, that a device (e.g., 700) effectively and virtually obscures at least a portion of a virtual object (e.g., 702, 712, 736, 738, 748, and / or 752) that is at a simulated distance further from the user's (e.g., 718, 746, and / or 754) viewpoint (e.g., 718, 746, and / or 754) than the physical object, while being within the same line of sight as the physical object. Enabling user permission for secure behavior based on the fact that the details of secure behavior are obscured by a physical object results in improved security / privacy by verifying whether the user is still viewing the details of the secure behavior.

[0188] In some embodiments, the amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) that are visible changes based on occlusion by the virtual object (e.g., 732). In some embodiments, a reduction in visibility is detected (e.g., a portion of the interface becoming invisible or the entire interface becoming invisible) in response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754). In some embodiments, the detected change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes detecting a virtual object (e.g., 732) appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) with respect to the user's (e.g., 704) viewpoint (e.g., as shown in FIGS. 7J and 7K). In some embodiments, the reduction in visibility is based on a detected virtual object (e.g., 732) appearing in front of a virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752). Enabling user permission for a secure action based on the details of the secure action being occluded by a virtual object provides improved security / privacy by verifying that the user is still viewing the details related to the secure action.

[0189] In some embodiments, the visibility of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) moving out of the user's (e.g., 704) field of view (e.g., as shown in Figure 7I). In some embodiments, in response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) becomes undetectable within the user's (e.g., 704) current viewpoint (e.g., 718, 746, and / or 754) (e.g., the user (e.g., 704) looks away from the interface (e.g., 702, 712, 736, 738, 748, and / or 752), causing the interface to be positioned "behind" the user in the environment). Enabling user permission for secure behavior based on the fact that the details of secure behavior are outside the user's field of view results in improved security / privacy by verifying whether the user is still viewing the details of the secure behavior.

[0190] In some embodiments, the visibility of a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on the virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) moving beyond a threshold distance from the center of the user's (e.g., 704) field of view (e.g., 718, 746, and / or 754) (e.g., the user (e.g., 704) looks away from the interface (e.g., 702, 712, 736, 738, 748, and / or 752) so that only a portion of the interface is visible (e.g., visible toward the side of the user's (e.g., 704) field of view), or the interface (e.g., 702, 712, 736, 738, 748, and / or 752) is moved by the user (e.g., 704) or another user). Enabling user permission for a secure action based on the details of the secure action being too far from the center of the user's field of view provides improved security / privacy by verifying that the user is still looking at the details related to the secure action.

[0191] In some embodiments, the visibility of a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on occlusion by a physical object. In some embodiments, movement of a physical object is detected (e.g., a balloon floating in front of a user (e.g., 704)), and the detected movement includes movement of a physical object in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) relative to the user's viewpoint (e.g., 718, 746, and / or 754) (e.g., a balloon floating in front of a user (e.g., 704) on an additional display (e.g., a display including a transparent portion through which the user (e.g., 704) views the actual surrounding physical environment (e.g., 730)). The movement of the balloon may cause portions of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) to disappear, or on an opaque display (e.g., a display coupled to a sensor and / or camera through which a user (e.g., 704) views an on-display representation (e.g., 730) of the surrounding physical environment), the movement of the balloon may cause a representation of the balloon to be displayed in a manner that obscures portions of the virtual interface objects (e.g., 702, 712, 736, 738, 748, and / or 752). Enabling user permission for a secure action based on the details of the secure action being occluded by a physical object may provide improved security / privacy by verifying that the user is still viewing details related to the secure action.

[0192] In some embodiments, determining that a user (e.g., 704) is authorized to perform a secure operation includes determining whether a gaze criterion is met when the user (e.g., 704) gazes at a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752). In some embodiments, the secure operation is initiated pursuant to a determination that the user (e.g., 704) gazes at a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) while a request to perform the secure operation is received (e.g., the user (e.g., 704) gazes at selected login information (e.g., 716a, 716b, 744a, and / or 744b) or the user (e.g., 704) gazes at payment information). Requiring the user to view details regarding the secure operation when requesting permission provides improved security / privacy by ensuring that the user has an opportunity to read details regarding the operation.

[0193] In some embodiments, determining that a user (e.g., 704) is authorized to perform a secure operation includes determining whether an eye criterion is met when the user's (e.g., 704) eyes are open. In some embodiments, the secure operation is initiated pursuant to a determination that both of the user's (e.g., 704) eyes are open or at least one of the user's (e.g., 704) eyes is open while a request to perform the secure operation is received. Requiring the user's eyes to be open to provide permission for a secure operation provides improved security / privacy by ensuring that the user has an opportunity to read details about the operation.

[0194] In some embodiments, a user (e.g., 704) may request performance of a secure operation (e.g., an indicator) pursuant to determining that at least a threshold amount of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are visible from the user's (e.g., 704) perspective (e.g., a line of sight (e.g., 718, 746, and / or 754) is directed toward the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the user's line of sight (e.g., 718, 746, and / or 754) is directed toward the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). The appearance of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) may be modified (e.g., as shown in FIG. 7D ) to indicate to the user (e.g., 704) that the secure operation has been performed (e.g., via 720, 734, and / or 756) (e.g., the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) may not be grayed out, a button may be marked as active, or text (e.g., "Please proceed with the request") may be provided in the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) indicating that the user (e.g., 704) may request the secure operation to be performed).In some embodiments, the amount of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) visible from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) is less than a threshold amount (e.g., the line of sight (e.g., 718, 746, and / or 754) is not directed towards the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the object of the user's line of sight (e.g., 718, 746, and / or 754) is not the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)), and the user (e.g., 704) is determined to be less than a threshold amount (e.g., the line of sight (e.g., 718, 746, and / or 754) is not directed towards the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)), and the user (e.g., 704) To indicate to the user (e.g., 704) that they cannot request the execution of a secure action, the appearance of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) is modified (e.g., as shown in Figure 7C) (e.g., the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is grayed out, buttons are marked as inactive, and / or text indicating that the user (e.g., 704) cannot request the execution of a secure action (e.g., "See to request") is provided to the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). Providing a visual indication that user authorization can be provided for a secure action provides improved visual feedback by informing the user that they can initiate a secure action.

[0195] In some embodiments, the three-dimensional environment (e.g., 730) includes a virtual object (e.g., a virtual representation of a shoe) associated with the secure action, and includes attaching a virtual user interface object to the virtual object associated with the secure action (e.g., attaching a virtual representation of a sheet containing payment information for the shoe to the shoe), such that when the virtual object is moved, the attached virtual user interface object moves with the virtual object (e.g., when the shoe is moved (e.g., placing the shoe on a table), the sheet remains attached to the shoe and moves with the shoe (e.g., the sheet remains attached to the shoe while the shoe is resting on the table)). Attaching the corresponding object to an interface having details about the secure action provides improved visual feedback by informing the user that the secure action is associated with the corresponding object.

[0196] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchanged, substituted, and / or added between these methods. For example, the determination of whether to enable user permission for secure operation in method 800 is optionally used as part of method 900 to determine whether to enable user permission. For the sake of brevity, those details will not be repeated here.

[0197] Additional description regarding FIGS. 7A-7N is provided below with reference to the method 900 described with respect to FIGS. 7A-7N.

[0198] 9 is a flow diagram of an exemplary method 900 of continuity of authentication for secure operation, according to some embodiments. In some embodiments, method 900 is performed on a computer system (e.g., computer system 101 of FIG. 1 (e.g., a smartphone, a smartwatch, a tablet, and / or a wearable device)) that includes a display generation component (e.g., display generation component 120 of FIGS. 1, 3, and 4 (e.g., a display controller, a touch-sensitive display system, a display (e.g., embedded and / or connected), a 3D display, a see-through display, a projector, and / or a head-up display)), one or more input devices, and a biometric sensor (e.g., a facial recognition device, a fingerprint recognition device, and / or an eye-tracking device). In some embodiments, method 900 is governed by a plurality of instructions that are stored on a non-transitory (or transitory) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 of FIG. 1). Some operations of method 900 are combined as optional choices, and / or the order of some operations is changed as optional choices.

[0199] As described below, method 900 provides an intuitive way to facilitate authentication continuity for secure operations. This method reduces the cognitive burden on the user while performing secure operations, thereby creating a more efficient human-machine interface. For battery-powered computing devices, facilitating authentication continuity for secure operations to be faster and more efficient conserves power and extends the time between battery charges.

[0200] At a first time, a user (e.g., 704) of the device biometrically authenticates (902) using a biometric sensor to perform a first type of biometric authentication (e.g., a first level (e.g., full or high-fidelity) iris feature verification is performed and / or a first level (e.g., full or high-fidelity) fingerprint verification is performed). At a second time after the first time, a request to perform a secure operation is received (e.g., a request to autofill (904) an input field (e.g., 706 and / or 708) (e.g., autofill a password and / or email information) and / or detect activation of a payment affordance).

[0201] In some embodiments, upon receiving a request to perform a secure action (906), the device performs a secure action (908) (for example, without performing a first type of biometric authentication after receiving a request to perform a secure action (for example, no first level (e.g., full or high fidelity) iris feature verification is performed between the request for the secure action and the performance of the secure action, and / or no first level (e.g., full or high fidelity) fingerprint verification is performed between the request for the secure action and the performance of the secure action), using autofill to enter predetermined values ​​into fields (e.g., 706 and / or 708) of the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or initiate a payment transaction). In some embodiments, the criterion is based on sensor measurements taken at multiple intermediate times between a first time and a second time, including sensor measurements taken at a first intermediate time and sensor measurements taken at a second intermediate time, which detect that the same user was using the device (e.g., 700) between the first time and the second time (e.g., the eye-tracking continuity policy has not had a “false” result since the user (e.g., 704) previously performed a first level (e.g., full or high fidelity) iris feature validation, and / or the fingerprint continuity policy has not had a “false” result since the user (e.g., 704) previously performed a first level (e.g., full or high fidelity) fingerprint validation).

[0202] In some embodiments, the secure operation is cancelled (910) upon a determination that the user (e.g., 704) of the device (e.g., 700) did not meet the respective criteria between the first and second times (e.g., an eye-tracking continuity policy evaluates to "false" (e.g., gaze was undetectable for a period of time, the user (e.g., 704) was previously authenticated by passcode only, and / or the user (e.g., 704) was previously authenticated by fingerprint only), and / or a fingerprint tracking continuity policy evaluates to "false"). Ensuring that biometric continuity conditions are met when the user requests the secure operation to be performed provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0203] In some embodiments, in response to receiving a request to perform a secure operation, following a determination that a user (e.g., 704) of the device did not meet the respective criteria between the first and second times (e.g., an eye tracking continuity policy evaluates to “false” (e.g., eyes are undetectable for a period of time, the user (e.g., 704) was previously authenticated by passcode only, and / or the user (e.g., 704) was previously authenticated by fingerprint only), and / or a fingerprint tracking continuity policy evaluates to “false”), the user (e.g., 704) of the device is biometrically authenticated using a biometric sensor to perform a first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, and / or a first level (e.g., full or high fidelity) fingerprint verification is performed). Authenticating a user when a biometric continuity condition is not met provides improved security / privacy by verifying that an authorized user is interacting with the device.

[0204] In some embodiments, the first type of biometric authentication is based on the biometric features of the eye (e.g., the entire eye or a part of the eye such as iris features), and the second type of biometric authentication is based on the biometric features of the eye (e.g., the entire eye or a part of the eye such as iris features). Ensuring that biometric continuity conditions are met based on eye features provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0205] In some embodiments, the first type of biometric authentication is based on facial biometric features (e.g., eyebrows, eyelids, skin around the eyes, forehead, nose), and the second type of biometric authentication is based on facial biometric features (e.g., eyebrows, eyelids, skin around the eyes, forehead, nose). Ensuring that biometric continuity conditions are met based on facial features provides improved security and privacy by verifying that the same user has been using the device since the initial authentication.

[0206] In some embodiments, a user of the device (e.g., 704) meeting each criterion involves repeatedly performing a second type of biometric authentication (e.g., iris feature verification of a second level (e.g., tracking eye information for continuity)) using one or more sensors of a computer system (e.g., 101 and / or 700), the second type of biometric authentication includes receiving an input from the user (e.g., 704) containing a first individual biometric (e.g., obtaining eye information from the current frame) and comparing the received input with a previously received input containing a second individual biometric (e.g., comparing eye information from the current frame with eye information from a previous frame). Obtaining multiple sensor measurements at multiple intermediate time points between a first time point and a second time point provides an improvement in security / privacy by verifying that the same user has been using the device since the initial authentication. In some embodiments, the one or more sensors used to perform the second type of biometric authentication include a biometric sensor used to perform the first type of biometric authentication (e.g., the one or more sensors include a biometric sensor used to acquire a new biometric authentication). Taking multiple sensor readings at multiple intermediate times between the first and second times provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0207] In some embodiments, the first type of biometric authentication is based on a first biometric characteristic (e.g., eye and / or iris characteristics of the user (e.g., 704)), and the second type of biometric authentication is based on a second type of biometric characteristic (e.g., continuous wear of the computer system and / or a fingerprint of the user (e.g., 704)), where the first biometric characteristic is different from the second biometric characteristic (e.g., iris feature verification is performed for initial authentication and facial features are monitored for continuity). In some embodiments, at a first time, a user (e.g., 704) of the device is biometrically authenticated based on a first biometric characteristic of the user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints). In some embodiments, a second biometric feature of the user (e.g., 704) is periodically detected (e.g., detecting the user's (e.g., 704) eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and / or cheeks), and each criterion is based on the periodic detection of the second biometric feature of the user (e.g., 704) (e.g., the continuity criterion depends on detected details related to the presence of the eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and / or cheeks). Utilizing a separate biometric feature for initial authentication and a different biometric feature for biometric continuity verification provides improved security / privacy by improving authentication when the feature used for initial authentication is unavailable for continuity verification.

[0208] In some embodiments, the first type of biometric authentication is based on a first biometric feature (e.g., eye and / or iris features of a user (e.g., 704), such that iris feature verification is performed for initial authentication), and the second type of biometric authentication is based on the first type of biometric feature (e.g., eye and / or iris features of a user (e.g., 704), such that iris feature verification is performed for continuity). In some embodiments, at a first time, a user (e.g., 704) of a device is biometrically authenticated based on a first biometric feature of the user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints). In some embodiments, a first biometric feature of a user (e.g., 704) is periodically detected (e.g., detecting details related to the presence of eyes, iris features, pupils, and / or eye glints), and each criterion is based on the periodic detection of the first biometric feature of the user (e.g., 704) (e.g., the continuity criterion depends on detected details related to the presence of eyes, iris features, pupils, and / or eye glints). Utilizing separate biometric features for both initial authentication and biometric continuity verification provides improved security / privacy by reducing the sensor types required for authentication.

[0209] In some embodiments, biometrically authenticating a user (e.g., 704) of the device at a first time includes obtaining sensor measurements at a plurality of intermediate times between the first time and the second time (e.g., detecting details related to the presence of eyes, iris features, pupils, and / or eye glints) based on a first biometric feature of the user (e.g., 704) (e.g., a first level (e.g., full or high-fidelity) iris feature verification is performed, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints), and the obtained sensor measurements include at least one sensor measurement indicating that the first biometric feature of the user (e.g., 704) was not detected at a respective intermediate time among the plurality of intermediate times (e.g., the sensor measurement indicates the absence of eyes, iris features, pupils, and / or eye glints (e.g., based on the user's (e.g., 704's) eyes being closed)). Utilizing a separate biometric feature for initial authentication that is hidden from view when biometric continuity verification is based on another feature provides security / privacy improvements by ensuring that continuity verification occurs despite the feature being hidden.

[0210] In some embodiments, biometrically authenticating a user (e.g., 704) of the device at a first time includes, based on a first biometric characteristic of the user (e.g., 704) (e.g., a first level (e.g., full or high-fidelity) iris feature verification is performed, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints), and periodically obtaining sensor measurements of a second biometric characteristic of the user (e.g., 704) at a plurality of intermediate times between the first time and the second time (e.g., detecting the user's (e.g., 704's) eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and / or cheeks), wherein the obtained sensor measurements indicate that the first biometric characteristic of the user (e.g., 704) was not detected at each of the intermediate times. The first biometric feature may include at least one sensor measurement (e.g., the sensor measurement does not indicate the presence of eyes, iris features, pupils, and / or eye glints (e.g., based on the user's (e.g., 704) eyes being closed)), and the acquired sensor measurements may include at least one sensor measurement indicating that a second biometric feature of the user (e.g., 704) was detected at the respective intermediate time (e.g., the sensor measurement indicates the presence of the user's (e.g., 704) eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and cheeks, while the presence of eyes, iris features, pupils, and / or eye glints is not detected), and the first biometric feature is different from the second biometric feature (e.g., iris feature verification is performed for initial authentication, but facial features are monitored for continuity). Utilizing a separate biometric feature for initial authentication that is hidden from view when biometric continuity verification occurs based on another feature that is visible creates an efficient and uninterrupted user experience for biometric authentication, encourages users to use biometric authentication, and therefore provides improved security / privacy by providing a more secure experience.

[0211] In some embodiments, each criterion (e.g., continuity criterion) between the first and second times includes a continuity criterion that is met when a biometric feature (e.g., an eye of the user (e.g., 704) and / or a fingerprint of the user (e.g., 704)) is repeatedly (e.g., continuously or periodically) detected between the first and second times (e.g., without determining that the biometric feature has not been detected between the first and second times). Taking multiple sensor measurements at multiple intermediate times between the first and second times provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0212] In some embodiments, biometrically authenticating a user (e.g., 704) of a device using a biometric sensor to perform a first type of biometric authentication includes providing (e.g., displaying via a display generation component (e.g., 120 and / or 700a)) an authentication guidance user interface (e.g., 724) to the user (e.g., 704) that includes instructions to perform the first type of biometric authentication, the instructions requesting the user (e.g., 704) to provide a specific movement of a biometric feature (e.g., instructions to gaze at a specific portion (e.g., 726) of the authentication guidance user interface, rotate their face (for facial authentication) along a specific path, and / or move a finger along a specific path). Displaying an authentication guidance user interface to facilitate user authentication provides improved security / privacy by reducing the chance of failed authentication attempts.

[0213] In some embodiments, secure operation includes providing payment information (for example, as shown in Figure 7N) (e.g., providing payment information to purchase items and / or services (e.g., providing credit card information to an e-commerce website to purchase shoes)). In some embodiments, an electronic device (e.g., 700) participates in a transaction. For example, upon detecting that input has been received in a hardware user interface element (e.g., 722 and / or 758) and that other criteria have been met, a secure element within the computer system (e.g., 700) releases the payment information so that an application on the computer system (e.g., 700) can access the information (e.g., send the information to a server associated with the application). In some embodiments, the secure element is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms. In some embodiments, the secure element provides (or exposes) payment information (e.g., an account number and / or a transaction-specific dynamic security code). In some embodiments, the secure element provides (or discloses) payment information in response to the device (e.g., 700) receiving authorization such as user authentication (e.g., fingerprint authentication, passcode authentication, detection of a double press of hardware buttons (e.g., 722 and / or 758) by providing authentication credentials to the device (e.g., 700) when the device (e.g., 700) is in an unlocked state, and optionally from the time the device (e.g., 700) is unlocked, while the device (e.g., 700) is continuously on the user's (e.g., 704) wrist. The continuous presence of the device (e.g., 700) on the user's (e.g., 704) wrist is determined by periodically checking whether the device (e.g., 700) is in contact with the user's (e.g., 704) skin).In some embodiments, the secure element provides (or releases) payment information in response to continuity criteria (e.g., a second level (e.g., continuity) of iris feature verification being performed) and / or gaze criteria (e.g., gaze is directed at the user interface and / or the user's gaze is directed at the user interface) being met. Using biometric continuity verification to enable user authorization of payment operations provides improved security / privacy by ensuring that the same user is using the device from the time of initial authentication.

[0214] In some embodiments, the secure operation includes providing access credentials to access an application (e.g., an application associated with a third party service (e.g., providing facial recognition data to access a stock trading application or a password storage application)). In some embodiments, the secure operation includes providing access credentials to access an application (e.g., an application associated with a third party service (e.g., providing facial recognition data to access a stock trading application or a password storage application)). Enabling user permission for application authentication using biometric continuity verification provides improved security / privacy by ensuring the same user is using the device from the time of initial authentication.

[0215] In some embodiments, the secure operation includes autofilling user credentials (e.g., username and / or password information to access secure information (e.g., providing login information (e.g., 706 and / or 708) to access a banking website). Using biometric continuity verification to enable user permission for the autofill operation provides improved security / privacy by ensuring the same user is using the device from the time of initial authentication.

[0216] In some embodiments, receiving a request to perform a secure operation includes detecting a physical input to activate a hardware button (e.g., 722 and / or 758) (e.g., a user (e.g., 704) engaging in a double press of a hardware button to initiate a payment operation and / or a user (e.g., 704) engaging in a double click of a hardware button to initiate an autofill operation). In some embodiments, the hardware button (e.g., 722 and / or 758), when activated using various inputs, performs various respective functions, such as navigating to a home user interface, powering off the device, displaying a system user interface for powering off the device, and / or activating a voice assistant. In some embodiments, the physical input includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a rotation followed by a press, and / or multiple presses in succession (e.g., a double press, a triple press). Utilizing a hardware element for authorization provides improved security / privacy by reducing the likelihood of erroneous authorization and / or preventing unauthorized input from malicious software.

[0217] In some embodiments, a system user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is provided to a user (e.g., 704) (e.g., displayed via a display generation component (e.g., 120 and / or 700a)), the system user interface including an affordance associated with a secure operation (e.g., an affordance labeled “Pay” or an affordance labeled “Autofill”), where receiving a request to perform the secure operation includes detecting activation of an affordance (e.g., 710 and / or 750) associated with the secure operation (e.g., the user (e.g., 704) activates the affordance with gaze, gaze and a hardware button, or a hardware button alone). In some embodiments, the user gazes at the affordance (e.g., 710 and / or 750), and the affordance (e.g., 710 and / or 750) is activated pursuant to a determination that the user has gazed at the affordance for a threshold period of time. In some embodiments, a user (e.g., 704) activates an affordance (e.g., 710 and / or 750) by activating a hardware button (e.g., 722 and / or 758) (e.g., double-pressing a hardware button and / or long-pressing a hardware button). In some embodiments, a user (e.g., 704) activates an affordance by activating a hardware button (e.g., 722 and / or 758) (e.g., double-pressing a hardware button and / or long-pressing a hardware button), and the affordance (e.g., 710 and / or 750) is activated pursuant to a determination that the user (e.g., 704) is gazing at the affordance (e.g., 710 and / or 750) while the user (e.g., 704) is activating the hardware button.In some embodiments, a user (e.g., 704) activates an affordance (e.g., 710 and / or 750) by controlling a secondary device (e.g., a controller) to select the affordance (e.g., 710 and / or 750) (e.g., by moving a cursor over the affordance while activating a hardware button and / or by using a pointing device while activating a hardware button (e.g., 722 and / or 758)). Leveraging affordances within the system user interface for authorization provides improved security / privacy when the device is utilized with one or more alternative input devices.

[0218] In some embodiments, receiving a request to perform a secure operation includes detecting a user activation of an affordance (e.g., 710 and / or 750) within a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., a user (e.g., 704) activates an affordance labeled “Payment Authorization,” or a user (e.g., 704) activates an affordance labeled “Autofill Authorization”) while providing (e.g., displaying via a display generation component (e.g., 120 and / or 700a)) a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) that includes information about the secure operation (e.g., 706, 708, 710, 716a, 716b, 744a, and / or 744b) to a user (e.g., 704). Displaying information about the secure operation when requesting permission provides improved security / privacy by ensuring that the user has an opportunity to review information related to the secure operation before allowing the secure operation to proceed.

[0219] In some embodiments, each criterion includes an eyes-open criterion that is met when one or more eyes (e.g., one and / or both eyes) of the user (e.g., 704) do not remain continuously closed for more than a threshold time period (e.g., non-zero) (e.g., the continuity criterion is not met when the user's (e.g., 704) eyes are closed for more than the threshold time period), and an eyes-open criterion that is not met when one or more eyes (e.g., one and / or both eyes) of the user (e.g., 704) remain continuously closed for more than a threshold time period (e.g., non-zero) (e.g., the continuity criterion is not met when the user's (e.g., 704) eyes are closed for more than the threshold time period). Determining that biometric continuity is not met when the eyes are detected as closed provides improved security / privacy by ensuring informed consent for the operation.

[0220] In some embodiments, each criterion is based on a biometric feature that is available for analysis regardless of whether the user's (e.g., 704) eyes are closed (e.g., regardless of how long the user's (e.g., 704) eyes are closed) (e.g., the continuity criterion may be met even when the user's (e.g., 704) eyes are closed for an extended period of time). Determining that biometric continuity is met when the eyes are detected as closed provides improved security / privacy by improving authentication when features used for initial authentication are unavailable for continuity verification.

[0221] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchanged, substituted, and / or added between these methods. For example, the determination in method 900 of whether a user has met particular (e.g., persistence) criteria is optionally used as part of method 800 to determine whether to validate user permissions. For the sake of brevity, those details will not be repeated here.

[0222] Additional description regarding FIGS. 7A-7N is provided below with reference to the method 1000 described with respect to FIGS. 7A-7N.

[0223] 10 is a flow diagram of an exemplary method 1000 of continuity of authentication for secure operation, according to some embodiments. In some embodiments, method 1000 is performed on a computer system (e.g., computer system 101 of FIG. 1 (e.g., a smartphone, a smartwatch, a tablet, and / or a wearable device)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a display controller, a touch-sensitive display system, a display (e.g., embedded and / or connected), a 3D display, a transparent display, a projector, and / or a head-up display)) (e.g., a head-up display, a display, a touchscreen, a projector, etc.) and / or one or more input devices. In some embodiments, method 1000 is governed by a plurality of instructions that are stored on a non-transitory (or transitory) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 of FIG. 1). Some operations of method 1000 are combined as optional choices, and / or the order of some operations is changed as optional choices.

[0224] As described below, method 1000 provides an intuitive way to authorize secure operations via an accessibility interface. This method reduces the cognitive load on the user while performing secure operations via the accessibility interface, thereby creating a more efficient human-machine interface. For battery-operated computing devices, faster and more efficient authorization of secure operations via the accessibility interface conserves power and extends the time between battery charges.

[0225] A request is received (1002) via one or more input devices (e.g., 125 and / or 150) to display a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) associated with performing a secure operation (e.g., a request to perform an autofill procedure and / or a request to perform a payment transaction). In response to a request to display a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) related to performing a secure operation, display (1004) via the display generation component (e.g., 120 and / or 700a) a first user interface (e.g., 702, 712, 736, 738, 748, and / or 752) that includes respective prompts (e.g., 720, 734, and / or 756) for providing additional input (e.g., login details, account details, and / or information related to a pending purchase) to authorize the device to perform the secure operation (e.g., an interface displayed in response to receiving a request to perform an autofill procedure and / or an interface displayed in response to receiving a request to perform a payment transaction).

[0226] In accordance with a determination that the computer system (e.g., 700) is operating in a first mode (e.g., a standard mode in which accessibility features are not enabled and / or a mode that allows secure operation using a physical input mechanism (e.g., 722 and / or 758)), a separate prompt is provided (1006) (e.g., a first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) prompts the user (e.g., 704) to provide a physical input (e.g., 722 and / or 758) (e.g., a physical input mechanism (e.g., 722 and / or 758) (e.g., prompts (e.g., 720, 734, and / or 756) indicate that if the user (e.g., 704) intends to authorize an autofill procedure, the user (e.g., 704) should provide physical input (e.g., 722 and / or 758)), press, and / or successive presses of buttons (e.g., 722 and / or 758)). (e.g., prompts (e.g., 720, 734, and / or 756) indicate that if the user (e.g., 704) intends to authorize a purchase, the user (e.g., 704) should provide physical input (e.g., 722 and / or 758)).

[0227] Upon determining that the computer system (e.g., 700) is operating in a second mode different from the first mode (e.g., an accessibility mode and / or a mode in which input at a physical input mechanism (e.g., 722 and / or 758) is not required to permit secure operation), the separate prompt (e.g., 720, 734 and / or 756) is a second prompt for displaying a second user interface (e.g., 742) (e.g., the first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) does not prompt the user (e.g., 704) that the user should provide physical input (e.g., 722 and / or 758) if the user (e.g., 704) intends to permit the autofill procedure, and / or the first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) does not prompt the user (e.g., 704) that the user should provide physical input (e.g., 722 and / or 758) if the user (e.g., 704) intends to permit the autofill procedure). (e.g., 704) that the user (e.g., 704) should provide a physical input (e.g., 722 and / or 758) if the user (e.g., 704) intends to authorize the purchase), in which case the second user interface (e.g., 742) enables (1008) the user's authorization of secure operation without using a physical input (e.g., 722 and / or 758) (e.g., enabling authorization of secure operation using a first level (e.g., full or high fidelity) iris feature verification (e.g., first level iris feature verification occurs while the user (e.g., 704) activates an affordance (e.g., 744) and / or first level iris feature verification occurs while the user (e.g., 704) gazes at an affordance (e.g., 744) and / or enabling authorization of secure operation using a first level (e.g., full or high fidelity) fingerprint verification). Allowing secure operation without physical input improves security and privacy when the device is used with one or more alternative input devices.

[0228] In some embodiments, a first separate affordance (e.g., 740 and / or 750) is provided within a second prompt (e.g., 736 and / or 738) (e.g., the affordance includes an indication (e.g., 734) that the authorization of a secure action is performed using a secondary input type different from the primary input type (e.g., "Autofill" is displayed next to "Confirm with AssistiveTouch" or an affordance for "Pay" is provided next to "Confirm with AssistiveTouch" (e.g., displayed via a display generation component (e.g., 120 and / or 700a))), then a user selection of the affordance (e.g., 740 and / or 750) is received (e.g., user (For example, 704) selects an "autofill" affordance (e.g., 740) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input, or a user (e.g., 704) selects a "pay" affordance (e.g., 750) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input). In some embodiments, the user (e.g., 704) gazes at the affordances (e.g., 740 and / or 750), and the affordances (e.g., 740 and / or 750) are activated according to the determination that the user (e.g., 704) gazed at the affordances (e.g., 740 and / or 750) over a threshold period.In some embodiments, a user (e.g., 704) activates an affordance (e.g., 740 and / or 750) by controlling an adaptive accessory (e.g., a switch control) to select an affordance (e.g., 740 and / or 750) (e.g., the user (e.g., 704) taps the switch control a first time to initiate the display of a sliding vertical line that slides back and forth across the display (e.g., 700a), the user (e.g., 704) taps the switch control a second time when the vertical line appears over the affordance (e.g., 740 and / or 750), and in response to the second tap, a horizontal line that slides up and down across the display (e.g., 700a) is displayed, and the user (e.g., 704) taps the switch control a third time when the horizontal line appears over the affordance (e.g., 740 and / or 750)). In some embodiments, a user (e.g., 704) controls an adaptive accessory to select an affordance (e.g., 740 and / or 750), and the affordance (e.g., 740 and / or 750) is activated pursuant to a determination that the user (e.g., 704) is gazing at the affordance (e.g., 740 and / or 750) while the user (e.g., 704) is selecting the affordance (e.g., 740 and / or 750). An adaptive accessory is an accessory that allows a user (e.g., 704) to provide input through alternative means compared to a typical usage scenario of the device; one example of an adaptive accessory is a switch control that offers a limited number of selection states (e.g., between 1 and 5) used to select among options offered by the device (e.g., 700).An example of a switch control is a device that can be activated by pressing, pulling, blinking, squeezing, and / or sucking / blowing through a straw, which allows users with cognitive or physical accessibility needs to interact with a device (e.g., 700) when the user (e.g., 704) is unable to easily interact with the device (e.g., 700) using standard input methods (e.g., using a touchscreen, physical buttons / knobs, and / or air gestures).

[0229] In some embodiments, pursuant to receiving a user selection of an affordance, a second user interface (e.g., 742) is displayed, where the second user interface (e.g., 742) includes a representation of multiple functions (e.g., displaying an accessibility menu (e.g., 742) including multiple affordances representing device functions), including a first individual function of the multiple functions that is invoked in response to a secondary input type that is different from the primary input type (e.g., the accessibility menu (e.g., 742) includes options such as “Home,” “Control Center,” “Devices,” “Notifications,” or “Custom”) and a second individual function of the multiple functions (e.g., 744) that is invoked in response to a tertiary input type that is different from the primary and secondary input types (e.g., the accessibility menu (e.g., 742) includes a new option “Allow Autofill” or a new option “Payments”). In some embodiments, the accessibility menu (e.g., 742) includes a grid of affordances and an outline surrounding one of the affordances (e.g., 744), which periodically (e.g., every 2 or 5 seconds) moves the outline to surround another affordance (e.g., 744), allowing the user (e.g., 704) to provide input (e.g., input to a switch control and / or gaze input directed at the surrounded affordance) to select the currently outlined affordance (e.g., 744). Displaying an accessibility interface with alternative input options provides security / privacy improvements when the device is utilized with one or more alternative input devices.

[0230] In some embodiments, a secondary input type simulates a primary input type input (e.g., calling "Home" in response to a swipe up from a button on the screen), and a tertiary input type simulates a primary input type input (e.g., calling "Home" in response to a custom input (e.g., long press) via an accessibility menu (e.g., 742)). Enabling simulated inputs instead of conventional input types provides improved security / privacy when the device is used with one or more alternative input devices.

[0231] In some embodiments, while the computer system (e.g., 700) is operating in a second mode, affordances associated with individual prompts (e.g., 740 and / or 750) are displayed (e.g., the affordance for "autofill" is displayed with the text "Confirm with AssistiveTouch" or the affordance for "payment" is displayed with the text "Confirm with AssistiveTouch"), and in accordance with the receipt of user input associated with the individual prompts (e.g., the user (e.g., 704) selects the affordance for "autofill" (e.g., 740) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input, or the user (e.g., 704) selects the affordance for "payment" (e.g., 750) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input), a second user interface (e.g., 742) is displayed, and the second user - The interface (e.g., 742) includes affordances (e.g., 744) to allow secure actions (e.g., displaying an accessibility menu (e.g., 742) with a new option "Allow Autofill" or a new option "Pay"), and the affordances (e.g., 744) to allow secure actions are invoked depending on a secondary input type different from the primary input type (e.g., the option "Allow Autofill" is invoked depending on a custom input (e.g., long press) via the accessibility menu (e.g., 742) instead of a primary input type (e.g., double-pressing a side button (e.g., 722 and / or 758)), or the option "Pay" is invoked depending on a custom input (e.g., long press) via the accessibility menu (e.g., 742) instead of a primary input type (e.g., double-pressing a side button (e.g., 722 and / or 758)). Displaying an accessibility interface that includes affordances to allow secure actions provides security / privacy improvements when the device is used with one or more alternative input devices.

[0232] In some embodiments, details related to performing a secure operation (e.g., an interface includes details for performing an autofill procedure (e.g., username and password fields) or an interface includes details for performing a payment transaction (e.g., product name, product description, product price)) are displayed within a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) related to performing a secure operation. In some embodiments, while the computer system (e.g., 700) is operating in a second mode (e.g., an accessibility mode and / or a mode in which input with a physical input mechanism (e.g., 722 and / or 758) is not required to permit the secure operation), an affordance (e.g., 740 and / or 750) related to a separate prompt is displayed (e.g., displaying an affordance for "autofill" with the text "confirm with assistive touch" or displaying an affordance for "pay" with the text "confirm with assistive touch").In some embodiments, in response to receiving user input associated with the respective prompt (e.g., a user (e.g., 704) selecting the “Autofill” affordance (e.g., 740) using gaze, gaze plus adaptive accessory input, and / or adaptive accessory input, or a user (e.g., 704) selecting the “Pay” affordance (e.g., 750) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input), and determining that the user (e.g., 704) is gazed at a user interface associated with performing a secure operation while the user input associated with the respective prompt is received (e.g., a user (e.g., 704) selecting the “Autofill” affordance (e.g., 740) while gazed at an interface including details for performing an Autofill procedure, or a user (e.g., 704) selecting the “Pay” affordance (e.g., 750) while gazed at an interface including details for performing a payment transaction), A second user interface (e.g., 742) is displayed, and the second user (e.g., 742) interface includes an affordance (e.g., 744) for allowing the secure action (e.g., displaying an accessibility menu (e.g., 742) including, for example, a new option “Allow Autofill” or a new option “Payment”), and the affordance (e.g., 744) for allowing the secure action is displayed in response to a secondary input type that is different from the primary input type (e.g., the option “Allow Autofill” is invoked in response to a custom input (e.g., a long press) via the accessibility menu (e.g., 742) (instead of the primary input type (e.g., a double press of the side buttons (e.g., 722 and / or 758))), or the option “Payment” is invoked in response to a custom input (e.g., a long press) via the accessibility menu (e.g., 742) instead of the primary input type (e.g., a double press of the side buttons (e.g., 722 and / or 758))).Displaying an accessibility interface in response to a combination of gaze and affordance activation provides security / privacy improvements when the device is utilized with one or more alternative input devices.

[0233] In some embodiments, while a computer system (e.g., 700) is operating in a first mode (e.g., a standard mode without accessibility features enabled and / or a mode that allows secure operation using a physical input mechanism), a user activation of a physical input (e.g., 722 and / or 758) is received (e.g., a user (e.g., 704) performs a predetermined activation pattern of a hardware input affordance, such as double-clicking a hardware button (e.g., 722 and / or 758)) while a first prompt (e.g., 720, 734, and / or 756) is displayed. In some embodiments, the predetermined activation pattern includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a rotation followed by a press, and / or multiple presses in succession (e.g., a double press or a triple press). In some embodiments, in response to receiving a user activation of a physical input (e.g., 722 and / or 758) while a first prompt (e.g., 720, 734, and / or 756) is displayed, a secure operation is permitted (e.g., permitting an autofill procedure in response to activation of a hardware button (e.g., 722 and / or 758) or permitting a payment procedure in response to activation of a hardware button (e.g., 722 and / or 758)) pursuant to determining that a user authentication criterion is met (e.g., the user (e.g., 704) is gazing at the prompt (e.g., 720, 734, and / or 756)). In some embodiments, pursuant to a determination that user authentication criteria are not met (e.g., the user (e.g., 704) does not gaze at a prompt (e.g., 720, 734, and / or 756)), permission for secure operation is withheld (e.g., preventing an autofill procedure in response to a double-click of a hardware button (e.g., 722 and / or 758) or preventing a payment procedure in response to a double-click of a hardware button (e.g., 722 and / or 758)).Utilizing accessible and inaccessible modes of operation provides improved security / privacy when the device is used with one or more alternative input devices.

[0234] In some embodiments, the authentication criteria include criteria that are met when the user (e.g., 704) provides specific biometric movements (e.g., gazing at a specific part of an authentication guidance user interface (e.g., 724), providing facial rotation along a specific path, and / or moving a finger along a specific path). Requiring specific biometric movements while the physical input mechanism is activated results in improved security / privacy by ensuring informed consent for secure operation.

[0235] In some embodiments, input is received from the user (e.g., 704) to allow secure operation (e.g., during standard mode, the user (e.g., 704) performs the activation of a hardware button (e.g., 722 and / or 758)) (e.g., single press, long press, long press after single press, rotation after press, press after rotation, multiple consecutive presses (e.g., double press, triple press), or during accessibility mode, the option "Allow Autofill" is invoked in response to custom input via the accessibility menu (e.g., 742) or the accessibility menu The option “Payment” is invoked in response to custom input via Nu (e.g., 742), and user authentication is performed in response to receiving input to authorize a secure operation (e.g., a first level (e.g., full or high fidelity) iris verification is performed, a second level (e.g., continuity) iris verification is performed, or passcode verification is performed), and the secure operation is authorized according to the determination that the user (e.g., 704) is authenticated (e.g., it determines whether authentication continuity has been true since the last full or high fidelity biometric authentication, performs full or high fidelity biometric authentication, and / or performs passcode authentication). In some embodiments, the secure operation is not authorized according to the determination that the user (e.g., 704) is not authenticated (e.g., the device revokes authorization for the secure operation). Requiring user authentication in addition to user authorization for a secure operation results in improved security / privacy by adding an additional verification layer specific to the user requesting authorization.

[0236] In some embodiments, performing user authentication includes performing a second type of biometric authentication (e.g., performing iris feature verification at a second level (e.g., tracking eye information for continuity)) (e.g., performing authentication every two frames (e.g., each frame includes a still image or set of images acquired from the device's sensors (e.g., biosensors)), every three frames, every four frames, every five frames, every six frames, every seven frames, every eight frames, every nine frames, or every ten frames), the second type of biometric authentication includes receiving an input from the user (e.g., 704) containing a first individual biometric (e.g., acquiring eye information from the current frame), and comparing the received input with a previously received input containing a second individual biometric (e.g., comparing eye information from the current frame with eye information from the previous frame). Utilizing stored authentication information in addition to user authorization for secure operation provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0237] In some embodiments, performing user authentication includes performing a first type of biometric authentication, the first type of biometric authentication includes detecting the presence of a first type of biometric feature (e.g., a first level (e.g., full or high fidelity) verification based on eye features, or a first level (e.g., full or high fidelity) verification based on facial features). Requiring full or high fidelity biometric authentication in addition to user authorization for secure operation provides an improvement in security / privacy by verifying that an authorized user is requesting the secure operation.

[0238] In some embodiments, performing user authentication includes providing a prompt (e.g., 728) requesting a passcode from the user (e.g., 704); receiving a passcode from the user (e.g., 704) in response to the prompt (e.g., 728) (e.g., receiving multiple numeric inputs (e.g., 0-9)); determining that the user (e.g., 704) is authenticated in accordance with a determination that the received passcode matches a stored passcode; and permitting a secure operation in accordance with a determination that the user (e.g., 704) is authenticated (e.g., the user interface is not grayed out, a button is marked as active, or the user interface provides text indicating that the user (e.g., 704) can request performance of a secure operation (e.g., “Proceed with request”)). In some embodiments, in accordance with a determination that the received passcode does not match the stored passcode, a determination is made that the user (e.g., 704) is not authenticated and the secure operation is not permitted. Requiring passcode authentication in addition to user authorization for secure operation provides improved security / privacy by providing an additional method for verifying a user's identity.

[0239] In some embodiments, an input for authorizing a secure operation is detected, and in response to detecting the input for authorizing a secure operation, pursuant to a determination that the input is a physical input for authorizing a secure operation (e.g., 722 and / or 758) (e.g., activating a physical button (e.g., 722 and / or 758) for authorizing an autofill or activating a physical button (e.g., 722 and / or 758) for authorizing a payment), in some embodiments the activation sequence includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a rotation followed by a press, or multiple consecutive presses (e.g., double press, triple press). In some embodiments, authentication information is retrieved, and the authentication information is associated with a recurring second type of biometric authentication (e.g., retrieving continuity information related to a previously performed second level (e.g., tracking eye information for continuity) of iris feature verification). In some embodiments, pursuant to a determination that the input is not a physical input for authorizing a secure operation, a determination is made whether the input is associated with a second user interface (e.g., 742). In some embodiments, secure operation is permitted pursuant to a determination that the user (e.g., 704) is authenticated based on the second type of biometric authentication (e.g., permitting autofill if a second level (e.g., tracking eye information for continuity) of iris feature verification is successful, or permitting payment if a second level (e.g., tracking eye information for continuity) of iris feature verification is successful). In some embodiments, secure operation is not permitted pursuant to a determination that the user (e.g., 704) is not authenticated based on the second type of biometric authentication (e.g., tracked eye information indicates that the user's (e.g., 704's) eyes were closed for a period of time (e.g., 5 seconds, 7 seconds, or 10 seconds) while interacting with the device).

[0240] In some embodiments, if an input associated with the second user interface (e.g., 742) enables user authorization of a secure operation that does not use a physical input (e.g., the detected input is associated with the option “autofill” or the new option “pay”), then a first type of biometric authentication is performed in accordance with a determination that the input is associated with the second user interface (e.g., 742) (e.g., detecting the input via an accessibility menu), and in accordance with a determination that the user (e.g., 704) is authenticated based on the first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed), the secure operation is permitted (e.g., permitting autofill if the first level (e.g., full or high fidelity) iris feature verification is successful, or permitting payment if the first level (e.g., full or high fidelity) iris feature verification is successful). In some embodiments, the secure operation is not permitted in accordance with a determination that the user (e.g., 704) is not authenticated based on the first type of biometric authentication. In some embodiments, following a determination that the input is not an input associated with the second user interface (e.g., 742), a determination is made whether the input is an additional input not associated with a physical input (e.g., 722 and / or 758) for authorizing secure operation. Requiring full or high-fidelity biometric authentication when the device operates in an accessible mode provides improved security / privacy by verifying that an authorized user is requesting secure operation.

[0241] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchanged, substituted, and / or added between these methods. For example, the determination in method 900 that a user has met particular (e.g., persistence) criteria is optionally used to grant secure operation via an accessibility interface as part of method 1000. For the sake of brevity, those details will not be repeated here.

[0242] The foregoing has been described in terms of specific embodiments for purposes of explanation. However, the exemplary discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments were chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular uses contemplated.

[0243] As mentioned above, one aspect of the present technology is the collection and use of data available from various sources to improve authorization for secure operation. This disclosure contemplates that, in some cases, this collected data may include personal information data that uniquely identifies a particular person or that can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, addresses, data or records regarding a user's (e.g., 704) health or fitness level (e.g., vital sign measurements, medication information, exercise information), birth date, or any other identifying or personal information.

[0244] This disclosure recognizes that the use of such personal information data in the present technology can be used to the benefit of the user. For example, the personal information data can be used to permit secure operation. Additionally, other uses of personal information data that benefit the user are contemplated by this disclosure. For example, health and fitness data can be used to provide insight into the user's overall wellness, or can be used as proactive feedback to individuals using the technology in pursuit of wellness goals.

[0245] This disclosure contemplates that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will adhere to robust privacy policies and / or privacy practices. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for maintaining the strict confidentiality of personal information data. Such policies should be easily accessible to users and should be updated as data collection and / or use changes. Personal information from users should be collected for the entity's lawful and legitimate use and should not be shared or sold except for those lawful uses. Furthermore, such collection / sharing should be carried out only after the user's informed consent is obtained. Furthermore, such entities should consider taking all necessary measures to protect and secure access to such personal information data and to ensure that others with access to the personal information data adhere to their privacy policies and procedures. Furthermore, such entities may be able to undergo third-party assessments to demonstrate their adherence to widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific types of personal data collected and / or accessed and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, collection of or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). Meanwhile, health data in other countries may be subject to other regulations and policies and should be addressed accordingly. Therefore, different privacy practices should be maintained in each country with respect to different types of personal data.

[0246] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of allowing secure operations, the present technology may be configured to allow a user to choose to “opt in” or “opt out” of participating in the collection of personal information data during service registration or at any time thereafter. In another example, a user may choose not to provide data for customization of secure operations such as payments and autofill. In yet another example, a user may choose to limit the length of time data is retained or completely prohibit the deployment of customized services to facilitate secure operations such as payments or autofill. In addition to providing “opt-in” and “opt-out” options, the present disclosure contemplates providing notifications regarding access or use of personal information. For example, a user may be notified upon downloading an app that will access the user's personal information data, and then again immediately before the app accesses the user's personal information data.

[0247] Furthermore, it is the intent of this disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risk can be minimized by limiting data collection and deleting data when it is no longer needed. Additionally, where applicable in certain health-related applications, data anonymization can be used to protect user privacy. De-identification can be facilitated by removing certain identifiers (e.g., date of birth, etc.) where appropriate, controlling the amount or specificity of data stored (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0248] Thus, while this disclosure broadly encompasses the use of personal information data to implement one or more various disclosed embodiments, this disclosure also contemplates that the various embodiments may be practiced without requiring access to such personal information data. That is, various embodiments of the present technology are not rendered inoperable by the absence of all or part of such personal information data. For example, authorizing secure operation may be facilitated based on non-personal information data or a minimal amount of personal information, such as content requested by a device associated with the user, other non-personal information available to the service, or publicly available information.

Claims

1. 1. A computer system in communication with one or more input devices and display generating components, comprising: detecting, via the one or more input devices, a change in a user's current viewpoint while a three-dimensional environment is visible via the display generation component, the three-dimensional environment including a virtual user interface object that includes information related to a secure operation; In response to detecting the change in the viewpoint of the user, enabling user authorization of the secure operation using the virtual user interface object in accordance with a determination that less than a threshold amount of the virtual user interface object is occluded by other objects from the user's viewpoint and the user is authorized to perform the secure operation; withdrawing from enabling user permission for the secure operation using the virtual user interface object in accordance with determining that at least the threshold amount of the virtual user interface object is occluded by another object from the user's viewpoint; A method comprising:

2. receiving input from the user to authorize performance of the secure operation; and in response to receiving the input to authorize performance of the secure operation, initiating execution of the secure operation in accordance with a determination that a set of one or more execution criteria is satisfied, the set including a visibility criterion that is satisfied when at least the threshold amount of the virtual user interface object is visible from the viewpoint of the user; aborting initiation of execution of the secure operation in accordance with a determination that the set of one or more execution criteria is not met; and The method of claim 1 , comprising:

3. The method of claim 2 , wherein the set of one or more performance criteria includes a user authentication criterion that is satisfied when the user is authenticated.

4. detecting biometric information from the user; comparing the biometric information with enrolled biometric information, wherein determining whether the user authentication criteria are met is based on the comparison of the biometric information with the enrolled biometric information. The method of claim 3.

5. The method of claim 3 , wherein determining that the user authentication criteria are met occurs in response to receiving the input to authorize performance of the secure operation.

6. 4. The method of claim 3, wherein determining whether the user authentication criteria are met includes retrieving stored authentication information, the stored authentication information being indicative of one or more previously performed biometric authentications.

7. The determination of whether the user authentication standard is met is performing a first type of biometric authentication; and repeatedly performing a second type of biometric authentication after performing the first type of biometric authentication, wherein the second type of biometric authentication comprises: receiving an input from the user including first individual biometric information; comparing the received input to a previously received input including a second individual biometric information; The method of claim 3.

8. performing the first type of biometric authentication includes detecting the presence of a first biometric feature type; performing the second type of biometric authentication includes detecting the presence of a second biometric feature type, and the first biometric feature type and the second biometric feature type correspond to the same biometric feature type; The method of claim 7.

9. The method of claim 3 , wherein the user authentication criterion is met when the user is authenticated based on an eye biometric.

10. The method of claim 3 , wherein the user authentication criteria is met when a passcode input matches a registered passcode.

11. providing a prompt to enable authentication using a passcode in response to a determination that the user authentication criteria is not met. The method of claim 3.

12. The method of claim 1 , wherein the user authorization of the secure operation using the virtual user interface object includes user activation of a hardware user interface element.

13. The method of claim 1 , wherein the secure action is a payment.

14. The method of claim 1 , wherein the secure action includes auto-filling user credentials.

15. varying an amount of the virtual user interface object that is visible to the user in response to detecting the change in the viewpoint of the user; The method of claim 1 , comprising:

16. The method of claim 15 , wherein the amount of the virtual user interface object that is visible varies based on occlusion by a physical object.

17. The method of claim 15 , wherein the amount of the virtual user interface object that is visible varies based on occlusion by a virtual object.

18. The method of claim 1 , wherein the amount of the virtual user interface object that is visible changes based on the virtual user interface object moving out of the user's field of view.

19. The method of claim 1 , wherein the amount of the virtual user interface object that is visible changes based on the virtual user interface object moving beyond a threshold distance from the center of the user's field of view.

20. The method of claim 1 , wherein the amount of the virtual user interface object that is visible varies based on occlusion by a physical object.

21. The method of claim 1 , wherein the determining that the user is authorized to perform the secure operation comprises determining whether a gaze criterion is met when the user gazes at the virtual user interface object.

22. The method of claim 1 , wherein the determining that the user is authorized to perform the secure operation comprises determining whether an eye criterion is met when the user's eyes are open.

23. modifying an appearance of the virtual user interface object to indicate to the user that the user may request performance of the secure operation in accordance with a determination that at least a threshold amount of the virtual user interface object is visible from the viewpoint of the user; modifying the appearance of the virtual user interface object to indicate to the user that the user cannot request performance of the secure operation in accordance with a determination that fewer than the threshold amount of the virtual user interface object is visible from the user's viewpoint; The method of claim 1 , comprising:

24. the three-dimensional environment includes a virtual object associated with the secure operation, the method comprising: pasting the virtual user interface object onto the virtual object associated with the secure operation, wherein moving the virtual object causes the pasted virtual user interface object to move with the virtual object. The method of claim 1.

25. A computer program causing a computer to carry out the method of any one of claims 1 to 24.

26. 1. A computer system comprising: a memory for storing the computer program of claim 25; one or more processors capable of executing the computer program stored in the memory; The computer system is configured to communicate with a display generation component and one or more input devices.

27. 1. A computer system comprising: A computer system comprising means for carrying out the method of any one of claims 1 to 24.

Images