Devices, methods, and graphical user interfaces for enabling secure operation

Abstract

To provide a computer system having improved methods and interfaces for allowing secure operation, making interaction with the computer system more efficient and intuitive for the user. [Solution] While the three-dimensional environment is visible, a change in the user's current viewpoint is detected. Based on the determination that at least a threshold amount of objects are visible from the user's viewpoint, user authorization for secure actions using virtual user interface objects is enabled. At a first time, the user is biometrically authenticated to perform a first type of biometric authentication. At a second time, a request to perform a secure action is received. Based on the determination that the user met the respective criteria between the first and second times, the secure action is performed after the request is received, without performing the first type of biometric authentication.

Description

Cross - reference to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 167767, filed on February 10, 2023, titled "DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR AUTHORIZING A SECURE OPERATION", and U.S. Provisional Application No. 63 / 314900, filed on February 28, 2022, titled "DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR AUTHORIZING A SECURE OPERATION", the entire contents of each of which are hereby incorporated by reference in their entirety for all purposes.

Technical Field

[0002] The present disclosure generally relates to computer systems that communicate with display - generating components, one or more input devices, and (optionally) a biosensor, and include an electronic device that provides virtual reality experiences and mixed reality experiences via a display, but are not limited thereto, for providing computer - generated experiences.

Background Art

[0003] The development of computer systems for augmented reality has advanced significantly in recent years. Exemplary extended reality environments include at least some virtual elements that replace or enhance the physical world. Input devices such as cameras, controllers, joysticks, touch - sensing surfaces, and touch - screen displays for computer systems and other electronic computing devices are used to interact with the virtual / extended reality environment. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements such as buttons and other graphics.

Summary of the Invention

[0004] Methods and interfaces for enabling secure operation in environments containing several virtual elements (e.g., augmented reality, mixed reality, and virtual reality environments) are cumbersome, inefficient, and restrictive. For example, systems that provide solutions for automatically filling in various text fields, systems that facilitate payments for items or other services, and systems that provide access to third-party applications (e.g., social networking, games, business services, etc.) are complex, cumbersome, error-prone, and impose a significant cognitive burden on the user, ruining the experience in the virtual / augmented reality environment. In addition, these methods take more time than necessary, thereby wasting the energy of the computer system. This latter consideration is particularly important in battery-powered devices.

[0005] Therefore, computer systems with improved methods and interfaces for enabling secure operation are needed to make interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces may optionally complement or replace conventional methods for enabling secure operation. Such methods and interfaces reduce the number, extent, and / or types of user input by helping the user understand the connection between the input provided and the device response to that input, thereby generating a more efficient human-machine interface.

[0006] The drawbacks and other problems associated with the user interface of a computer system described above are mitigated or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a wristwatch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a “touchscreen” or “touchscreen display”). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to display-generating components, the output devices include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in memory for performing multiple functions. In some embodiments, the user interacts with the GUI (and / or computer system) through stylus and / or finger touch and gestures on a touch-sensitive surface, the movement of the user's eyes and hands in space relative to the user's body as captured by a camera and other motion sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through interaction optionally include image editing, drawing, presentation, word processing, spreadsheet creation, gameplay, making phone calls, video conferencing, sending emails, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note-taking, and / or digital video playback. The executable instructions for performing those functions optionally include primary computer-readable storage media and / or non-primary computer-readable storage media, or other computer program products configured to be executed by one or more processors.

[0007] Electronic devices with improved methods and interfaces for enabling secure operation are needed. Such methods and interfaces may complement or replace conventional methods for enabling secure operation. Such methods and interfaces reduce the number, extent, and / or type of user input, resulting in a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces conserve power and extend the interval between battery charges. Furthermore, these methods and interfaces help reduce the number of repetitions of user input by automatically filling in or otherwise inputting sensitive information such as passwords or payment information. Such methods and interfaces also reduce processing power and display usage by reducing the amount of time the user spends interacting with the device when enabling secure operation.

[0008] A method is described according to several embodiments. The method is performed in a computer system communicating with one or more input devices and display generation components. The method includes detecting a change in the user's current viewpoint via one or more input devices while a three-dimensional environment, which includes virtual user interface objects containing information about secure operations, is visible via a display generation component; enabling user authorization for secure operations using virtual user interface objects in response to detecting a change in the user's viewpoint, based on a determination that at least a threshold amount of virtual user interface objects is visible from the user's viewpoint and the user is permitted to perform secure operations; and deactivating user authorization for secure operations using virtual user interface objects based on a determination that the amount of virtual user interface objects visible from the user's viewpoint is less than the threshold amount.

[0009] According to some embodiments, a non-temporary computer-readable storage medium is described. This non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices and display generation components, the one or more programs include instructions for detecting a change in the user's current viewpoint via one or more input devices while a three-dimensional environment, which includes virtual user interface objects containing information about secure operations, is visible via a display generation component, and in response to detecting a change in the user's viewpoint, enabling user authorization for secure operations using virtual user interface objects according to a determination that at least a threshold amount of virtual user interface objects is visible from the user's viewpoint and the user is permitted to perform secure operations, and deactivating user authorization for secure operations using virtual user interface objects according to a determination that the amount of virtual user interface objects visible from the user's viewpoint is less than a threshold amount.

[0010] According to several embodiments, a computer system is described. The computer system communicates with one or more input devices and display generation components. The computer system comprises one or more processors and a memory that stores one or more programs configured to be executed by one or more processors, the one or more programs including a three-dimensional environment, which includes virtual user interface objects containing information about secure operations, and while the three-dimensional environment is visible via a display generation component, it detects a change in the user's current viewpoint via one or more input devices, and in response to detecting a change in the user's viewpoint, it enables user authorization for secure operations using virtual user interface objects according to a determination that at least a threshold amount of virtual user interface objects is visible from the user's viewpoint and the user is permitted to perform secure operations, and it deactivates user authorization for secure operations using virtual user interface objects according to a determination that the amount of virtual user interface objects visible from the user's viewpoint is less than a threshold amount.

[0011] According to several embodiments, a computer system is described. This computer system communicates with one or more input devices and display generation components. This computer system is a three-dimensional environment, and while the three-dimensional environment, which includes virtual user interface objects containing information about secure operations, is visible via the display generation components, it detects a change in the user's current viewpoint via one or more input devices, and in response to detecting a change in the user's viewpoint, it includes means for enabling user permission to perform secure operations using virtual user interface objects according to a determination that at least a threshold amount of virtual user interface objects is visible from the user's viewpoint and the user is permitted to perform secure operations, and means for deactivating user permission to perform secure operations using virtual user interface objects according to a determination that the amount of virtual user interface objects visible from the user's viewpoint is less than a threshold amount.

[0012] A method is described according to several embodiments. The method is performed in a computer system communicating with one or more input devices, display generating components, and biosensors. The method includes: biometric authentication of a device user using a biosensor to perform a first type of biometric authentication at a first time; receiving a request to perform a secure operation at a second time following the first time; and, in response to receiving the request to perform a secure operation, performing a secure operation without performing a first type of biometric authentication after receiving the request to perform a secure operation, based on a determination that the device user met the criteria between the first and second time, which are based on a plurality of sensor measurements obtained at a plurality of intermediate time points between the first and second time points, including sensor measurements obtained at a first intermediate time point and sensor measurements obtained at a second intermediate time point, and which determine that the same user was using the device between the first and second time points; and ceasing to perform the secure operation based on a determination that the continuity criteria were not met between the first and second time points.

[0013] According to some embodiments, non-temporary computer-readable storage media are described. A non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, display generating components, and biosensors, the one or more programs include instructions for biometric authentication of a device user using a biosensor to perform a first type of biometric authentication at a first time, receiving a request to perform a secure operation at a second time after the first time, and in response to receiving the request to perform a secure operation, performing a secure operation without performing a first type of biometric authentication after receiving a request to perform a secure operation, based on the determination that the device user met the criteria between the first and second time, which are based on the sensor measurements obtained at a first intermediate time and a second intermediate time, and which indicate that the same user was using the device between the first and second time, and ceasing to perform the secure operation based on the determination that the continuity criteria were not met between the first and second time.

[0014] According to several embodiments, a computer system is described. The computer system communicates with one or more input devices, display generation components, and biosensors. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for biometric authentication of a device user using a biosensor to perform a first type of biometric authentication at a first time, receiving a request to perform a secure operation at a second time after the first time, and in response to receiving a request to perform a secure operation, performing a secure operation without performing a first type of biometric authentication after receiving a request to perform a secure operation, based on a determination that the user of the device met a criterion between the first and second time, which is a criterion based on a criterion that the user of the device met a criterion between the first and second time, which is a criterion based on a criterion that the user of the device met a criterion between the first and second time, which is a criterion based on a criterion that the user of the device met a criterion based on based on a criterion that the user of the device met a criterion based on a criterion based on a criterion based on a criterion based on a criterion based on a criterion that the user of the device met a criterion based on a criterion based on a criterion based on a criterion based on a criterion based on a criterion based on a criterion based on a criterion that the user of the device met a criter

[0015] According to several embodiments, a computer system is described. This computer system communicates with one or more input devices and display generating components. The computer system includes means for biometrically authenticating a user of a device using a biometric sensor to perform a first type of biometric authentication at a first time; means for receiving a request to perform a secure operation at a second time after the first time; means for performing a secure operation without performing a first type of biometric authentication after receiving a request to perform a secure operation, in response to receiving a request to perform a secure operation, based on a determination that the user of the device met a criterion between the first and second time, which is a criterion based on a criterion obtained at a 1st intermediate time and a 2nd intermediate time, including a criterion obtained at a 1st intermediate time and a criterion obtained at a 2nd intermediate time, and which determines that the user of the device met a criterion between the first and second time that the same user was using the device between the first and second time; and means for canceling the execution of the secure operation in accordance with a determination that the continuity criterion was not met between the first and second time.

[0016] A method is described according to several embodiments. The method is performed in a computer system communicating with one or more input devices and a display generating component. The method includes receiving a request via one or more input devices for the display of a user interface relating to performing a secure operation, and, in response to the request for the display of a user interface relating to performing a secure operation, displaying a first user interface via a display generating component that includes a separate prompt for providing additional input to allow the device to perform a secure operation, wherein, according to a determination that the computer system is operating in a first mode, the separate prompt is a first prompt that provides physical input to allow a secure operation, and according to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt that displays a second user interface which enables user authorization of a secure operation without using physical input.

[0017] According to some embodiments, a non-temporary computer-readable storage medium is described. The non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices and display generating components, the one or more programs receiving a request via one or more input devices for the display of a user interface relating to performing a secure operation, and including instructions for displaying a first user interface via a display generating component, which includes a separate prompt for providing additional input to allow the device to perform a secure operation, wherein, according to a determination that the computer system is operating in a first mode, the separate prompt is a first prompt that provides physical input to allow a secure operation, and according to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt that displays a second user interface, which enables user authorization of a secure operation without using physical input.

[0018] According to several embodiments, a computer system is described. The computer system communicates with one or more input devices and a display generating component. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by one or more processors, the one or more programs receiving a request via one or more input devices for the display of a user interface relating to performing a secure operation, and including instructions for displaying a first user interface via a display generating component, which includes a separate prompt for providing additional input to the device to authorize the execution of the secure operation, wherein, according to a determination that the computer system is operating in a first mode, the separate prompt is a first prompt that provides physical input to authorize the secure operation, and according to a determination that the computer system is operating in a second mode different from the first mode, the separate prompt is a second prompt that displays a second user interface, which enables user authorization of the secure operation without using physical input.

[0019] According to several embodiments, a computer system is described. The computer system communicates with one or more input devices and a display generating component. The computer system includes means for receiving a request via one or more input devices for the display of a user interface relating to performing a secure operation, and means for displaying a first user interface via a display generating component in response to a request for the display of a user interface relating to performing a secure operation, wherein, according to a determination that the computer system is operating in a first mode, the individual prompt is a first prompt that provides physical input for permitting a secure operation, and according to a determination that the computer system is operating in a second mode different from the first mode, the individual prompt is a second prompt that displays a second user interface which enables user permission for a secure operation without using physical input.

[0020] It should be noted that the various embodiments described herein can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification and claims. Furthermore, it should be noted that the language used herein has been selected solely for readability and explanatory purposes and not to define or limit the subject matter of the invention. [Brief explanation of the drawing]

[0021] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings, and similar reference numbers throughout the following drawings refer to the corresponding parts.

[0022] [Figure 1]A block diagram showing the operating environment of a computer system for providing an XR experience according to some embodiments.

[0023] [Figure 2] A block diagram showing a controller of a computer system configured to manage and adjust a user's XR experience according to some embodiments.

[0024] [Figure 3] A block diagram showing a display generation component of a computer system configured to provide a visual component of an XR experience to a user according to some embodiments.

[0025] [Figure 4] A block diagram showing a hand tracking unit of a computer system configured to capture a user's gesture input according to some embodiments.

[0026] [Figure 5] A block diagram showing an eye tracking unit of a computer system configured to capture a user's gaze input according to some embodiments.

[0027] [Figure 6A] A flowchart showing a glint-assisted gaze tracking pipeline according to some embodiments.

[0028] [Figure 6B] A diagram showing an exemplary device connected via one or more communication channels according to some embodiments.

[0029] [Figure 7A] A diagram showing an example technique for permitting secure operation according to some embodiments. [Figure 7B]This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7C] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7D] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7E] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7F] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7G] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7H] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7I] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7J] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7K] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7L] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7M] This figure shows examples of techniques for enabling secure operation through several embodiments. [Figure 7N] This figure shows examples of techniques for enabling secure operation through several embodiments.

[0030] [Figure 8] This is a flowchart illustrating various embodiments of methods for facilitating user consent for secure operation.

[0031] [Figure 9]This is a flowchart illustrating methods for ensuring authentication continuity for secure operation through various embodiments.

[0032] [Figure 10] This is a flowchart illustrating various embodiments of methods for enabling secure operation through an accessibility interface. [Modes for carrying out the invention]

[0033] This disclosure relates to user interfaces that provide users with Extended Reality (XR) experiences, in several embodiments.

[0034] Figures 1 to 6B provide an explanation of an example computer system for providing an XR experience to a user. Figures 7A to 7N show examples of techniques for enabling secure operation in several embodiments. Figure 8 is a flowchart of a method for facilitating informed consent for secure operation in various embodiments. The process in Figure 8 is explained using the user interfaces in Figures 7A to 7N. Figure 9 is a flowchart of a method for ensuring authentication continuity for secure operation in various embodiments. The process in Figure 9 is explained using the user interfaces in Figures 7A to 7N. Figure 10 is a flowchart of a method for enabling secure operation via an accessibility interface in various embodiments. The process in Figure 10 is explained using the user interfaces in Figures 7A to 7N.

[0035] The following processes, through various technical reports, enhance the functionality of the device and streamline the user-device interface (for example, by helping users make appropriate inputs when operating / interacting with the device and reducing user errors). These techniques include providing users with improved visual feedback, reducing the number of inputs required to perform actions, adding control options without cluttering the user interface by adding displayed controls, performing actions without requiring further user input when a set of conditions is met, improving privacy and / or security, and / or other techniques. These techniques also reduce power consumption and improve the battery life of the device by enabling users to use the device more quickly and efficiently.

[0036] Furthermore, in any method described herein that is conditional on one or more conditions being met in one or more steps, it should be understood that the method described can be repeated in multiple iterations such that all the conditions that the steps of the method are conditional on are met in different iterations of the method. For example, if a method requires that a first step be performed if a condition is met, and a second step be performed if the condition is not met, a person skilled in the art will understand that the steps described in the claim are repeated in a specific order until the conditions are met and then not met. Thus, a method described in one or more steps that depends on one or more conditions being met can be rewritten as a method that is repeated until each of the conditions described in the method is met. However, this is not required for a claim of a system or computer-readable medium in which the system or computer-readable medium includes instructions that perform a conditional action based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency has been met without explicitly repeating the steps of the method until all the conditions that the steps of the method are conditional on are met. Those skilled in the art will also understand that, as with a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0037] In some embodiments, as shown in Figure 1, the XR experience is provided to the user via an operating environment 100 which includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or remote server), display generation components 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a velocity sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a home appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with the display generation component 120 (for example, within a head-mounted device or handheld device).

[0038] When describing an XR experience, various terms are used to specifically refer to several related but distinct environments that the user can perceive and / or interact with (for example, using inputs detected by the computer system 101 to cause the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101 that generates the XR experience). The following is a subset of these terms.

[0039] Physical Environment: The physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic systems. Examples of physical environments, such as a physical park, include physical objects such as physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through their senses of sight, touch, hearing, taste, and smell.

[0040] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people perceive and / or interact with through an electronic system. In XR, a subset of a person's body motion or its representation is tracked, and accordingly, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. For example, an XR system can detect a person's head rotation and, accordingly, adjust the graphic content and sound field presented to the person in a similar way to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the adjustment of the properties(s) of a virtual object(s) in an XR environment may be done in response to a representation of body motion (e.g., voice commands). A person may perceive and / or interact with an XR object using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person can perceive and / or interact with audio objects that create a 3D or spatial audio environment, providing the perception of point audio sources in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person may perceive and / or interact with only audio objects.

[0041] Examples of XR include virtual reality and mixed reality.

[0042] Virtual reality: A virtual reality (VR) environment refers to a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can perceive and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can perceive and / or interact with virtual objects in a VR environment through a simulation of their presence within the computer-generated environment and / or through a simulation of a subset of their physical movement within the computer-generated environment.

[0043] Mixed Reality: A mixed reality (MR) environment is a simulated environment designed to incorporate sensory input from or its representation from a physical environment, in addition to including computer-generated sensory input (e.g., virtual objects), in contrast to a virtual reality (VR) environment designed to rely entirely on computer-generated sensory input. On a virtual continuum, a mixed reality environment is any location between, but not including, the complete physical environment at one end and the virtual reality environment at the other end. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Also, some electronic systems for presenting an MR environment may track location and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical articles or their representations from the physical environment). For example, the system may take motion into account so that a virtual tree appears stationary relative to the physical ground.

[0044] Examples of mixed reality include extended reality and augmented virtual reality.

[0045] Extended Reality: An Extended Reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on or onto a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display that allows a person to directly view the physical environment. The system may also be configured to present virtual objects on the transparent or translucent display, thereby allowing a person to use the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture an image or video of the physical environment, which is a representation of the physical environment. The system composites the image or video with the virtual objects and presents the composite on the opaque display. A person uses this system to perceive the virtual objects superimposed on the physical environment by indirectly viewing the physical environment through the image or video of the physical environment. As used herein, a video of the physical environment shown on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example, as holograms, into or onto the physical environment, so that a person can use the system to perceive the virtual objects superimposed on the physical environment. Extended reality environments also refer to simulated environments in which the representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may transform one or more sensor images to plane a selected perspective (e.g., viewpoint) different from the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion of it, so that the modified portion is a non-photorealistic alteration of the original captured image.As a further example, the representation of the physical environment may be altered by graphically removing or obscuring parts of it.

[0046] Augmented virtuality (AV) refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. These sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park might have virtual trees and virtual buildings, while people with faces are realistically reproduced from images of real people. Another example is that virtual objects may adopt the shape or color of physical articles captured by one or more imaging sensors. A further example is that virtual objects may adopt shadows that correspond to the position of the sun in the physical environment.

[0047] Viewpoint-locked virtual objects: A virtual object is viewpoint-locked when the computer system displays the virtual object in the same location and / or position within the user's view, even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked in the forward direction of the user's head (e.g., the user's viewpoint is at least a portion of the user's field of view when the user is looking straight ahead). Thus, the user's viewpoint remains fixed even if the user's gaze moves, without moving the user's head. In embodiments where the computer system has a display generation component (e.g., a display screen) that can be repositioned relative to the user's head, the user's viewpoint is the extended reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper-left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) will continue to be displayed in the upper-left corner of the user's viewpoint even if the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the location and / or position in which a viewpoint-locked virtual object is displayed from the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so that the virtual object is also referred to as a "head-locked virtual object."

[0048] Environment-Locked Virtual Objects: A virtual object is environment-locked (or "world-locked") when a computer system displays it at a location and / or position in the user's viewpoint that is based on (e.g., selected by reference to and / or fixed to) a location and / or object in a three-dimensional environment (e.g., a physical or virtual environment). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, and as a result, the environment-locked virtual object will appear at a different location and / or position in the user's viewpoint. For example, an environment-locked virtual object locked to a tree directly in front of the user will appear centered in the user's viewpoint. If the user's viewpoint shifts to the right (e.g., the user's head is turned to the right) and the tree becomes left-leaning in the user's viewpoint (e.g., the tree's position in the user's viewpoint shifts), the environment-locked virtual object locked to the tree will appear left-leaning in the user's viewpoint. In other words, the location and / or position in which an environment-locked virtual object is displayed in the user's viewpoint depends on the location and / or object's position and / or orientation in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a fixed location in the physical environment and / or a coordinate system fixed to an object) to determine the position in which the environment-locked virtual object is displayed from the user's viewpoint. The environment-locked virtual object can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object) or to a moving part of the environment (e.g., a vehicle, animal, person, or a representation of a part of the user's body that moves independently of the user's viewpoint, such as the user's hands, wrists, arms, or feet), so that the virtual object moves as the viewpoint or the part of the environment moves in order to maintain a fixed relationship between the virtual object and the part of the environment.

[0049] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits delayed tracking behavior, reducing or delaying the motion of the environment-locked or viewpoint-locked virtual object in response to the movement of a reference point that the virtual object is following. In some embodiments, when exhibiting delayed tracking behavior, the computer system detects movement of the reference point that the virtual object is following (e.g., a part of the environment, a viewpoint, or a point fixed to the viewpoint, such as a point between 5 and 300 cm from the viewpoint) and intentionally delays the movement of the virtual object. For example, when the reference point (e.g., a part of the environment or viewpoint) moves at a first velocity, the virtual object is moved by the device so as to remain locked to the reference point, but at a second velocity slower than the first velocity (e.g., the virtual object begins to catch up to the reference point until the reference point stops or slows down). In some embodiments, when a virtual object exhibits delayed tracking behavior, the device ignores small movements of the reference point (e.g., ignoring movements of the reference point that are below a threshold movement amount, such as a movement of 0 to 5 degrees or a movement of 0 to 50 cm). For example, when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and then decreases as the amount of movement of the reference point increases beyond a threshold (e.g., a "delayed tracking" threshold) as the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point.In some embodiments, a virtual object that maintains a substantially fixed position with respect to a reference point includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the reference point in one or more dimensions (e.g., above / below, left / right, and / or forward / behind the position of the reference point).

[0050] Hardware: There are many different types of electronic systems that enable a person to perceive and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (e.g., contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing sounds of the physical environment. A head-mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface. In some embodiments, the controller 110 is configured to manage and adjust the XR experience for the user.In some embodiments, the controller 110 includes a preferred combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to Figure 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server located outside the scene 105 (e.g., a cloud server, a central server, etc.). In some embodiments, the controller 110 is communicably coupled to a display generation component 120 (e.g., an HMD, display, projector, touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within a housing (e.g., a physical housing) of one or more of the display generation components 120 (e.g., a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0051] In some embodiments, the display generation component 120 is configured to provide the user with an XR experience (e.g., at least the visual components of an XR experience). In some embodiments, the display generation component 120 includes a preferred combination of software, firmware, and / or hardware. The display generation component 120 is described in more detail below with reference to Figure 3. In some embodiments, the functions of the controller 110 are provided by and / or combined with the display generation component 120.

[0052] According to some embodiments, the display generation component 120 provides the user with an XR experience while the user is virtually and / or physically present in the scene 105.

[0053] In some embodiments, the display generation component is mounted on a part of the user's body (e.g., their head or hand). Thus, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device, which has a display directed towards the user's field of view and a camera directed towards scene 105. In some embodiments, the handheld device is optionally placed in a housing mounted on the user's head. In some embodiments, the handheld device is optionally placed on a support in front of the user (e.g., a tripod). In some embodiments, the display generation component 120 is an XR chamber, housing, or room configured to present XR content when the user is not wearing or holding the display generation component 120. Many user interfaces described in reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) can also be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface that demonstrates interaction with XR content triggered by interaction occurring in the space in front of a handheld or tripod-mounted device can be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the XR content response is displayed through the HMD. Similarly, a user interface that demonstrates interaction with XR content triggered by movement of a handheld or tripod-mounted device relative to a physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)) can be implemented similarly to an HMD where the movement is triggered by movement of the HMD relative to a physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)).

[0054] While relevant features of the operating environment 100 are shown in Figure 1, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more appropriate embodiments of the exemplary embodiments disclosed herein.

[0055] Figure 2 is a block diagram of an example of the controller 110 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), central processing unit (CPU), processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global Mobile Communication System (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZiGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0056] In some embodiments, one or more communication buses 204 include circuits for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices 206 include at least one of the following: a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0057] Memory 220 includes high-speed random-access memory such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-temporary computer-readable storage medium. In some embodiments, memory 220, or the non-temporary computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 230 and XR experience module 240.

[0058] The operating system 230 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for each group of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0059] In some embodiments, the data acquisition unit 241 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the display generation components 120 of Figure 1, and optionally one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data acquisition unit 241 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0060] In some embodiments, the tracking unit 242 is configured to map scene 105 and track the position / location of at least the display generation component 120 relative to scene 105 in Figure 1, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand, and / or the motion of one or more parts of the user's hand relative to scene 105 in Figure 1, relative to the display generation component 120, and / or relative to a coordinate system defined for the user's hand. The hand tracking unit 244 is described in more detail below with reference to Figure 4. In some embodiments, the eye-tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to the scene 105 (e.g., the physical environment and / or the user (e.g., the user's hands)) or to XR content displayed via the display generation component 120. The eye-tracking unit 243 is described in more detail below with reference to Figure 5.

[0061] In some embodiments, the adjustment unit 246 is configured to manage and adjust the XR experience presented to the user by the display generation component 120 and optionally by one or more of the output devices 155 and / or peripheral devices 195. For this purpose, in various embodiments, the adjustment unit 246 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0062] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0063] While the data acquisition unit 241, tracking unit 242 (including, for example, eye-tracking unit 243 and hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 are shown as residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, tracking unit 242 (including, for example, eye-tracking unit 243 and hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 may be located in separate computing devices.

[0064] Furthermore, Figure 2 is intended to illustrate the function of various features that may be present in a particular embodiment, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, in various embodiments, several functional modules shown separately in Figure 2 can be implemented within a single module, and the various functions of a single functional block can be performed by one or more functional blocks. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0065] Figure 3 is a block diagram of an example of a display generation component 120 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. For that purpose, in some non-limiting examples, the display generation component 120 (e.g., HMD) may include one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, Bluetooth, ZiGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional in-facing and / or out-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0066] In some embodiments, one or more communication buses 304 include circuits for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include at least one of the following: an inertial measuring unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.).

[0067] In some embodiments, one or more XR displays 312 are configured to provide the user with an XR experience. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more XR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holographic. For example, a display generation component 120 (e.g., HMD) includes a single XR display. In another embodiment, the display generation component 120 includes an XR display for each of the user's eyes. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0068] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hands and optionally, at least a portion of the user's arms (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene that the user would view if a display generation component 120 (e.g., an HMD) were not present (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., complementary metal-oxide-semiconductor (CMOS) image sensors or charge-coupled device (CCD) image sensors), one or more infrared (IR) cameras, one or more event-based cameras, and / or similar.

[0069] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-temporary computer-readable storage medium. In some embodiments, memory 320, or the non-temporary computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 330 and XR presentation module 340.

[0070] The operating system 330 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to the user via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0071] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 in Figure 1. For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0072] In some embodiments, the XR presentation unit 344 is configured to present XR content via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation unit 344 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0073] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (for example, a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate extended reality) based on media content data. For this purpose, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0074] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110 and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral devices 195. For this purpose, in various embodiments, the data transmission unit 348 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0075] While the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 are shown as existing on a single device (e.g., the display generation component 120 in Figure 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 may be located in separate computing devices.

[0076] Furthermore, Figure 3 is intended to illustrate the functionality of various features that may be present in a particular implementation, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, several functional modules shown separately in Figure 3 can be realized within a single module, and the various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0077] Figure 4 is a schematic diagram of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 (Figure 1) is controlled by the hand tracking unit 244 (Figure 2) to track the location / position of one or more parts of the user's hand and / or the motion of one or more parts of the user's hand relative to the scene 105 of Figure 1 (e.g., relative to a part of the physical environment surrounding the user, relative to the display generation component 120, or relative to a part of the user (e.g., the user's face, eyes, or head), and / or the user's hand, in a coordinate system defined therein). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0078] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures a hand image with sufficient resolution to allow for the distinction of fingers and their respective positions. The image sensor 404 can typically capture images of other parts of the user's body, or images of the entire body, and may have either a zoom function or a dedicated sensor with high magnification to capture an image of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures a 2D color video image of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor or a portion thereof is used to define an interaction space in which hand movements captured by the image sensor are processed as input to the controller 110.

[0079] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, thereby extracting high-level information from the map data. This high-level information is typically provided to an application running on the controller via an application programming interface (API), which drives the display generation components 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving their hand 406 to change the orientation of their hand.

[0080] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spot in the pattern. This approach is advantageous in that the user does not need to hold or wear any kind of beacon, sensor, or other marker. This gives the depth coordinates of points in the scene relative to a given reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines a set of orthogonal x, y, and z axes such that the depth coordinates of points in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods such as stereoscopic imaging or time-of-flight measurement based on one or more cameras or other types of sensors.

[0081] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves their hand (e.g., the entire hand or one or more fingers). Software running on the processor in the image sensor 404 and / or controller 110 processes the 3D map data to extract patch descriptors of the hand within these depth maps. Based on a previous learning process, the software matches these descriptors against patch descriptors stored in the database 408 to estimate the hand pose in each frame. The pose typically includes the 3D location of the user's wrist and fingertips.

[0082] The software can also analyze the trajectory of the hand and / or fingers across multiple frames in a sequence to identify gestures. The pose estimation function described herein may be interleaved with the motion tracking function so that patch-based pose estimation is performed only once every two frames (or more), and tracking is used to detect changes in pose that occur over the remaining frames. Pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify an image presented on the display generation component 120, or perform other functions, depending on the pose and / or gesture information.

[0083] In some embodiments, the gesture includes an air gesture. An air gesture is a gesture detected by the user without (or independently of) touching an input element that is part of a device (e.g., a computer system 101, one or more input devices 125, and / or a hand tracking device 140), and is based on the detected motion of a part of the user's body in the air (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs), including the motion of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), the motion of the user's body relative to another part of the user's body (e.g., the movement of the user's hand relative to the user's shoulder, the movement of one of the user's hands relative to the user's other hand, and / or the movement of the user's fingers relative to another finger or part of the user's hand), and / or the absolute motion of a part of the user's body (e.g., a tap gesture including the movement of the hand in a predetermined position by a predetermined amount and / or speed, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body).

[0084] In some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures, as in some embodiments, performed by moving one or more of the user's fingers relative to other fingers or parts of the user's hand for interacting with an XR environment (e.g., a virtual or mixed reality environment). In some embodiments, an air gesture is a gesture detected without the user touching an input element that is part of the device (or independently of an input element that is part of the device), and is based on detected motion of a part of the user's body, including motion of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), motion of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute motion of a part of the user's body (e.g., a tap gesture involving movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture involving rotation of a part of the user's body by a predetermined speed or amount).

[0085] In some embodiments where the input gesture is an air gesture (i.e., without physical contact with an input device that provides the computer system with information about which user interface element is the target of user input, such as contact with a user interface element displayed on a touchscreen or contact with a mouse or trackpad to move a cursor over a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of user input (e.g., in the case of direct input, as described below). Thus, in implementations involving air gestures, the input gesture is the detected attention (e.g., gaze) to the user interface element in combination (e.g., simultaneously) with the movement of the user's fingers (one or more) and / or hand to perform pinch and / or tap input, as described in more detail below.

[0086] In some embodiments, input gestures directed towards a user interface object are performed directly or indirectly with respect to the user interface object. For example, user input is performed directly on the user interface object in conjunction with the user performing an input gesture with their hand at a position corresponding to the user interface object's position in the three-dimensional environment (e.g., determined based on the user's current viewpoint). In some embodiments, the input gesture is performed indirectly on the user interface object in conjunction with the user performing an input gesture while the user's attention (e.g., gaze) to the user interface object is detected, while the user's hand position is not at the position corresponding to the user interface object's position in the three-dimensional environment. For example, in the case of a direct input gesture, the user can direct their input towards the user interface object by initiating the gesture at or near a position corresponding to the user interface object's display position (e.g., within a distance of 0.5 cm, 1 cm, 5 cm, or 0-5 cm from the optional outer edge or optional central portion). In the case of indirect input gestures, the user can direct their input towards the user interface object by paying attention to the user interface object (for example, by gazing at the user interface object), and while paying attention to the options, the user initiates the input gesture (for example, at any position detectable by the computer system) (for example, at a position that does not correspond to the display position of the user interface object).

[0087] In some embodiments, the input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch and tap inputs for interacting with virtual or mixed reality environments, as in some embodiments. For example, the pinch and tap inputs described later are performed as air gestures.

[0088] In some embodiments, a pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch-and-drag gesture, or a double pinch gesture. For example, a pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other, i.e., including an optional interruption (e.g., within 0 to 1 second) immediately after the touch. A long pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other for at least a threshold time amount (e.g., at least 1 second) before detecting an interruption of contact between them. For example, a long pinch gesture includes the user holding a pinch gesture (e.g., if two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture that is an air gesture includes two (e.g., or more) pinch inputs (e.g., performed with the same hand) that are detected directly and consecutively (e.g., within a predetermined period of time) to each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaks contact between two or more fingers), and then performs a second pinch input within a predetermined period (e.g., within 1 second or 2 seconds) after releasing the first pinch input.

[0089] In some embodiments, an air gesture, a pinch-and-drag gesture, includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in relation to (e.g., after) a drag input that changes the user's hand position from a first position (e.g., a drag initiation position) to a second position (e.g., a resistance termination position). In some embodiments, the user maintains the pinch gesture while performing the drag input and releases the pinch gesture (e.g., spreading two or more fingers) to terminate the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together and touches them to each other, and then moves the same hand to a second position in the air with a drag gesture). In some embodiments, the pinch input is performed by the user's first hand and the drag input is performed by the user's second hand (e.g., the user's second hand moves from the first position to the second position in the air while the user continues the pinch input with the user's first hand). In some embodiments, an input gesture that is an air gesture includes an input (e.g., a pinch input and / or a tap input) performed using both of the user's hands. For example, an input gesture includes two (e.g., or more) pinch inputs performed in relation to each other (e.g., simultaneously or within a predetermined period of time). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) performed using the user's first hand, and a second pinch input performed using the other hand (e.g., a second hand of the user's hands) in relation to performing a pinch input using the first hand. In some embodiments, movement between the user's hands (e.g., to increase and / or decrease the distance or relative orientation between the user's hands).

[0090] In some embodiments, a tap input performed as an air gesture (e.g., directed towards a user interface element) includes the movement of one or more of the user's fingers toward the user interface element, the movement of the user's hand toward the user interface element with the user's fingers (one or more) optionally extended toward the user interface element, downward motion of the user's fingers (e.g., mimicking a mouse click motion or a tap on a touchscreen), or other default movements of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand performing the tap gesture, moving away from the user's viewpoint and / or toward the object that is the target of the tap input, followed by the end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand performing the tap gesture (e.g., moving away from the user's viewpoint and / or the end of the movement toward the object that is the target of the tap input, a reversal of the direction of the finger or hand movement, and / or a reversal of the direction of acceleration of the finger or hand movement).

[0091] In some embodiments, the user's attention is determined to be directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards that part of the three-dimensional environment (optionally, without requiring any other conditions). In some embodiments, for the device to determine that the user's attention is directed towards a part of the three-dimensional environment, the device determines that the user's attention is directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards a part of the three-dimensional environment, with one or more additional conditions such as the gaze being directed towards the part of the three-dimensional environment for at least a threshold duration (e.g., dwell time) while the user's viewpoint is within a distance threshold from the part of the three-dimensional environment, and / or the gaze being directed towards a part of the three-dimensional environment. If one of the additional conditions is not met, the device determines that the user's attention is not directed towards the part of the three-dimensional environment to which the gaze is directed (e.g., until one or more additional conditions are met).

[0092] In some embodiments, the detection of a ready state configuration of the user or a part of the user is detected by the computer system. The detection of a ready state configuration of the hand is used by the computer system as an indicator that the user is likely to be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of a hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape where the thumb and one or more fingers are extended and spaced apart, ready to perform a pinch or grab gesture, or a pre-tap shape where one or more fingers are extended and the palm is facing away from the user), whether the hand is in a predetermined position relative to the user's line of sight (e.g., below the user's head, above the user's waist, or extended at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular way (e.g., moved towards the area in front of the user above the user's waist, below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interaction element of the user interface is responsive to attentional (e.g., gaze) input.

[0093] In scenarios where the input is described in reference to an air gesture, similar gestures may also be detected using hardware input devices attached to or held by one or more of the user's hands, in which case the position of the hardware input device in space may be tracked using optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers, and / or one or more inertial measurement units, and it should be understood that the position and / or movement of the hardware input device is used instead of the position and / or movement of one or more hands in the corresponding air gesture(s). In scenarios where input is described in reference to an air gesture, similar gestures may also be detected using hardware input devices attached to or held by one or more of the user's hands, in which case user input may be detected using controls included in the hardware input device, such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, one or more hand or finger covers, and / or other hardware input device controls, and the hand or finger covers may detect the position or change in position of parts of the hand and / or fingers relative to each other, relative to the user's body, and / or the user's physical environment, in which case user input using controls included in the hardware input device is used instead of hand gestures and / or finger gestures such as air taps or air pinches in the corresponding air gesture(s). For example, a selection input described as being performed with an air tap or air pinch input may alternatively be detected by a button press, a tap on a touch-sensitive surface, a press on a pressure-sensitive surface, or other hardware input.As another example, a movement input described as being performed by air pinch and drag can alternatively be detected based on interaction with hardware input controls such as button press and hold, touch on a touch-sensitive surface, or press on a pressure-sensitive surface, or based on hardware input that follows the movement of other hardware input devices in space (e.g., accompanying the hand to which the hardware input device is associated). Similarly, two-handed inputs, including movements of both hands relative to each other, can also be performed using various combinations of inputs detected by air gestures and / or one or more of the aforementioned hardware input devices, using one air gesture and one hardware input device held in the hand not performing the air gesture, two hardware input devices held in separate hands, or two air gestures performed by separate hands.

[0094] In some embodiments, the software may be downloaded electronically to the controller 110, for example, over a network, or instead, it may be provided on a tangible non-temporary medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively or additionally, some or all of the computer's described functions may be implemented in dedicated hardware such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in Figure 4, for example, as a separate unit from the image sensor 404, some or all of the controller's processing functions may be associated with the image sensor 404 by a suitable microprocessor and software, or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand-tracking device), or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or by any other suitable computerized device such as a game console or media player. The sensing function of the image sensor 404 can also be integrated into a computer or other computerized device controlled by the sensor output.

[0095] Figure 4 further includes schematic diagrams of depth maps 410 captured by image sensor 404 according to several embodiments. The depth map includes a matrix of pixels, each having a depth value, as described above. Pixels 412 corresponding to the hand 406 are segmented in this map from the background and the wrist. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from image sensor 404, with the gradation becoming darker as the depth increases. Controller 110 processes these depth values ​​to identify and segment image components (i.e., groups of adjacent pixels) that have the characteristics of a human hand. These characteristics may include, for example, the overall size, shape, and frame-to-frame motion of the sequence of depth maps.

[0096] Figure 4 also schematically shows the hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to several embodiments. In Figure 4, the hand skeleton 414 is superimposed on the hand background 416, which has been segmented from the original depth map. In some embodiments, the hand (e.g., finger joints, fingertips, center of the palm, end of the hand connected to the wrist), and optionally major feature points on the wrist or arm connected to the hand, are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these major feature points across multiple image frames are used by the controller 110 to determine, according to several embodiments, a hand gesture performed by the hand or the current state of the hand.

[0097] Figure 5 shows an exemplary embodiment of the eye-tracking device 130 (Figure 1). In some embodiments, the eye-tracking device 130 is controlled by an eye-tracking unit 243 (Figure 2) to track the position and movement of the user's gaze toward the scene 105 or toward the XR content displayed via the display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, if the display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device positioned in a wearable frame, the head-mounted device includes both a component for generating XR content for user viewing and a component for tracking the user's gaze toward the XR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, if the display generation component is a handheld device or an XR chamber, the eye-tracking device 130 is optionally a separate device from the handheld device or XR chamber. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used with a display generation component that is mounted on the head or a display generation component that is not mounted on the head. In some embodiments, the eye-tracking device 130 is not a head-mounted device, but is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye-tracking device 130 is not a head-mounted device, but is optionally part of a non-head-mounted display generation component.

[0098] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames containing left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include, or be coupled to, one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display on which the user can directly view the physical environment and display virtual objects on a transparent or translucent display. In some embodiments, the display generation component projects virtual objects onto the physical environment. The virtual objects are projected, for example, onto a physical surface or as holograms, so that the individual can use the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0099] As shown in Figure 5, in some embodiments, the eye-tracking device 130 (e.g., gaze tracking device) includes at least one eye-tracking camera (e.g., an infrared (IR) camera or a near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be directed toward the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, it may be directed toward a "hot" mirror positioned between the user's eye and a display panel that reflects IR or NIR light from the eye to the eye-tracking camera while allowing visible light to pass through. The eye-tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and communicates the gaze tracking information to the controller 110. In some embodiments, both of the user's eyes are tracked separately by their respective eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked by a separate eye-tracking camera and light source.

[0100] In some embodiments, the eye-tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye-tracking device for a specific operating environment 100, e.g., the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at the factory or another facility before delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automated calibration process or a manual calibration process. The user-specific calibration process may include estimating the eye parameters of a particular user, e.g., pupil location, central visual location, optical axis, visual axis, interpupillary distance. According to some embodiments, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, the images captured by the eye-tracking camera can be processed using a Glint-assisted method to determine the user's current visual axis and point of fixation relative to the display.

[0101] As shown in Figure 5, the eye-tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye-tracking system which includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eyes(s) 592. The eye-tracking camera 540 is positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display or projector of a handheld device) and may be directed towards a mirror 550 that transmits visible light while reflecting IR or NIR light from the eye(s) 592 (e.g., as shown at the top of Figure 5), or may be directed towards the user's eye(s) 592 to receive reflected IR or NIR light from the eye(s) 592 (e.g., as shown at the bottom of Figure 5).

[0102] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames of left and right display panels) and provides the frames 562 to the display 510. For various purposes, for example, when processing the frames 562 for display, the controller 110 uses gaze tracking input 542 from the eye-tracking camera 540. Optionally, the controller 110 estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye-tracking camera 540, using a Glint-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0103] The following describes, but is not intended to be limiting, several possible use cases of the user's current gaze direction. As an exemplary use case, the controller 110 may render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content at a higher resolution in the central visual region determined from the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content within the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content within the view based at least partially on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may capture the physical environment of the XR experience and orient an external camera to focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface in the environment that the user is currently viewing on the display 510. In another exemplary use case, the eyepiece 520 may be a focusing lens, and the controller uses eye-tracking information to adjust the focus of the eyepiece 520 so that the virtual object currently being viewed by the user has appropriate binocular coordination to match the convergence of the user's eye 592. The controller 110 can use the eye-tracking information to orient and adjust the focus of the eyepiece 520 so that the nearby object being viewed by the user appears at the correct distance.

[0104] In some embodiments, the eye-tracking device is part of a head-mounted device mounted on a wearable housing, which includes a display (e.g., display 510), two eyepieces (e.g., one or more eyepieces 520), an eye-tracking camera (e.g., one or more eye-tracking cameras 540), and a light source (e.g., a light source 530 (e.g., an IR LED or NIR LED)). The light source emits light (e.g., IR light or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in Figure 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0105] In some embodiments, the display 510 emits light within the visible light range and does not emit light within the IR or NIR range, thus not introducing noise into the eye-tracking system. Note that the location and angle of the eye-tracking camera(s) 540 are given as examples and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is positioned on each side of the user's face. In some embodiments, two or more NIR cameras 540 can be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0106] Embodiments of eye-tracking systems, such as those shown in Figure 5, can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0107] Figure 6A shows a glint-assisted eye-tracking pipeline according to several embodiments. In some embodiments, the eye-tracking pipeline is implemented by a glint-assisted eye-tracking system (e.g., an eye-tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye-tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the glint-assisted eye-tracking system tracks the pupil contour and glint in the current frame by using prior information from previous frames when analyzing the current frame. When not in tracking state, the glint-assisted eye-tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues in tracking state for the next frame.

[0108] As shown in Figure 6A, the eye-tracking camera can capture left and right images of the user's left and right eyes. The captured images are then fed into the eye-tracking pipeline for processing, which begins at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each set of captured images may be fed into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0109] At 610, if the tracking status for the currently captured image is "yes", the method proceeds to element 640. At 610, if the tracking status is "no", the image is analyzed to detect the user's pupil and glint in the image, as shown in 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. If they are not successfully detected, the method returns to element 610 and processes the next image of the user's eyes.

[0110] In 640, if the process proceeds from element 610, the current frame is analyzed to track the pupil and glints based in part on previous information from previous frames. In 640, if the process proceeds from element 630, the tracking state is initialized based on the detected pupil and glints in the current frame. The results of the processing in element 640 are checked to verify that the tracking or detection results are reliable. For example, the results may be checked to determine whether the pupil and a sufficient number of glints for gaze estimation have been successfully tracked or detected in the current frame. In 650, if the results are unreliable, the tracking state is set to "no" in element 660, and the method returns to element 610 to process the next image of the user's eyes. In 650, if the results are reliable, the method proceeds to element 670. In 670, the tracking state is set to "yes" (if not already "yes"), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0111] Figure 6A is intended to serve as an example of an eye-tracking technology that may be used in a particular implementation. As will be recognized by those skilled in the art, other eye-tracking technologies that currently exist or may be developed in the future may be used in place of, or in combination with, the glint-assisted eye-tracking technology described herein in the computer system 101 to provide users with XR experiences in various embodiments.

[0112] Figure 6B shows exemplary devices connected via one or more communication channels to participate in a transaction, according to several embodiments. One or more exemplary electronic devices (e.g., devices 602 and 604) are optionally configured to detect input (e.g., specific user input, NFC field) and optionally to transmit payment information (e.g., using NFC). One or more electronic devices optionally include NFC hardware and are configured to be NFC enabled.

[0113] Electronic devices (e.g., devices 602 and 604) are optionally configured to store payment account information associated with each of one or more payment accounts. The payment account information includes, for example, one or more of the following: personal or company name, billing address, login, password, account number, expiration date, security code, telephone number, bank associated with the payment account (e.g., issuing bank), and card network identifier. In some embodiments, the payment account information includes an image, such as a photograph of a payment card (e.g., taken by and / or received by the device). In some embodiments, the electronic device receives user input containing at least some payment account information (e.g., receiving a credit, debit, account, or gift card number and expiration date entered by the user). In some embodiments, the electronic device detects at least some payment account information from an image (e.g., a payment card captured by the device's camera sensor). In some embodiments, the electronic device receives at least some payment account information from another device (e.g., another user device or server). In some embodiments, the electronic device receives payment account information from a server associated with another service (e.g., an app for renting or selling audio and / or video files) to which the user or user device or identified payment account data has previously made a purchase.

[0114] In some embodiments, payment accounts are added to electronic devices (e.g., devices 602 and 604) so ​​that payment account information is securely stored on the electronic devices. In some embodiments, after a user initiates such a process, the electronic devices send information about the payment accounts to a transaction coordinating server, which then communicates with a server (e.g., a payment server) operating on the payment network for the account to ensure the validity of the information. The electronic devices are optionally configured to receive scripts from the server that enable the electronic devices to program payment information for the account onto a secure element.

[0115] In some embodiments, communication between electronic devices 602 and 604 facilitates transactions (e.g., general or specific transactions). For example, the first electronic device (e.g., 602) can act as a provisioning or management device and can send notifications of new or updated payment account data (e.g., information about new accounts, updated information about existing accounts, and / or alerts for existing accounts) to the second electronic device (e.g., 604). In another example, the first electronic device (e.g., 602) can send data to the second electronic device that reflects information about a payment transaction facilitated by the first electronic device. The information optionally includes one or more of the following: the payment amount, the account used, the time of purchase, and whether the default account has been changed. The second device (e.g., 604) optionally uses such information to update the default payment account (e.g., based on a learning algorithm or explicit user input).

[0116] Electronic devices (e.g., 602, 604) are configured to communicate with each other via one of a variety of networks. For example, devices communicate using a Bluetooth connection 616 (including, for example, a conventional Bluetooth connection or a Bluetooth low-energy connection) or a WiFi network 614. Communication between user devices is optionally tuned to reduce the possibility of inappropriate sharing of information between devices. For example, communication relating to payment information requires that the communicating devices be paired (e.g., associated with each other through explicit user interaction) or associated with the same user account.

[0117] In some embodiments, electronic devices (e.g., 602, 604) are optionally used to communicate with an NFC-enabled point-of-sale (POS) payment terminal 606. This communication is optionally performed using various communication channels and / or techniques. In some embodiments, electronic devices (e.g., 602, 604) communicate with the payment terminal 606 using NFC channel 618. In some embodiments, the payment terminal 606 communicates with the electronic devices (e.g., 602, 604) using peer-to-peer NFC mode. Electronic devices (e.g., 602, 604) are optionally configured to send signals to the payment terminal 606 containing payment information for a payment account (e.g., a default account or an account selected for a particular transaction).

[0118] In some embodiments, advancing a transaction involves transmitting a signal containing payment information to an account, such as a payment account. In some embodiments, advancing a transaction involves reconfiguring an electronic device (e.g., 602, 604) to respond as a contactless payment card, such as an NFC-enabled contactless payment card, and then transmitting account credentials to a payment terminal 606, etc., via NFC. In some embodiments, after transmitting account credentials via NFC, the electronic device is reconfigured not to respond as a contactless payment card (for example, permission is required before it can be reconfigured to respond as a contactless payment card again via NFC).

[0119] In some embodiments, signal generation and / or transmission are controlled by a secure element within an electronic device (e.g., 602, 604). The secure element optionally requires specific user input before disclosing payment information. For example, the secure element optionally requires detection of the electronic device being worn, detection of a button press, detection of a passcode entry, detection of a touch, detection of one or more option selections (e.g., received during interaction with an application), detection of a fingerprint signature, detection of a voice or voice command, and / or detection of a gesture or movement (e.g., rotation or acceleration). In some embodiments, if a communication channel (e.g., an NFC communication channel) is established with another device (e.g., a payment terminal 606) within a specified period from the time input is detected, the secure element discloses payment information to be sent to the other device (e.g., a payment terminal 606). In some embodiments, the secure element is a hardware component that controls the disclosure of secure information. In some embodiments, the secure element is a software component that controls the disclosure of secure information.

[0120] In some embodiments, the protocols involved in the transaction depend, for example, on the device type. For example, the conditions for generating and / or transmitting payment information may differ between a wearable device (e.g., device 604) and a telephone (e.g., device 602). For example, the conditions for generation and / or transmission on a wearable device include detecting that a button has been pressed (e.g., after security verification), while the conditions on a telephone do not require a button press but instead require the detection of a specific interaction with an application. In some embodiments, the conditions for transmitting and / or disclosing payment information include receiving a specific input on each of multiple devices. For example, disclosing payment information may optionally require the detection of a fingerprint and / or passcode on one device (e.g., device 602) and the detection of a machine input (e.g., a button press) on another device (e.g., device 604).

[0121] The payment terminal 606 optionally generates a signal to send to the payment server 612 using payment information to determine whether the payment is authorized. The payment server 612 optionally includes any device or system configured to receive payment information related to a payment account and determine whether the proposed purchase is authorized. In some embodiments, the payment server 612 includes a server of the issuing bank. The payment terminal 606 communicates with the payment server 612 directly or indirectly through one or more other devices or systems (e.g., a server of the acquiring bank and / or a server of the card network).

[0122] The payment server 612 optionally uses at least a portion of the payment information to identify a user account from a database of user accounts (e.g., 608). For example, each user account includes payment information. An account is optionally located by locating the account that has specific payment information matching the payment information from the POS communication. In some embodiments, payment is rejected if the provided payment information is inconsistent (e.g., the expiration date does not correspond to a credit, debit, or gift card number) or if there is no account with payment information matching the payment information from the POS communication.

[0123] In some embodiments, the data for a user account further identifies one or more limits (e.g., credit limit), current or previous balance, previous transaction date, location, and / or amount, account status (e.g., active or frozen), and / or authorization instruction. In some embodiments, a payment server (e.g., 612) uses such data to determine whether to authorize the payment. For example, the payment server may reject the payment if the purchase amount added to the current balance exceeds the account limit, if the account is frozen, if the amount of previous transactions exceeds a threshold, or if the number or frequency of previous transactions exceeds a threshold.

[0124] In some embodiments, the payment server 612 responds to the POS payment terminal 606 with an indicator of whether the proposed purchase has been approved or rejected. In some embodiments, the POS payment terminal 606 transmits a signal to an electronic device (e.g., 602, 604) to identify the result. For example, if the purchase is approved, the POS payment terminal 606 sends a receipt to the electronic device (e.g., 602, 604) (e.g., via a transaction coordinating server that manages transaction applications on the user device). In some cases, the POS payment terminal 606 presents an output (e.g., visual or audio output) indicating the result. The payment may be sent to the retailer as part of the approval process or at a later date.

[0125] In some embodiments, electronic devices (e.g., 602, 604) participate in transactions that are completed without the intervention of the POS payment terminal 606. For example, when detecting that machine input has been received, a secure element within the electronic device (e.g., 602, 604) exposes payment information so that an application on the electronic device can access the information (e.g., send the information to a server associated with the application).

[0126] In some embodiments, electronic devices (e.g., 602, 604) are in a locked or unlocked state. In the locked state, the electronic device is powered on and operational, but is prevented from performing a default set of operations in response to user input. This default set of operations may include navigation between user interfaces, activation or deactivation of a default set of functions, and activation or deactivation of specific applications. The locked state may be used to prevent unintended or unauthorized use of some functionality of the electronic device, or activation or deactivation of some functions on the electronic device. In the unlocked state, electronic device 602 is powered on and operational, and is not prevented from performing at least a portion of the default set of operations that cannot be performed in the locked state.

[0127] When a device is in a locked state, it is said to be locked. In some embodiments, a locked device may be subject to a limited set of user inputs, including inputs corresponding to attempts to transition the device out of an unlocked state, or inputs corresponding to turning off the device.

[0128] In some embodiments, the secure element is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms. In some embodiments, the secure element provides (or discloses) payment information (e.g., an account number and / or a transaction-specific dynamic security code). In some embodiments, the secure element provides (or discloses) payment information in response to the device receiving authorization such as user authentication (e.g., fingerprint authentication, passcode authentication, detecting a double press of a hardware button by providing authentication credentials to the device when the device is in an unlocked state, and optionally when the device has been continuously on the user's wrist since it was unlocked; continuous presence of the device on the user's wrist is determined by periodically checking whether the device is in contact with the user's skin). For example, the device detects a fingerprint with the device's fingerprint sensor (e.g., a fingerprint sensor integrated into a button). The device determines whether this fingerprint matches a registered fingerprint. In accordance with the determination that this fingerprint matches a registered fingerprint, the secure element provides (or discloses) payment information. If the system determines that this fingerprint does not match any registered fingerprints, it will suspend the provision (or disclosure) of payment information.

[0129] This disclosure describes various input methods for interaction with computer systems. Where one example is provided using one input device or method, and another example is provided using a different input device or method, each example may be compatible with the input device or method described in the other example, and their use should be considered optional. Similarly, various output methods for interaction with computer systems are described. Where one example is provided using one output device or method, and another example is provided using a different output device or method, each example may be compatible with the output device or method described in the other example, and their use should be considered optional. Similarly, various methods for interaction with virtual or mixed reality environments via computer systems are described. Where one example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, each example may be compatible with the method described in the other example, and their use should be considered optional. Therefore, this disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User interface and related processes

[0130] Here, we focus on embodiments of user interfaces (UIs) and related processes that can be implemented in a computer system such as a portable multifunction device or head-mounted device that communicates with display generation components, one or more input devices, and (optionally) biosensors.

[0131] Figures 7A to 7N show examples of allowing secure operation. Figure 8 is a flowchart of exemplary method 800 for facilitating user consent for secure operation. Figure 9 is a flowchart of authentication continuity method 900 for secure operation. Figure 10 is a flowchart of exemplary method 1000 for allowing secure operation via an accessibility interface. The user interfaces in Figures 7A to 7N are used to illustrate the processes described below, including the processes in Figures 8, 9, and 10.

[0132] Figure 7A shows an electronic device 700 including a display 700a. In Figure 7A, a user 704 is interacting with the electronic device 700. The electronic device 700 may correspond to a tablet device, a wearable device (e.g., a head-mounted display), a smartphone, and / or a smartwatch. In addition, the display 700a may include, or be coupled to, display generation components (e.g., a display controller, a touch-sensitive display system, a display (e.g., embedded and / or connected), a 3D display, a transparent display, a projector, and / or a head-up display). The electronic device 700 displays one or more interface objects on the display 700a, such as a display login user interface 702. In some embodiments, the login user interface 702 corresponds to a virtual interface object in an XR environment. The login user interface 702 optionally includes various display fields, such as a username field 706 and a password field 708. The login user interface 702 also optionally includes affordances 710 (e.g., "autofill") for initiating a secure action process, such as authorizing various displayed fields. In some embodiments, the secure action corresponds to payment, such as providing credit card information to an e-commerce website to purchase shoes (as described in more detail with respect to Figure 7N), or providing payment information to a third-party service to purchase an item. In some embodiments, the secure action corresponds to providing access credentials to access an application. For example, the secure action provides access credentials to an application associated with a third-party service (e.g., a stock trading application) that requires user authentication before accessing that application. In some embodiments, the electronic device 700 allows a user 704 to manually enter information into fields 706 and / or 708 (e.g., using a displayed keyboard and / or using voice commands).

[0133] In some embodiments, as shown in Figure 7A, the electronic device 700 detects a selection of the affordance 710 by the user 704 to initiate an autofill function that automatically fills in information in fields 706 and / or 708. For example, the display 700a is a touch-sensitive display, and the electronic device 700 is configured to detect touch input (e.g., tap or tap-and-hold) on the affordance 710. In some embodiments, when the device 700 is implemented in an XR environment, the electronic device 700 optionally detects a selection of the affordance 710 using a controller communicatively coupled to the electronic device 700. In addition, the electronic device 700 detects a selection of the affordance 710 using gaze, for example, by detecting that the user 704 is looking at the affordance 710 for a predetermined period of time and / or (optionally) while detecting one or more gestures. For example, the electronic device 700 optionally corresponds to a head-mounted display, and the electronic device 700 is configured to detect that user 704 is performing one or more gestures, such as air gestures, to activate affordances 710. The electronic device 700 optionally displays additional / other elements on the display 700a, such as additional user interfaces (e.g., email applications), representations of other users (e.g., video calls, via avatars of other users in the XR environment), and / or representations of the physical environment, such as user 704's physical environment.

[0134] In Figure 7B, in response to the activation of affordance 710, the electronic device 700 displays the authorization interface 712. In some embodiments, the authorization interface 712 corresponds to a displayed virtual interface object in the XR environment. The authorization interface 712 includes a search function 714 and one or more credential options 716a and 716b. Credential option 716a is associated with information such as a username "janeapples@mail.com" for logging into the website "abc.com" (for example, used for selection). In some embodiments, the electronic device 700 detects that a credential option such as credential option 716b (shown in Figure 7B) has been selected. Credential option 716b is associated with information such as a username "janeapples@mail.com" for logging into the website "xyz.com". In some embodiments, the credential option is associated with a separate password, which may be displayed as part of the authorization interface 712 or may not be displayed for privacy reasons. In some embodiments, the electronic device 700 receives information (e.g., text and / or audio) via the search function 714 and, accordingly, displays one or more search result options corresponding to the received information for use with the autofill function.

[0135] In some embodiments, in order to enable user authorization of secure operations, various requirements (a set of one or more criteria) must be met for the electronic device 700, including visibility criteria and / or user authentication criteria. For example, the visibility criterion is met when a threshold amount is visible from the user 704's perspective to one or more interface objects, such as the login user interface 702 and / or the authorization interface 712 (for example, the threshold amount may correspond to any part of the interface object, a non-zero amount of the interface object, or the entire interface object). In addition, the user authentication criterion is met when the electronic device 700 receives a request to perform a secure operation. For example, the electronic device 700 detects the user 704's selection of a displayed individual credential option, such as credential option 716b (as shown in Figure 7B). Once a credential option is selected, the user 704 may provide an input (e.g., a double press of a hardware button) to authorize the execution of the secure operation, as will be described in more detail with respect to Figure 7D. Upon determining that input has been received that permits the execution of a secure operation, the electronic device 700, provided that one or more of its set of criteria are met, uses its individual credential options to automatically populate fields such as the displayed username field 706 and password field 708 with individual credential information. In Figure 7B, the electronic device 700 receives the selection of credential option 716b.

[0136] In Figure 7C, upon receiving the selection of credential option 716b, the electronic device 700 updates the authorization interface as shown in Figure 7C. In Figure 7C, the electronic device 700 detects a user gaze direction 718 related to (e.g., corresponding to) the user 704's gaze direction. Typically, the user gaze direction 718 is not part of the displayed user interface of the electronic device 700, but is provided to aid in illustrating this technique. While the user 704 is interacting with the device 700, the electronic device 700 detects a change in the user 704's viewpoint (e.g., a change from a first viewpoint to a second viewpoint different from the first viewpoint) based on the change in the user gaze direction 718. In some embodiments, the user gaze direction is detected within the context of the XR environment (e.g., via a head-mounted display). The user gaze direction 718 optionally indicates that the user 704's gaze is not directed towards the login user interface 702 or the authorization interface 712. Therefore, the electronic device 700 determines that the login user interface 702 and the authorization interface 712 are sufficiently invisible from the user 704's viewpoint. For example, the user's line of sight direction 718 may be directed towards a portion of the display 700a that is not close to the location where the login user interface 702 and the authorization interface 712 are displayed. In some embodiments, portions of the login user interface 702 or the authorization interface 712 that are within a predetermined distance (e.g., a predetermined number of pixels and / or a predetermined length) from the center of the user's line of sight direction 718 are determined to be visible from the user 704's viewpoint. For example, 10% of the authorization interface 712 may be determined to be within a predetermined distance from the center of the user's line of sight direction 718, and 5% of the login user interface 702 may be determined to be within a predetermined distance from the center of the user's line of sight direction 718. In this example, the electronic device 700 determines that the amount of the login user interface 702 and / or the authorization interface 712 that is visible from the user's viewpoint is less than a threshold amount (e.g., a 75% threshold amount or a 90% threshold amount).

[0137] In Figure 7C, the electronic device 700 disables and / or deactivates user authorization for the autofill function based on a determination that the amount of the login user interface 702 and / or authorization interface 712 visible from the user's perspective is below a threshold. In some embodiments, the electronic device 700 modifies the appearance of one or more interface objects to indicate that the user cannot request the execution of a secure action (e.g., by graying out the authorization interface 712). For example, in response to detecting the selection of credential option 716b, the electronic device 700 does not automatically fill in the username field 706 and password field 708 while the login user interface 702 and authorization interface 712 are not sufficiently visible from the user 704's perspective (e.g., based at least on the user's line of sight direction 718).

[0138] In Figure 7D, the electronic device 700 determines that user 704 is authorized to perform a secure operation, and therefore, user authorization for the autofill function is enabled. In some examples, user authorization for a secure operation is enabled on the condition that user 704 is authorized to perform a secure operation. For example, the electronic device 700 determines that user 704 is authorized to perform a secure operation based on gaze criteria or other biometric criteria. The gaze criteria are optionally met when user 704 gazes at individual interface objects such as the login user interface 702 and / or the authorization interface 712. Optionally, the login user interface 702 and / or the authorization interface 712 are displayed in the central portion of the display 700a. In some embodiments, the electronic device 700 determines that user 704 is not authorized to perform a secure action when the user is not gazing at the login user interface 702 and / or the authorization interface 712 (for example, the user's eyes are closed, or the user is looking at the corner of the display 700a (away from the interfaces 702 and / or 712)). In some embodiments, user authorization for a secure action is enabled on the condition that the user has or provides a specific movement of a certain biometric feature. For example, such a specific movement may be the user gazing at a specific part of the login user interface 702 (e.g., affordances 710) or a specific part of the authorization interface 712 (e.g., a specific credential option), the user providing a specific facial rotation, and / or the user moving their fingers along a specific path.

[0139] Typically, the electronic device 700 operates in various modes. For example, the electronic device 700 may optionally operate in a first mode, such as a standard mode in which accessibility features are not enabled, and / or a mode in which secure operation is permitted using a default physical input mechanism, such as a hardware button 722 (as described with respect to Figure 7D). In some embodiments, the device 700 may optionally operate in a second mode, such as an accessibility mode, and / or a mode in which input via a physical input mechanism is not required to permit secure operation (as described in more detail with respect to Figures 7L to 7M). Returning to Figure 7D, the electronic device 700 enables user permission for the autofill function, based on the determination that at least a threshold amount of the login user interface 702 and / or the permission interface 712 is visible from the user 704's viewpoint. For example, the user's line of sight 718 may be directed towards the permission interface 712. Therefore, it is determined that the user's line of sight is the authorization interface 712, and that 95% or 100% of the authorization interface 712 is visible from the user 704's viewpoint. As a result, the electronic device 700 determines that at least a threshold amount (e.g., a 75% threshold amount or a 90% threshold amount) of the login user interface 702 and / or the authorization interface 712 is visible from the user's viewpoint. In some embodiments, the electronic device 700 then enables user authorization for secure operation and indicates to the user that user authorization for secure operation is enabled via an indication 720 that includes the text instruction “Double-click to authorize”. Specifically, the indication 720 notifies the user that the autofill function is authorized by double-pressing a hardware button, such as a hardware button 722. In some embodiments, user authorization for the autofill function remains enabled for a certain period (e.g., until 1 second, until 3 seconds, etc.) after it is determined that the login user interface 702 and / or authorization interface 712 are visible from the user's perspective, and optionally is disabled after that period.In some embodiments, device 700 visually indicates that the viewpoint includes the authorization interface 712 and therefore its individual authorization is enabled (for example, by de-graying out the authorization interface 712 and / or buttons).

[0140] If the autofill function is enabled, user 704 may proceed to enable secure operation. Specifically, the electronic device 700 may optionally modify the appearance of one or more interface objects and / or display one or more additional interface objects to indicate to the user that the user can request the execution of secure operation. For example, if the electronic device 700 is implemented in an XR environment, the electronic device 700 may display indications such as commands for performing one or more air gestures (e.g., instructions to move one or more hands or fingers in a specific motion) and / or commands for performing one or more inputs on a controller adapted to the XR environment.

[0141] In Figure 7D, the electronic device 700 receives user permission for the autofill function via a double-click of the hardware button 722. Upon receiving user permission for the autofill function, optionally, a determination is made as to whether a set of one or more execution criteria is met. For example, visibility may need to be met before, when, and / or after receiving a user permission request to perform a secure action. In some embodiments, one or more execution criteria may optionally include the requirement that, after the user 704 requests the performance of a secure action, at least a threshold amount of the login user interface 702 and / or the authorization interface 712 is visible from the user 704's perspective. In some embodiments, the set of one or more execution criteria includes a user authentication criterion that is met when the user is authenticated (e.g., via password and / or biometric authentication). For example, as described in more detail with respect to Figure 7F, the criteria may optionally include a positive indicator of a first level (e.g., perfect or high fidelity) of iris feature verification, a positive indicator of continuous (e.g., a second level) iris feature verification from the time user 704 began interacting with device 700, and / or a positive indicator of proper passcode authentication. As a result, upon receiving a request to perform a secure operation, the electronic device 700 initiates the execution of a secure operation according to the determination that one or more sets of execution criteria are met.

[0142] In Figure 7E, the electronic device 700 provides user 704 with an indication that user authorization for the autofill function has been successful. Specifically, the electronic device 700 modifies the authorization interface 712 to include a positive indicator (e.g., a check mark next to the text "Done") that user authorization for the autofill function has been successful (otherwise, it is replaced with an additional interface). In addition, the electronic device 700 updates the login user interface 702 to indicate that user authorization for the autofill function has been successful by updating the username field 706 to include the username of the username corresponding to the selected credential option, and updating the password field 708 to include masking characters that hide the password corresponding to the selected credential option. In some embodiments, the electronic device 700 updates the password field 708 to include readable characters without masking the password corresponding to the selected credential option.

[0143] Figure 7F shows the user authentication process. Specifically, following Figure 7D, in response to receiving a request to perform a secure action, a determination is made as to whether user 704 meets the user authentication criteria in order to perform the secure action. In some embodiments, it is determined that the set of execution criteria is not met. Typically, a first type of biometric authentication is performed at a first time (e.g., the first time of device interaction), so that the electronic device 700 biometrically authenticates user 704 using one or more biosensors of the device 700. The first time may correspond to the first time when the user first activates the electronic device 700 and / or the first time when the user attaches the electronic device 700 to a part of the user's body. In embodiments where device 700 corresponds to a head-mounted display, the first time may correspond to the time when user 704 fixes / attaches the electronic device 700 to user 704's head. The first type of biometric authentication may include a first level (e.g., full or high fidelity) of iris feature verification. The first level of iris feature verification includes a retinal scan to obtain one or more images and / or measurements of at least one eye of user 704. In some embodiments, different users will have different iris features. The one or more images and / or measurements are then compared to one or more stored images and / or measurements corresponding to the user's registered retinal scans in order to determine the match and / or similarity between the obtained images / measurements and the registered images / measurements. If the obtained images / measurements and the registered images / measurements match with sufficient similarity (e.g., 95% or 99% similarity), the electronic device 700 determines that user 704 is authenticated. In some embodiments, the first type of biometric authentication optionally includes fingerprint verification, facial recognition verification, and / or voiceprint verification (in addition to or instead of iris feature verification).

[0144] The user authentication criteria used to determine whether to perform secure operation are optionally based on sensor measurements taken at multiple intermediate time points during a specific time frame, specifically after the initial time when user 704 began interacting with the electronic device 700 and / or after user 704 has worn the electronic device 700. The sensor measurements are optionally based on measurements taken to detect whether the same user (e.g., user 704) is using the device 700 throughout the entire duration of the interaction. For example, in an embodiment where the electronic device 700 corresponds to a head-mounted display, optionally the electronic device 700 authenticates user 704 via a first type of biometric authentication (e.g., full or high fidelity) when user 704 attaches / wears the electronic device 700 on user 704's head. If authentication via the first type of biometric authentication is successful, the electronic device 700 repeatedly takes sensor measurements using a second type of biometric authentication. In some embodiments, the first type of biometric authentication corresponds to high-fidelity biometric verification, so that the electronic device 700 analyzes characteristics related to the entire eye or a portion of the user 704's eye, such as iris features. In some embodiments, the second type of biometric authentication corresponds to one or more sensor measurements that directly or indirectly confirm whether a biometric feature has been continuously present since high-fidelity biometric verification was performed for that biometric feature. The second type of biometric authentication optionally analyzes the same (or similar) amount of biometric features as the first type of biometric authentication, or optionally analyzes fewer biometric features compared to the first type of biometric authentication. In some embodiments, the second type of biometric authentication is used to verify that a user authenticated using the first type of biometric authentication has remained a user of the electronic device 700 since the first type of biometric authentication was performed (e.g., without interruption of the continuity of use of that biometric feature), so in some embodiments, the second type of biometric authentication may be referred to as continuity verification.

[0145] In some embodiments, both the first and second types of biometric authentication are enabled while the electronic device 700 is operating in a first mode (e.g., a standard mode in which accessibility features are not enabled, and / or a mode that allows secure operation using a default physical input mechanism). In some embodiments, while the electronic device 700 is operating in a second mode (e.g., an accessibility mode, and / or a mode in which input at a physical input mechanism is not required to allow secure operation), the first type of biometric authentication is enabled and the second type of biometric authentication is disabled. Thus, in some embodiments, while operating in a second mode (as described in more detail with respect to Figures 7L and 7M), the first type of biometric authentication (e.g., full or high fidelity) is optionally required in response to receiving a request to perform a secure operation.

[0146] In some embodiments, at least one (e.g., the same or different) biosensor is used to perform both a first type of biometric authentication and a second type of biometric authentication (e.g., a retinal scanner, a depth camera, and / or a proximity sensor).

[0147] For example, the second type of biometric authentication optionally focuses on detecting features and characteristics of user 704, such as pupil, eye glint, and iris features (e.g., shape, size, and / or color pattern). Specifically, electronic device 700 captures information about the appearance of the user's eyes and records information about the detected pupil, eye glint, and / or iris features. Electronic device 700 then captures information about the appearance of the user's eyes again at a later time (e.g., 1 second later, 2 seconds later, etc.) and stores information about the detected pupil, eye glint, and / or iris features. A comparison is made between the information recorded at the first time and the information recorded at the second time to verify that the same user (e.g., user 704) was using the device at both times. For example, if the first detected pupil, eye glint, and / or iris features match (e.g., perfectly and / or with sufficient confidence) the second detected pupil, eye glint, and / or iris features, the electronic device 700 determines that the same user used the device at both times and therefore the respective criteria remain met. This process may be repeated up to a specific time, such as when the user provides a request to perform a secure action. If, at this specific time, it has been determined (since the first type of biometric authentication) that the same user did not use the device at any of the given times (e.g., the first detected information did not match the second detected information), then the respective criteria are determined to be not met, and as a result, the execution of the secure action is canceled (e.g., until further authentication is performed). For example, if sensor measurements indicate that the presence of an eye, iris features, pupil, and / or eye glint was not detected, then the respective criteria are determined to be not met. The absence of eye features may be based on the user's eyes being closed or the user's eyes being undetectable by the sensor (for example, the head-mounted display being improperly fitted). For example, in an embodiment where the electronic device 700 corresponds to a head-mounted display, the user may have removed the head-mounted display.In embodiments where the electronic device 700 corresponds to a tablet device or smartphone device, the user 704 may hold the device 700 so that the user 704's eyes are not within the field of view of the sensor (for example, the electronic device 700 is placed face down on a certain surface). In other words, each criterion optionally includes an “eye open” criterion that is met when at least one of the user 704's eyes is not continuously closed for a threshold period (for example, a non-zero period such as one or two seconds).

[0148] In some embodiments, the first type of biometric authentication is based on different biometric features than the second type of biometric authentication. Specifically, optionally, the first type of biometric authentication corresponds to full iris feature verification, so that the electronic device 700 analyzes features related to the entire eye of user 704 or a part of user 704's eye, such as the pupil, eyelid, and / or iris features. The second type of biometric authentication may correspond to partial facial recognition verification. Partial facial recognition optionally includes analyzing features related to parts of the face surrounding (and excluding) the eye, such as the eyebrows, eyelids, skin around the eye, forehead, cheeks, and / or nose of user 704. For example, the electronic device 700 captures the appearance of the eye-surrounding portion of user 704's face. At a later time (e.g., 1 second or 2 seconds later), the electronic device 700 captures the appearance of the same eye-surrounding portion of user 704's face again. The electronic device 699 compares the information recorded at the first time with the information recorded at the second time to determine if the same user (e.g., user 704) was using the device at both times. For example, if the first detected eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose match (e.g., completely or with sufficient confidence) the second detected eyebrows, eyelids, skin surrounding the eyes, forehead, cheeks, and / or nose, then it is determined that the same user was using the device at both times, and therefore the respective criteria remain met. The process may be repeated until a specific time, such as when the electronic device 700 receives a request from the user to perform a secure operation. If, at this specific time, it is determined that the same user was not using the device since the first type of authentication (e.g., the first detected information did not match the second detected information), then the respective criteria are not met, and as a result, the electronic device 700 stops performing the secure operation (e.g., until further authentication is performed). In other words, each criterion may optionally include non-eye criteria that can be met even if the user 704's eyes are undetectable (e.g., undetectable over a threshold period of 1, 5, 10, 15, or 60 seconds).

[0149] In some embodiments, authentication is improved by using a first biometric feature for a first type of biometric authentication and another biometric feature for a second type of biometric authentication. For example, if eye features are analyzed for a first type of biometric authentication and facial features are analyzed for a second type of biometric authentication, authentication is improved when eye features are not always available for analysis. Specifically, user 704 is initially authenticated based on eye features, but facial features (e.g., eyebrows, eyelids, skin around the eyes, forehead, cheeks, and / or nose) are used for the second type of biometric authentication. In this case, even if user 704's eyes cannot be detected by one or more sensors of the electronic device 700, the electronic device 700 can still detect the user's facial features and perform the second type of authentication, allowing the secure operation to be performed.

[0150] Returning to Figure 7F, upon receiving a request to perform a secure action, the electronic device 700 determines whether user 704 meets the respective authentication criteria for performing the secure action. In some embodiments, user 704 is biometrically authenticated at a first time, so the request to perform a secure action is received at a second time (after the first time). In this case, if it is determined that user 704 did not meet the respective criteria between the first and second times, the authentication guidance user interface 724 is provided to the user to biometrically authenticate user 704 using the first type of biometric authentication. In other words, if the electronic device 700 cannot verify that the same user has been using the device since the first first type of biometric authentication (or can verify that the same user has not been using the device since the first first type of biometric authentication), the electronic device 700 attempts to verify the user's identity again using the first type of biometric authentication. Failure to meet the respective authentication criteria can be the result of various factors. For example, the eyes of user 704 may be undetectable during the period between a first time and a second time. In some embodiments, initial authentication may be based on another type of biometric authentication (e.g., fingerprint authentication) or on non-biometric authentication (e.g., passcode entry). In certain initial authentications, such as fingerprint authentication or passcode authentication, the respective authentication criteria are not automatically met, so in order to allow secure operation, user 704 is required to provide a specific first type of biometric authentication (e.g., full or high-fidelity iris feature verification). In some embodiments, the authentication guidance user interface 724 includes instructions for performing the first type of biometric authentication, such as instructions that require the user to provide specific movements of individual biometric features. For example, the authentication guidance user interface 724 may include an initial prompt such as "Biometric authentication required," with an option for the user to "continue" and another option for the user to "cancel." In Figure 7F, the electronic device 700 detects that the "continue" option has been activated.

[0151] In Figure 7G, upon detection of the activation of the "Continue" option, the electronic device 700 initiates a first type of biometric authentication. When the first type of biometric authentication corresponds to iris feature verification, the electronic device 700 instructs the user to look at different parts of the display when biometric authentication begins (for example, via the prompt "Look here"). During the first type of biometric authentication, the electronic device 700 optionally analyzes the gaze 726 corresponding to user 704 to obtain characteristics related to user 704's entire eye or a part of user 704's eye, such as iris features, pupil, and / or glint. When the first type of biometric authentication corresponds to face verification, the electronic device 700 instructs user 704 to rotate their face at a specific angle. In some embodiments, when the first type of biometric authentication corresponds to fingerprint authentication, the electronic device 700 instructs the user to move their finger along a specific path and / or touch the touch-sensitive contact area of ​​device 700 or another device in a specific manner.

[0152] Figure 7H shows a passcode input prompt 728. In some embodiments, the user authentication criterion is satisfied when the passcode input matches a registered passcode entered via the passcode input prompt 728. For example, the electronic device 700 is optionally configured so that user authentication (or user authentication for performing secure operations) is performed via a passcode or password (e.g., the user has disabled iris feature verification, facial verification, and / or other types of biometric authentication). Alternatively, in accordance with a determination that the user authentication criterion is not satisfied, based on a failure of the first type of biometric authentication (e.g., the device 700 was unable to successfully complete the first type of biometric authentication), the electronic device 700 displays the passcode input prompt 728 to provide an additional attempt to satisfy the user authentication criterion. In some embodiments, it may be determined that the initial first type of biometric authentication was successful (e.g., the user successfully passed biometric authentication when first accessing the device), but the second type of biometric authentication for detecting continuity may have failed (e.g., the user's eyes were undetectable for a period of time after the initial time). In this case, the electronic device 700 attempts to authenticate the user by displaying a passcode input prompt 728 to the user.

[0153] In some embodiments, if continuity verification fails, the electronic device 700 first attempts to authenticate the user by means other than passcode verification before attempting passcode verification. For example, the electronic device 700 attempts a first type of biometric authentication, such as full or high-fidelity iris feature verification (e.g., as shown in Figure 7G), or another type of full or high-fidelity biometric authentication, such as fingerprint authentication and / or facial recognition authentication. If full or high-fidelity biometric authentication fails, passcode verification is used as the default option, optionally. In some embodiments, the passcode input prompt 728 includes a passcode input portion and an affordance for rejecting passcode input. If the user selects the option to reject passcode input, user authorization for secure operation is revoked.

[0154] Figure 7I, following Figure 7D, shows the change in the user 704's viewpoint. Typically, the visibility of interface objects can change based on the user 704's viewpoint. In some embodiments, for example, based on the fact that parts of the login interface and authorization interface are not fully displayed on the display 700a, at least parts of the login user interface 702 and authorization interface 712 are invisible to the user 704. In some embodiments, when the electronic device 700 is implemented in an XR environment, certain user motions optionally make the interface objects invisible or partially invisible. For example, while wearing the device 700 (e.g., a head-mounted display), the user turns their head in a particular direction. The login user interface 702 and authorization interface 712 optionally become environment-locked within the environment so that their interfaces remain in the same overall location within the environment regardless of the user's head movements or other motions. Therefore, at least a portion of the interface is optionally invisible to user 704 based on the interface being positioned to the side of user 704's field of view (or behind the user in the XR environment). Consequently, the visible amount of the login user interface 702 and the authorization interface 712 may be reduced based on user 704's movement. As a result, it is determined that the amount of the login user interface 702 and / or the authorization interface 712 visible from user 704's viewpoint is less than a threshold amount. Based on this determination, the electronic device 700 modifies the appearance of the authorization interface 712, for example, to indicate that user 704 cannot request the execution of a secure operation.

[0155] In Figure 7J, an exemplary environment, including a background area 730, is depicted on the display 700a. In some embodiments, the environment is implemented in the context of an XR environment, so the electronic device 700 corresponds to a head-mounted display. The background area 730 optionally corresponds to a representation of the user 704's physical environment, virtual environment, or a combination of the physical and virtual environments. The user 704 can interact with various interface objects, virtual objects, or other objects depicted within the environment. For example, the electronic device 700 displays a messaging interface 732 through which the user 704 can send and receive messages. In addition, the electronic device 700 also displays a login user interface 702 and an authorization interface 712. While the electronic device 700 detects the user 704 interacting with a particular interface, that interface may be fully visible to the user 704 so that it is not obscured by objects or other elements in the environment. The electronic device 700 optionally detects that user 704 is currently looking at the authorization interface 712 as depicted by the user's line of sight 718. Optionally, as described herein, user authorization for secure operation is enabled. Thus, the appearance of the authorization interface 712 (e.g., not grayed out) and the indication 734 inform the user that they can authorize the autofill function by performing the function indicated by the indication 734 (e.g., double-pressing button 722).

[0156] Figure 7K shows an exemplary environment including a change in the user 704's viewpoint. In some embodiments, the visibility of objects depicted in the environment changes based on occlusion by other objects in the environment. Such occlusion occurs based on the movement of the object itself (which is occluded or occluding) or a change in the user 704's viewpoint. For example, in relation to Figure 7J, the electronic device 700 detects that the user 704 has turned left in the environment. Specifically, if the environment is implemented in the context of an XR environment, the electronic device 700 detects that the user 704 has turned their head to the left while wearing the electronic device 700 (e.g., a head-mounted display). Various objects in the environment may be viewpoint-locked or environment-locked. Specifically, the messaging interface 732 may be viewpoint-locked so that the messaging interface 732 appears at the same location and / or position in the user 704's viewpoint even when the user 704's viewpoint changes. Alternatively, the login user interface 702 and authorization interface 712 may be environment-locked, so that they appear in a position within the user's viewpoint that is based on their location within the XR environment (for example, in front of a sofa representation). Therefore, when user 704 adjusts their viewpoint to the left (relative to Figure 7J), the messaging interface 732 appears to move over the login user interface 702 and authorization interface 712. In other words, the login user interface 702 and authorization interface 712 remain in the same overall location relative to the background of the XR environment from the user's viewpoint, and as a result of the user's viewpoint shift, the messaging interface 732 is overlaid on top of the login user interface 702 and authorization interface 712. When the overlay by the messaging interface 732 occurs based on the change in viewpoint, the visibility of the login user interface 702 and authorization interface 712 is reduced.The electronic device 700 detects that the user's line of sight 718 remains directed towards the authorization interface 712, but determines that the amount of the login user interface 702 and / or authorization interface 712 visible from the user's viewpoint is less than a threshold. Therefore, the appearance of the authorization interface 712 is adjusted to reflect that user authorization for secure operation has been disabled (e.g., it is grayed out).

[0157] In Figure 7L, the electronic device 700 displays an interface for allowing secure operation while the electronic device 700 is operating in a second mode different from the first mode (described with respect to Figures 7A to 7K). In some embodiments, the electronic device 700 operates in a second mode, such as an accessibility mode and / or a mode in which input at a physical input mechanism is not required to allow secure operation. While in the second mode, the electronic device 700 displays a login user interface 736. The login user interface 736 corresponds to a login interface that includes functionality to facilitate interaction for users with accessibility requirements (e.g., limited fine motor skills, vision, and / or hearing abilities). The electronic device 700 also optionally displays an authorization interface 738 (e.g., an "autofill" function) that facilitates processes for secure operation, such as authorizing information into certain displayed fields.

[0158] In some embodiments, the login user interface 736 and the authorization interface 738 correspond to displayed virtual interface objects in the XR environment. The authorization interface 738 includes a search function 742 and one or more credential options 744a and 744b. In some embodiments, the credential options are associated with a username and / or password. In some embodiments, the electronic device 700 receives a search query for additional credential options via the search function 742 and provides any retrieved result options based on the search. While in the second mode, the user 704 may interact with the authorization interface 738 using an input or combination of inputs such as eye-tracking, controller input, and / or voice input that conforms to the accessibility features (e.g., selecting either credential option 744a or 744b, or searching for additional credential options via the search function 742). In some embodiments, the electronic device 700 detects the selection of a certain credential option (e.g., credential option 744a). After the credential option is selected, user 704 may activate affordance 740 using an input or combination of inputs such as eye-tracking, controller input, and / or voice input, which are compatible with accessibility features. Specifically, affordance 740 may be displayed with the text “Confirm with AssistiveTouch” to indicate to the user that the device is operating in the second mode, and therefore no physical input mechanism is required to allow autofill operation. In Figure 7L, the electronic device 700 detects the selection of affordance 740.

[0159] In Figure 7M, upon detection of the selection of affordance 740, the electronic device 700 displays an assistive input interface 742 containing representations of multiple functions to facilitate various tasks. In some embodiments, user 704 needs to gaze at affordance 740 while activating it in order to induce the selection of affordance 740. For example, user 704 gazes at affordance 704 and activates a special input while gazing at affordance 740 (described in more detail in paragraphs

[0272] to

[0273] ). In addition, in response to the user selection of affordance 740, the electronic device 700 also modifies the appearance of the authorization interface 738 to indicate that the credential option has been selected by the user. In some embodiments, the position or location of the authorization interface 738 is adjusted (for example, in an XR environment, the authorization interface 738 is moved to the side of the user's field of view).

[0160] The multiple functions included in the assistive input interface 742 include a variety of options. For example, the various options include an option to navigate to the main screen (e.g., a “Home” option), an option to display device notifications (e.g., “Notifications”), and / or an option to display a settings menu (e.g., “Control Menu”). The assistive input interface 742 includes a secure operation permission affordance 744. The multiple functions can be optionally invoked using a specific input type, which may be pre-configured by the electronic device 700 or a specific user. For example, while in the first mode, the option to navigate to the main screen is executed in response to the electronic device 700 receiving a primary input type (e.g., a “swipe” input at a specific location), while while in the second mode, the option to navigate to the main screen is executed in response to the electronic device 700 receiving an accessibility-based input type (e.g., the user looking at the “Home” icon in combination with a long press of the controller). In addition, while in the first mode, enabling secure operation is performed in response to the electronic device 700 receiving a primary input type such as a double press of a specific hardware button, while while in the second mode, enabling secure operation is performed by accessibility-based input types such as activating a secure operation enablement affordance (for example, the user gazing at the secure operation enablement affordance 744 in combination with a long press of the controller).

[0161] In some embodiments, in response to receiving an input to authorize secure operation, the electronic device 700 performs user authentication. For example, a first type of biometric authentication is performed, so that user 704 is biometrically authenticated using one or more biosensors of device 700. The first type of biometric authentication optionally includes a first level (e.g., full or high fidelity) of iris feature verification. Specifically, while operating in a second mode, the second type of biometric authentication (e.g., passive iris feature verification, in which the electronic device 700 repeatedly analyzes specific characteristics related to the entire eye or a portion of the eye of user 704 while user 704 is interacting with device 700) is disabled. Thus, when an input to authorize secure operation is received, the first type of biometric authentication is required. In particular (referring back to Figure 7F), the electronic device 700 provides the user with an authentication guidance user interface to biometrically authenticate user 704 using the first type of biometric authentication. The authentication guidance user interface includes instructions for performing a first type of biometric authentication, such as instructions that require the user to provide specific movements of individual biometric features. For example, the authentication guidance user interface includes an initial prompt such as "Biometric authentication required," along with options for the user to "continue" and another option for the user to "cancel."

[0162] When the electronic device 700 detects that the option to "continue" has been activated, the electronic device 700 initiates a first type of biometric authentication. If the first type of biometric authentication corresponds to iris feature verification, the electronic device 700 optionally instructs the user to look at different parts of the display when biometric authentication begins (for example, via the prompt "Look here"). During the first type of biometric authentication, the electronic device 700 analyzes the gaze corresponding to user 704 to obtain characteristics related to user 704's entire eye or part of user 704's eye, such as iris features, pupil, and / or glint. If the first type of biometric authentication corresponds to face verification, the electronic device 700 instructs user 704 to rotate their face at a specific angle. In some embodiments, if the first type of biometric authentication corresponds to fingerprint authentication, the electronic device 700 instructs the user to move their finger along a specific path and / or touch the touch-sensitive contact area of ​​device 700 or another device in a specific manner. In some embodiments, the first type of biometric authentication is based on another type of biometric authentication (e.g., voice authentication) or on a non-biometric authentication (e.g., passcode entry).

[0163] In Figure 7N, an additional exemplary user interface for enabling secure operation is displayed on the display 700a of device 700. Specifically, Figure 7N illustrates a secure operation related to payment. In some embodiments, the secure operation related to payment is implemented in the context of an XR environment, so device 700 corresponds to a head-mounted display. The electronic device 700 receives user input and, accordingly, navigates to an e-commerce website displayed via a web browser 748 to purchase an item (e.g., shoes). In some embodiments, when the electronic device 700 is operating in a first mode (e.g., a standard mode with accessibility features not enabled, and / or a mode that enables secure operation using a default physical input mechanism), the electronic device 700 detects the selection of an affordance 750 and initiates the payment function. For example, since the display 700a is a touch-sensitive display, user 704 presses the affordance 750 with their finger. In some embodiments, when device 700 is implemented in an XR environment, the electronic device 700 detects the selection of the affordance 750 using a controller communicatively coupled to the electronic device 700. In addition, the electronic device 700 detects the selection of an affordance 750 using gaze, for example, by detecting that the user 704 is looking at the affordance 750 for a predetermined period of time and / or has received one or more gestures. In some embodiments, the electronic device 700 corresponds to a head-mounted display, so the electronic device 700 detects that the user 704 is performing one or more gestures to activate the affordance 750, such as an air gesture. Once the electronic device 700 detects the activation of the affordance 750, the electronic device 700 displays a payment details interface 752. The payment details interface 752 includes details regarding information for completing an order to purchase an item displayed in the associated web browser 748. For example, the payment details interface 752 includes credit card billing information, shipping information, and / or pricing information.

[0164] In some embodiments, the electronic device 700 enables user authorization for a payment action on the condition that it determines that the user 704 is authorized to perform the payment action. For example, the user is determined to be authorized to perform the secure action based on a gaze criterion with respect to the user's line of sight 754. The gaze criterion is met when the user 704 is gazing at an interface object such as the web browser 748 and / or the payment details interface 752. For example, the electronic device 700 optionally displays the web browser 748 and / or the payment details interface 752 in the central portion of the display 700a. In some embodiments, the electronic device 700 determines that the user is not authorized to perform the secure action when the user is not gazing at the web browser 748 and / or the payment details interface 752 (e.g., the user's eyes are closed, or the user is looking at the corner of the display 700a (away from the interfaces 748 and 752)). In some embodiments, user authorization for a secure action is enabled on the condition that it determines that the user 704 has or is providing a specific movement of a certain biometric feature. For example, such a particular movement may include the user gazing at a specific part of the web browser 748 (e.g., looking at an affordance 750) or a payment details interface 752 (e.g., a displayed price), the user providing a specific facial rotation, or the user moving their finger along a specific path.

[0165] Once the payment function authorization is enabled, the user proceeds to authorize the payment function. Specifically, the electronic device 700 modifies the appearance of one or more interface objects and / or displays one or more additional interface objects to indicate to the user that the user can request to perform a payment action. Indication 756 includes the instruction “Double-click to authorize,” which may be displayed adjacent to a hardware button 758. In some embodiments, if the electronic device 700 is implemented in an XR environment, the electronic device 700 displays indications such as instructions for performing one or more air gestures (e.g., instructions for moving one or more hands or fingers in a specific motion) and / or instructions for performing one or more inputs on an XR-compatible controller.

[0166] In some embodiments, upon receiving a request to perform a secure operation, the electronic device 700 determines whether user 704 meets the user authentication criteria for performing the payment operation, as discussed with respect to Figures 7F to 7H. If the user does not meet the user authentication criteria, user authorization for the payment operation is invalidated. Therefore, the user authentication process is performed, as discussed with respect to Figures 7F to 7H. If the user does not meet the user authentication criteria, the electronic device 700 initiates the payment transaction and provides feedback to the user regarding the payment transaction.

[0167] Additional information regarding Figures 7A to 7N is provided below with reference to the method 800 described with respect to Figures 7A to 7N.

[0168] Figure 8 is a flowchart of exemplary method 800 for facilitating user consent for secure operation, according to several embodiments. In some embodiments, method 800 is executed in a computer system (e.g., computer system 101 in Figure 1 (e.g., a smartphone, smartwatch, tablet, and / or wearable device)) which includes display generating components (e.g., display generating components 120 in Figures 1, 3, and 4 (e.g., display controller, touch-sensitive display system, display (e.g., embedded and / or connected), 3D display, transparent display, projector, and / or head-up display)) (e.g., head-up display, display, touchscreen, projector, etc.) and / or one or more input devices. In some embodiments, method 800 is governed by a plurality of instructions, which are stored in a non-temporary (or temporary) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 in Figure 1). Some operations of method 800 are combined as optional choices, and / or the order of some operations is changed as optional choices.

[0169] As will be described later, Method 800 provides an intuitive method for obtaining user consent for secure operation. This method reduces the cognitive burden on the user while performing secure operation, thereby creating a more efficient human-machine interface. In the case of battery-powered computing devices, streamlining user consent for secure operation to make it faster and more efficient saves power and extends the time between battery charges.

[0170] While a three-dimensional environment (e.g., 730) containing virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) containing information related to secure operation (e.g., 706, 708, 710, 716a, 716b, 744a, and / or 744b) is visible via display generation components (e.g., 120, and / or 700a), a change in the user's (e.g., user 704) current viewpoint (e.g., from a first viewpoint to a second viewpoint different from the first viewpoint) (802) is detected via one or more input devices (e.g., 125, and / or 150) (e.g., receiving a request (e.g., with a password and / or email information) to autofill an input field, and / or detecting the activation of a payment affordance).

[0171] In response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), the system enables user authorization for secure actions using the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) based on the determination that at least a threshold amount of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are visible from the user's viewpoint (e.g., 718, 746, and / or 754) (e.g., the gaze is directed towards the user interface and / or the object of the user's gaze is the user interface), and that the user (e.g., 704) is authorized to perform secure actions (804) (e.g., enabling the autofill permission option to automatically fill fields in a second user interface (e.g., a form user interface), and / or enabling the purchase permission option). Activation of the option (e.g., a double-press procedure of a physical button). In some embodiments, a computer system (e.g., 700) visually indicates that a viewpoint (e.g., 718, 746, and / or 754) includes a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and therefore an individual authorization option is activated (e.g., by not graying out a part of the second user interface (e.g., 702, 712, 736, 738, 748, and / or 752), such as a button (e.g., 710)). In some embodiments, the activatable purchase authorization options include a single press, a long press, a long press following a single press, a press followed by a rotation, a rotation followed by a press, and / or a series of presses (e.g., a double press, a triple press).

[0172] If the number of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) visible from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) is less than a threshold amount (e.g., the line of sight (e.g., 718, 746, and / or 754) is not directed towards that user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the object of the user's line of sight In accordance with the determination that it is not 02, 712, 736, 738, 748, and / or 752, user permission for secure actions using virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) is revoked (806) (e.g., disabling the autofill permission option to automatically fill in fields in a second user interface (e.g., a form user interface), and / or disabling the activation of a purchase permission option (e.g., the double-press procedure for a physical button)). In some embodiments, upon determining that a viewpoint (e.g., 718, 746, and / or 754) does not include a user interface (e.g., 702, 712, 736, 738, 748, and / or 752), the computer system (e.g., 700) visually indicates that the viewpoint (e.g., 718, 746, and / or 754) does not include a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and therefore the individual permission options are not enabled (e.g., by graying out parts of the second user interface (e.g., 702, 712, 736, 738, 748, and / or 752), such as a button (e.g., 710). Enabling user permission for secure actions when individual user interfaces are visible improves security and privacy by ensuring that users have the opportunity to review information related to the secure action before authorizing its progress.

[0173] In some embodiments, an input to authorize the execution of a secure action (e.g., a double-click of a hardware button (e.g., 722 and / or 758) and / or activation of a specific affordance) is received from the user (e.g., 704). In some embodiments, the input to authorize includes a single press, a long press, a long press following a single press, a press followed by a rotation, a rotation followed by a press, and multiple consecutive presses (e.g., a double press, a triple press). In some embodiments, upon receiving input that permits the execution of a secure action, the execution of a secure action is initiated according to the determination that one or more execution criteria are met (for example, a gaze (e.g., 718, 746, and / or 754) is directed towards a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the object of the user's gaze (e.g., 718, 746, and / or 754) is a user interface (e.g., 702, 712, 736, 738, 748, and / or 752)) in response to receiving input that permits the execution of a secure action (e.g., a user fills in a field in a second user interface (e.g., a form user interface) and / or makes a payment). To initiate a transaction, a set of one or more execution criteria includes a visibility criterion, which is satisfied when at least a threshold amount of a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) is visible from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754). (e.g., the threshold amount corresponds to any part of the interface object (e.g., 702, 712, 736, 738, 748, and / or 752), a non-zero amount of the interface object (e.g., 702, 712, 736, 738, 748, and / or 752), or the entirety of the interface object (e.g., 702, 712, 736, 738, 748, and / or 752)).In some embodiments, the execution of a secure action is not initiated based on the determination that one or more sets of execution criteria are not met (for example, that the user's gaze (e.g., 718, 746, and / or 754) is not directed towards a user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the target of the user's gaze (e.g., 718, 746, and / or 754) is not a user interface (e.g., 702, 712, 736, 738, 748, and / or 752)) (e.g., that the user refrains from filling in fields (e.g., 706 and / or 708) of a second user interface (e.g., a form user interface) and / or refrains from initiating a payment transaction). Initiating secure actions when a threshold amount is visible within individual user interfaces improves security and privacy by ensuring that users have the opportunity to review information related to the secure action before authorizing its progress.

[0174] In some embodiments, a set of one or more execution criteria includes user authentication criteria that are met when a user (e.g., 704) is authenticated (e.g., the criteria include positive indicators of continuous iris feature verification from the time the user (e.g., 704) interacted with the device (e.g., 700), first level (e.g., full or high fidelity) iris feature verification, and / or positive indicators of proper passcode authentication). Requiring user authentication in addition to user authorization for secure operation results in improved security / privacy by adding an additional verification layer specific to the user requesting authorization.

[0175] In some embodiments, biometric information is detected from the user (e.g., 704) (e.g., a first level (e.g., full or high fidelity) iris feature validation is performed and / or a second level (e.g., tracking eye information (e.g., detected glint information and / or detected pupil information) for continuity) iris feature validation is performed). In some embodiments, detecting biometric information includes receiving input (e.g., one or more images of the user's face) from the user (e.g., 704) in response to a prompt (e.g., 724) to gaze at a specific part (e.g., 726) of the screen (e.g., 700a). In some embodiments, detecting biometric information includes passively detecting biometric information while the user (e.g., 704) is using the device (e.g., 700) (e.g., wearing a head-mounted display). In some embodiments, detecting biometric information includes detecting biometric information while a hardware button (e.g., 722 and / or 758) is activated (e.g., during a single press, long press, long press after a single press, rotation after a press, press after rotation, or during multiple consecutive presses (e.g., double press, triple press)), or in response to the activation of a hardware button (e.g., 722 and / or 758). In some examples, the biometric information is compared with registered biometric information (e.g., comparing an acquired iris scan with a stored iris pattern and / or comparing the current eye information with eye information from a previous frame), and the determination of whether the user authentication criteria are met is based on the comparison of the biometric information with the registered biometric information (e.g., if the comparison meets a sufficient similarity threshold (e.g., 95% similarity), the user authentication criteria are met). In some embodiments, a first level (e.g., full or high fidelity) iris feature validation is performed, and a second level (e.g., tracking eye information for continuity) iris feature validation is performed at a later time (e.g., one hour later) to link the first level of iris feature validation to the execution criteria for initiating the execution of secure operation.Requiring biometric authentication in addition to user authorization for secure operation improves security and privacy by adding a user-specific biometric verification layer for the user requesting the authorization.

[0176] In some embodiments, the determination that user authentication criteria are met is made in response to receiving input that permits the execution of a secure action (e.g., activation of hardware buttons (e.g., 722 and / or 758) (e.g., single press, long press, long press after single press, rotation after single press, press after rotation, multiple consecutive presses (e.g., double press, triple press), in response to verification that user authentication criteria are met). Requiring user authentication at the time of the user request to execute a secure action provides an improvement in security / privacy by verifying the identity of the user requesting the action.

[0177] In some embodiments, determining whether a user authentication criterion is met involves retrieving stored credentials, which indicate one or more previously performed biometric authentications (e.g., retrieving a continuity result indicating a positive result (e.g., iris feature continuity and / or facial continuity are maintained) or a negative result (e.g., iris feature continuity and / or facial continuity are not maintained)). Utilizing stored credentials in addition to user authorization for secure operation provides security / privacy improvements by creating an efficient and uninterrupted user experience for biometric authentication, encouraging users to use biometric authentication, and thus providing a more secure experience.

[0178] In some embodiments, the user authentication criterion is to perform a first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed), and after performing the first type of biometric authentication (e.g., after successfully performing the first level (e.g., full or high fidelity) iris feature verification), a second type of biometric authentication (e.g., a second level (e.g., tracking eye information for continuity) iris feature verification is performed), and to receive an input containing first individual biometric information from the user (e.g., 704) (e.g., obtaining eye information from the current frame), and to process the received input into a second individual Determining whether a second type of biometric authentication is performed repeatedly, including comparing it with previously received inputs that include biometric information (e.g., comparing eye information from the current frame with eye information from a previous frame), and performing it every two frames (e.g., every three frames, every four frames, every five frames, every six frames, every seven frames, every eight frames, every nine frames, or every five or one-tenth frame). Obtaining multiple sensor measurements at multiple intermediate time points between the first and second time points improves security / privacy by verifying that the same user has been using the device since the initial authentication.

[0179] In some embodiments, performing a first type of biometric authentication includes detecting the presence of a first type of biometric feature (e.g., a first level (e.g., full or high fidelity) verification based on eye features, or a first level (e.g., full or high fidelity) verification based on facial features), and performing a second type of biometric authentication includes detecting a second type of biometric feature (e.g., a second level of verification based on eye features (e.g., tracking eye information for continuity), or a second level of verification based on facial features (e.g., tracking eyebrows, eyelids, forehead, and / or nose for continuity)), where the first and second types of biometric features correspond to the same type of biometric feature (e.g., a first level of verification and / or a second level of verification based on eye features, or a first level of verification and / or a second level of verification based on facial features). For secure operation, continuously verifying the user's presence based on the same biometric characteristics as the initial authentication, in addition to user authorization, creates an efficient and uninterrupted user experience for biometric authentication, encouraging users to use biometric authentication and thus providing a more secure experience, thereby improving security and privacy.

[0180] In some embodiments, the user authentication criterion is met when the user (e.g., 704) is authenticated based on biometric information of the eye (e.g., a first pattern is detected based on multiple features corresponding to the user's (e.g., 704) eye (e.g., a pattern formed based on the eye's fibers, creases, grooves, corona, and / or ring)). In some embodiments, the detected first pattern is compared to a reference pattern, which is based on multiple features corresponding to the user's eye (e.g., a stored pattern based on full or high-fidelity iris feature verification, or a stored pattern based on partial iris feature verification for continuity). Requiring biometric authentication of the eye in addition to user authorization for secure operation results in improved security / privacy by adding a user-specific biometric verification layer for the user requesting authorization.

[0181] In some embodiments, the user authentication criterion is met when a passcode input matches a registered passcode (e.g., receiving multiple digit inputs (e.g., 0-9) and comparing them to a previously registered passcode (e.g., a passcode containing digits 0-9)). Requiring passcode authentication in addition to user authorization for secure operation results in improved security / privacy by providing an additional method for verifying the user's identity.

[0182] In some embodiments, following a determination that user authentication criteria are not met (e.g., the device cannot obtain a sufficient iris feature scan, the result of a first-level iris feature verification indicates that the iris feature scan does not match a reference scan, or the user refuses to provide iris feature verification), a prompt (e.g., 728) (e.g., a button that the user activates to invoke a passcode input interface) is provided (e.g., displayed via a display generating component (e.g., 120 and / or 700a)). In some embodiments, the prompt (e.g., 704) includes a passcode input portion and an affordance for refusing passcode input. Requiring passcode authentication after biometric authentication failure results in improved security / privacy by providing a backup option for verifying the user's identity.

[0183] In some embodiments, user authorization for secure operations using virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., initiating a payment operation and / or initiating an autofill operation) includes user activation of hardware user interface elements (e.g., 722 and / or 758) (e.g., single press, long press, long press after a single press, press after rotation of the hardware user interface element, press after rotation, two, three, or more consecutive presses). Utilizing hardware elements for authorization results in improved security / privacy by reducing the possibility of false authorization and / or preventing unauthorized input from malicious software.

[0184] In some embodiments, a secure action is a payment (for example, providing payment information to a third-party service to purchase an item, such as providing credit card information to an e-commerce website to purchase shoes). Enabling user permission for payment actions when a separate user interface is visible results in improved security / privacy by ensuring that the user has the opportunity to review information related to the secure action before authorizing the secure action to proceed.

[0185] In some embodiments, secure actions include automatically filling in user credentials (e.g., providing login information (e.g., 706 and / or 708) (e.g., username and / or password) to access secure information (e.g., banking websites, stock trading applications, and / or user profile information)). Enabling user permission for autofill actions when individual user interfaces are visible results in improved security / privacy by ensuring that users have the opportunity to review information related to secure actions before authorizing the secure action to proceed.

[0186] In some embodiments, in response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), the amount of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) visible to the user (e.g., 704) changes (e.g., as a result of a viewpoint shift (e.g., looking left or right), the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are partially (e.g., as shown in Figures 7I and 7K) or completely removed from the field of view, or, as a result of a viewpoint shift, the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) become visible after being completely removed from the field of view). Enabling user permission for secure behavior based on changes in the user's viewpoint results in improved security / privacy by verifying whether the user is still viewing details related to secure behavior.

[0187] In some embodiments, the amount of visible virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on occlusion by physical (e.g., real) objects. In some embodiments, the amount of visible virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) is detected, and the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes physical objects represented in a three-dimensional environment. In some embodiments, the user (e.g., 704) views a physical object (e.g., a computer monitor) through an additional display, or the user (e.g., 704) views a representation of the physical object reproduced on an opaque display, where the physical object is closer to the user (e.g., 704) than representations of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752). In some embodiments, a reduction in visible quantity is detected in response to a detection of a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), and the detected change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes detecting physical objects appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) with respect to the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754). In some embodiments, the reduction in visible quantity is based on detected physical objects appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., virtual interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) appearing less than they did before the viewpoint change).In some embodiments, a physical object may move in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., a physical balloon floats in front of the user's (e.g., 704) central field of view). In some embodiments, the modified user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) may include a physical object close to the user (e.g., 704) that obstructs the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., the user (e.g., 704) turns towards a tree one foot in front of the user (e.g., 704)). Where it is stated that a physical object is obscuring a virtual object (e.g., 702, 712, 736, 738, 748, and / or 752), this means, optionally, that a device (e.g., 700) effectively and virtually obscures at least a portion of a virtual object (e.g., 702, 712, 736, 738, 748, and / or 752) that is at a simulated distance further from the user's (e.g., 718, 746, and / or 754) viewpoint (e.g., 718, 746, and / or 754) than the physical object, while being within the same line of sight as the physical object. Enabling user permission for secure behavior based on the fact that the details of secure behavior are obscured by a physical object results in improved security / privacy by verifying whether the user is still viewing the details of the secure behavior.

[0188] In some embodiments, the amount of visible virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on occlusion by a virtual object (e.g., 732). In some embodiments, a reduction in the amount of visible objects is detected in response to a detection of a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) (e.g., part of the interface becomes invisible, or the entire interface becomes invisible). In some embodiments, the detected change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) includes detecting a virtual object (e.g., 732) that is appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) with respect to the user's (e.g., 704) viewpoint (e.g., as shown in Figures 7J and 7K). In some embodiments, the reduction of visibility is based on detected virtual objects (e.g., 732) appearing in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752). Enabling user authorization of secure actions based on the fact that the details of the secure action are obscured by virtual objects results in improved security / privacy by verifying whether the user is still viewing the details of the secure action.

[0189] In some embodiments, the visibility of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) moving out of the user's (e.g., 704) field of view (e.g., as shown in Figure 7I). In some embodiments, in response to detecting a change in the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754), the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) becomes undetectable within the user's (e.g., 704) current viewpoint (e.g., 718, 746, and / or 754) (e.g., the user (e.g., 704) looks away from the interface (e.g., 702, 712, 736, 738, 748, and / or 752), causing the interface to be positioned "behind" the user in the environment). Enabling user permission for secure behavior based on the fact that the details of secure behavior are outside the user's field of view results in improved security / privacy by verifying whether the user is still viewing the details of the secure behavior.

[0190] In some embodiments, the visibility of a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on the virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) moving beyond a threshold distance from the center of the user's (e.g., 704) field of view (e.g., 718, 746, and / or 754) (e.g., the user (e.g., 704) looks away from the interface (e.g., 702, 712, 736, 738, 748, and / or 752) so that only a portion of the interface is visible (e.g., visible toward the side of the user's (e.g., 704) field of view), or the interface (e.g., 702, 712, 736, 738, 748, and / or 752) is moved by the user (e.g., 704) or another user). Enabling user permission for secure actions based on the fact that the details of secure actions are too far from the user's center of view improves security and privacy by verifying whether the user is still viewing the details of the secure actions.

[0191] In some embodiments, the visibility of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) changes based on occlusion by physical objects. In some embodiments, the movement of a physical object is detected (e.g., a balloon floats in front of the user (e.g., 704)), and the detected movement includes the movement of the physical object in front of the virtual user interface (e.g., 702, 712, 736, 738, 748, and / or 752) with respect to the user's viewpoint (e.g., 718, 746, and / or 754) (e.g., on an additional display (e.g., a display including a transparent portion through which the user (e.g., 704) sees the actual surrounding physical environment (e.g., 730)), the balloon The movement of the balloon causes parts of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) to become invisible, or, on an opaque display (e.g., a sensor and / or camera-coupled display where a user (e.g., 704) sees a display representation of the surrounding physical environment (e.g., 730)), the movement of the balloon causes the balloon's representation to obscure parts of virtual interface objects (e.g., 702, 712, 736, 738, 748, and / or 752). Enabling user permission for secure actions based on the fact that the details of secure actions are obscured by physical objects improves security / privacy by verifying whether the user is still viewing the details of the secure actions.

[0192] In some embodiments, the determination that a user (e.g., 704) is permitted to perform a secure action includes determining whether a gaze criterion is met when the user (e.g., 704) is gazing at a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752). In some embodiments, the secure action is initiated upon determination that the user (e.g., 704) is gazing at a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) while a request for the execution of the secure action is received (e.g., the user (e.g., 704) is gazing at selected login information (e.g., 716a, 716b, 744a, and / or 744b), or the user (e.g., 704) is gazing at payment information). Requiring the user to view details about the secure action when requesting permission results in improved security / privacy by ensuring that the user has the opportunity to read the details about the action.

[0193] In some embodiments, determining whether a user (e.g., 704) is authorized to perform a secure action includes determining whether the eye criterion is met when the user's (e.g., 704) eyes are open. In some embodiments, the secure action is initiated according to the determination that both of the user's (e.g., 704) eyes are open, or at least one of the user's (e.g., 704) eyes are open, while a request to perform the secure action is being received. Requiring the user's eyes to be open in order to provide authorization for a secure action results in improved security / privacy by ensuring that the user has the opportunity to read the details of the action.

[0194] In some embodiments, a user (e.g., 704) can request the execution of a secure action (e.g., an indicator) based on the determination that at least a threshold amount of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) are visible from the user's (e.g., 704) viewpoint (e.g., the line of sight (e.g., 718, 746, and / or 754) is directed towards the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the target of the user's line of sight (e.g., 718, 746, and / or 754) is the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). To indicate to the user (e.g., 704) that (via 720, 734, and / or 756) the appearance of the virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) is modified (e.g., as shown in Figure 7D) (e.g., the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is not grayed out, buttons are marked as active, or text (e.g., "Proceed with request") indicating that the user (e.g., 704) can request the execution of a secure action is provided to the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)).In some embodiments, the amount of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) visible from the user's (e.g., 704) viewpoint (e.g., 718, 746, and / or 754) is less than a threshold amount (e.g., the line of sight (e.g., 718, 746, and / or 754) is not directed towards the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or the object of the user's line of sight (e.g., 718, 746, and / or 754) is not the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)), and the user (e.g., 704) is determined to be less than a threshold amount (e.g., the line of sight (e.g., 718, 746, and / or 754) is not directed towards the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)), and the user (e.g., 704) To indicate to the user (e.g., 704) that they cannot request the execution of a secure action, the appearance of virtual user interface objects (e.g., 702, 712, 736, 738, 748, and / or 752) is modified (e.g., as shown in Figure 7C) (e.g., the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is grayed out, buttons are marked as inactive, and / or text indicating that the user (e.g., 704) cannot request the execution of a secure action (e.g., "See to request") is provided to the user interface (e.g., 702, 712, 736, 738, 748, and / or 752)). Providing a visual indication that user authorization can be provided for a secure action provides improved visual feedback by informing the user that they can initiate a secure action.

[0195] In some embodiments, a three-dimensional environment (e.g., 730) includes a virtual object related to a secure action (e.g., a virtual representation of a shoe), and includes attaching a virtual user interface object to the virtual object related to the secure action (e.g., attaching a virtual representation of a sheet containing payment information about the shoe to the shoe), and when the virtual object moves, the attached virtual user interface object moves with the virtual object (e.g., when the shoe moves (e.g., the shoe is placed on a table), the sheet remains attached to the shoe and moves with the shoe (e.g., the sheet remains attached to the shoe while the shoe is on the table)). Attaching the corresponding object to an interface having details about the secure action provides improved visual feedback by informing the user that the secure action is associated with the corresponding object.

[0196] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchangeable, substituted, and / or added to among these methods. For example, the determination in method 800 of whether to enable user authorization for secure operation may optionally be used as part of method 900 to determine whether to enable user authorization. For brevity, those details will not be repeated here.

[0197] Additional information regarding Figures 7A to 7N is provided below with reference to the method 900 described with respect to Figures 7A to 7N.

[0198] Figure 9 is a flowchart illustrating Method 900, an exemplary method for authentication continuity for secure operation, according to several embodiments. In some embodiments, Method 900 is executed in a computer system (e.g., computer system 101 in Figure 1 (e.g., a smartphone, smartwatch, tablet, and / or wearable device)), which includes display generation components (e.g., display generation components 120 in Figures 1, 3, and 4 (e.g., display controllers, touch-sensitive display systems, displays (e.g., embedded and / or connected), 3D displays, transparent displays, projectors, and / or head-up displays)), one or more input devices, and biosensors (e.g., facial recognition devices, fingerprint recognition devices, and / or eye-tracking devices). In some embodiments, Method 900 is governed by a plurality of instructions, which are stored in a non-temporary (or temporary) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control 110 in Figure 1). Some operations of method 900 are combined as optional choices, and / or the order of some operations is changed as optional choices.

[0199] As will be described later, Method 900 provides an intuitive method for facilitating authentication continuity for secure operation. This method reduces the cognitive burden on the user while performing secure operation, thereby creating a more efficient human-machine interface. In the case of battery-powered computing devices, power is saved and battery charging intervals are extended by streamlining authentication continuity for secure operation, making it faster and more efficient.

[0200] At a first time, the user of the device (e.g., 704) biometrically authenticates using a biometric sensor to perform a first type of biometric authentication (902) (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed and / or a first level (e.g., full or high fidelity) fingerprint verification is performed). At a second time after the first time, a request is received to perform a secure action (e.g., a request is received to auto-fill in input fields (e.g., 706 and / or 708) (904) (e.g., auto-fill in a password and / or email information) and / or to detect the activation of a payment affordance).

[0201] In some embodiments, upon receiving a request to perform a secure action (906), the device performs a secure action (908) (for example, without performing a first type of biometric authentication after receiving a request to perform a secure action (for example, no first level (e.g., full or high fidelity) iris feature verification is performed between the request for the secure action and the performance of the secure action, and / or no first level (e.g., full or high fidelity) fingerprint verification is performed between the request for the secure action and the performance of the secure action), using autofill to enter predetermined values ​​into fields (e.g., 706 and / or 708) of the user interface (e.g., 702, 712, 736, 738, 748, and / or 752) and / or initiate a payment transaction). In some embodiments, the criterion is based on sensor measurements taken at multiple intermediate times between a first time and a second time, including sensor measurements taken at a first intermediate time and sensor measurements taken at a second intermediate time, which detect that the same user was using the device (e.g., 700) between the first time and the second time (e.g., the eye-tracking continuity policy has not had a “false” result since the user (e.g., 704) previously performed a first level (e.g., full or high fidelity) iris feature validation, and / or the fingerprint continuity policy has not had a “false” result since the user (e.g., 704) previously performed a first level (e.g., full or high fidelity) fingerprint validation).

[0202] In some embodiments, the execution of a secure action is aborted (910) upon determination that the user (e.g., 704) of the device (e.g., 700) did not meet the respective criteria between a first time and a second time (e.g., the evaluation of the eye-tracking continuity policy is "false" (e.g., eye gaze was undetectable for a certain period, the user (e.g., 704) was previously authenticated by passcode only, and / or the user (e.g., 704) was previously authenticated by fingerprint only), and / or the evaluation of the fingerprint tracking continuity policy is "false"). Ensuring that biometric continuity conditions are met when a user requests the execution of a secure action provides an improvement in security / privacy by verifying that the same user has been using the device since initial authentication.

[0203] In some embodiments, upon receiving a request to perform a secure action, the device user (e.g., 704) is biometrically authenticated using a biosensor to perform a first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed and / or a first level (e.g., full or high fidelity) fingerprint verification is performed) according to a determination that the device user (e.g., 704) did not meet the respective criteria between a first and second time (e.g., the eye tracking continuity policy evaluation is "false" (e.g., the eyes are undetectable for a certain period of time, the user (e.g., 704) was previously authenticated by passcode only, and / or the user (e.g., 704) was previously authenticated by fingerprint only, and / or the fingerprint tracking continuity policy evaluation is "false").

[0204] In some embodiments, the first type of biometric authentication is based on the biometric features of the eye (e.g., the entire eye or a part of the eye such as iris features), and the second type of biometric authentication is based on the biometric features of the eye (e.g., the entire eye or a part of the eye such as iris features). Ensuring that biometric continuity conditions are met based on eye features provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0205] In some embodiments, the first type of biometric authentication is based on facial biometric features (e.g., eyebrows, eyelids, skin around the eyes, forehead, nose), and the second type of biometric authentication is based on facial biometric features (e.g., eyebrows, eyelids, skin around the eyes, forehead, nose). Ensuring that biometric continuity conditions are met based on facial features provides improved security and privacy by verifying that the same user has been using the device since the initial authentication.

[0206] In some embodiments, a user of the device (e.g., 704) meeting each criterion involves repeatedly performing a second type of biometric authentication (e.g., iris feature verification of a second level (e.g., tracking eye information for continuity)) using one or more sensors of a computer system (e.g., 101 and / or 700), the second type of biometric authentication includes receiving an input from the user (e.g., 704) containing a first individual biometric (e.g., obtaining eye information from the current frame) and comparing the received input with a previously received input containing a second individual biometric (e.g., comparing eye information from the current frame with eye information from a previous frame). Obtaining multiple sensor measurements at multiple intermediate time points between a first time point and a second time point provides an improvement in security / privacy by verifying that the same user has been using the device since the initial authentication. In some embodiments, one or more sensors used to perform a second type of biometric authentication include biometric sensors used to perform a first type of biometric authentication (for example, those one or more sensors include biometric sensors used to obtain new biometric authentication). Obtaining multiple sensor measurements at multiple intermediate time points between a first time point and a second time point results in improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0207] In some embodiments, a first type of biometric authentication is based on a first biometric feature (e.g., the user's (e.g., 704) eye and / or iris features), and a second type of biometric authentication is based on a second type of biometric feature (e.g., continuous wear of a computer system and / or the user's (e.g., 704) fingerprint), where the first biometric feature is different from the second biometric feature (e.g., iris feature verification is performed for initial authentication, and facial features are monitored for continuity). In some embodiments, at a first time, the user of the device (e.g., 704) is biometrically authenticated based on the user's (e.g., 704) first biometric feature (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, which includes detecting details related to the presence of the eyes, iris features, pupil, and / or eye glint). In some embodiments, a second biometric feature of the user (e.g., 704) is periodically detected (e.g., the user's (e.g., 704) eyebrows, eyelids, skin around the eyes, forehead, nose, mouth, and / or cheeks), and each criterion is based on the periodic detection of the user's (e.g., 704) second biometric feature (e.g., the continuity criterion depends on the detected details related to the presence of eyebrows, eyelids, skin around the eyes, forehead, nose, mouth, and / or cheeks). Utilizing a separate biometric feature for initial authentication and a different biometric feature for biometric continuity verification results in improved security / privacy by enhancing authentication when the feature used for initial authentication is unavailable for continuity verification.

[0208] In some embodiments, a first type of biometric authentication is based on a first biometric feature (e.g., the user's (e.g., 704) eyes and / or iris features, such as iris feature verification being performed for initial authentication), and a second type of biometric authentication is based on a first type of biometric feature (e.g., the user's (e.g., 704) eyes and / or iris features, such as iris feature verification being performed for continuity). In some embodiments, at a first time, the user of the device (e.g., 704) is biometrically authenticated based on the user's (e.g., 704) first biometric feature (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, which includes detecting details related to the presence of the eyes, iris features, pupils, and / or eye glints). In some embodiments, a first biometric feature of the user (e.g., 704) is periodically detected (e.g., details related to the presence of eyes, iris features, pupils, and / or eye glints), and each criterion is based on the periodic detection of the first biometric feature of the user (e.g., 704) (e.g., the continuity criterion depends on the detected details related to the presence of eyes, iris features, pupils, and / or eye glints). Utilizing separate biometric features for both initial authentication and biometric continuity verification results in improved security / privacy by reducing the number of sensors required for authentication.

[0209] In some embodiments, biometric authentication of the device user (e.g., 704) at a first time includes obtaining sensor measurements at a plurality of intermediate time points between the first and second time points (e.g., detecting details related to the presence of eyes, iris features, pupils, and / or eye glints) based on the first biometric features of the user (e.g., 704) (e.g., performing a first level (e.g., full or high fidelity) iris feature verification, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints), the obtained sensor measurements include at least one sensor measurement indicating that the first biometric features of the user (e.g., 704) were not detected at any of the intermediate time points (e.g., the sensor measurement indicates the absence of eyes, iris features, pupils, and / or eye glints (e.g., based on the user's (e.g., 704) eyes being closed)). When biometric continuity verification is performed based on other features, utilizing individual biometric features for initial authentication that are hidden from view improves security and privacy by ensuring that continuity verification is performed even though the features are hidden.

[0210] In some embodiments, biometric authentication of the device user (e.g., 704) at a first time includes, based on the user's (e.g., 704) first biometric features (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed, including detecting details related to the presence of eyes, iris features, pupils, and / or eye glints), and periodically acquiring sensor measurements of the user's (e.g., 704) second biometric features at multiple intermediate time points between the first and second time points (e.g., detecting the user's (e.g., 704) eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and / or cheeks), where the acquired sensor measurements indicate that the user's (e.g., 704) first biometric features were not detected at any of the individual intermediate time points. The acquired sensor measurement includes at least one sensor measurement (e.g., the sensor measurement does not indicate the presence of eyes, iris features, pupils, and / or eye glints (e.g., based on the user's (e.g., 704) eyes being closed)), and the acquired sensor measurement includes at least one sensor measurement indicating that a second biometric feature of the user (e.g., 704) was detected at that individual intermediate time (e.g., the sensor measurement indicates the presence of the user's (e.g., 704) eyebrows, eyelids, skin surrounding the eyes, forehead, nose, mouth, and cheeks, while the presence of eyes, iris features, pupils, and / or eye glints is not detected), and the first biometric feature is different from the second biometric feature (e.g., iris feature verification is performed for initial authentication, but facial features are monitored for continuity). Leveraging a separate biometric feature for initial authentication that is hidden from view when biometric continuity verification is performed, based on another visible feature, results in improved security / privacy by creating an efficient and uninterrupted user experience for biometric authentication, encouraging users to use biometric authentication, and thus providing a more secure experience.

[0211] In some embodiments, each criterion between a first time and a second time (e.g., a continuity criterion) includes a continuity criterion that is met when a biometric feature (e.g., the user's (e.g., 704) eyes and / or the user's (e.g., 704) fingerprint) is repeatedly detected (e.g., continuously or periodically) between the first time and the second time (e.g., without determining that the biometric feature was not detected between the first time and the second time). Obtaining multiple sensor measurements at multiple intermediate time points between the first and second time points provides an improvement in security / privacy by verifying that the same user has been using the device since initial authentication.

[0212] In some embodiments, biometric authentication of a device user (e.g., 704) using a biosensor to perform a first type of biometric authentication includes providing the user (e.g., 704) with an authentication guidance user interface (e.g., 724) containing instructions prompting the user to perform a first type of biometric authentication (e.g., displayed via a display generating component (e.g., 120 and / or 700a)), the instructions prompting the user (e.g., 704) to provide specific movements of biometric features (e.g., instructions prompting the user to gaze at a specific part of the authentication guidance user interface (e.g., 726), rotate their face along a specific path (for facial recognition), and / or move their fingers along a specific path). Displaying the authentication guidance user interface to facilitate user authentication results in improved security / privacy by reducing the chance of authentication attempt failure.

[0213] In some embodiments, secure operation includes providing payment information (for example, as shown in Figure 7N) (e.g., providing payment information to purchase items and / or services (e.g., providing credit card information to an e-commerce website to purchase shoes)). In some embodiments, an electronic device (e.g., 700) participates in a transaction. For example, upon detecting that input has been received in a hardware user interface element (e.g., 722 and / or 758) and that other criteria have been met, a secure element within the computer system (e.g., 700) releases the payment information so that an application on the computer system (e.g., 700) can access the information (e.g., send the information to a server associated with the application). In some embodiments, the secure element is a hardware component (e.g., a secure microcontroller chip) configured to securely store data or algorithms. In some embodiments, the secure element provides (or exposes) payment information (e.g., an account number and / or a transaction-specific dynamic security code). In some embodiments, the secure element provides (or discloses) payment information in response to the device (e.g., 700) receiving authorization such as user authentication (e.g., fingerprint authentication, passcode authentication, detection of a double press of hardware buttons (e.g., 722 and / or 758) by providing authentication credentials to the device (e.g., 700) when the device (e.g., 700) is in an unlocked state, and optionally from the time the device (e.g., 700) is unlocked, while the device (e.g., 700) is continuously on the user's (e.g., 704) wrist. The continuous presence of the device (e.g., 700) on the user's (e.g., 704) wrist is determined by periodically checking whether the device (e.g., 700) is in contact with the user's (e.g., 704) skin).In some embodiments, the secure element provides (or releases) payment information in response to the fulfillment of continuity criteria (e.g., a second level (e.g., continuity) iris feature verification being performed) and / or gaze criteria (e.g., gaze is directed towards the user interface and / or the object of the user's gaze is the user interface). Enabling user authorization for payment actions using biometric continuity verification provides enhanced security / privacy by ensuring that the same user has been using the device since initial authentication.

[0214] In some embodiments, secure operation includes providing access credentials to access an application (e.g., an application related to a third-party service, e.g., providing facial recognition data to access a stock trading application or a password memory application). Enabling user authorization for application authentication using biometric continuity verification results in improved security / privacy by ensuring that the same user has been using the device since initial authentication.

[0215] In some embodiments, secure operation includes automatically filling in user credentials (e.g., providing username and / or password information for accessing secure information (e.g., login information for accessing a banking website (e.g., 706 and / or 708))). Enabling user authorization for autofill operation using biometric continuity verification provides enhanced security / privacy by ensuring that the same user has been using the device since initial authentication.

[0216] In some embodiments, receiving a request to perform a secure action includes detecting a physical input to activate a hardware button (e.g., 722 and / or 758) (e.g., a user (e.g., 704) engaging in a double press of a hardware button to initiate a payment action and / or a user (e.g., 704) engaging in a double-click of a hardware button to initiate an autofill action). In some embodiments, when activated using various inputs, the hardware buttons (e.g., 722 and / or 758) perform various respective functions, such as navigating to the home user interface, powering off the device, displaying the system user interface for powering off the device, and / or activating a voice assistant. In some embodiments, the physical input includes a single press, a long press, a long press after a single press, a rotation after a press, a press after a rotation, and / or multiple consecutive presses (e.g., a double press, a triple press). Utilizing hardware elements for authorization results in improved security / privacy by reducing the possibility of false authorizations and / or preventing unauthorized input from malicious software.

[0217] In some embodiments, a system user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is provided to a user (e.g., 704) (e.g., displayed via a display generation component (e.g., 120 and / or 700a)) which includes affordances related to secure operations (e.g., affordances labeled “Pay” or “Auto Fill”), in which case receiving a request to perform a secure operation includes detecting the activation of affordances related to the secure operation (e.g., 710 and / or 750) (e.g., the user (e.g., 704) activates the affordances by gaze, gaze and hardware buttons, or hardware buttons only). In some embodiments, the user gazes at the affordances (e.g., 710 and / or 750), and the affordances (e.g., 710 and / or 750) are activated according to the determination that the user has gazed at the affordances over a threshold period. In some embodiments, a user (e.g., 704) activates an affordance (e.g., 710 and / or 750) by activating a hardware button (e.g., 722 and / or 758) (e.g., double-pressing and / or long-pressing the hardware button). In some embodiments, a user (e.g., 704) activates an affordance by activating a hardware button (e.g., 722 and / or 758) (e.g., double-pressing and / or long-pressing the hardware button), and the affordance (e.g., 710 and / or 750) is activated according to the determination that the user (e.g., 704) is gazing at the affordance (e.g., 710 and / or 750) while the user (e.g., 704) is activating the hardware button.In some embodiments, a user (e.g., 704) activates affordances (e.g., 710 and / or 750) by controlling a secondary device (e.g., a controller) to select the affordances (e.g., by moving the cursor over the affordance while activating a hardware button, and / or by using a pointing device while activating a hardware button (e.g., 722 and / or 758)). Utilizing affordances within the system user interface for authorization provides improved security / privacy when the device is used with one or more alternative input devices.

[0218] In some embodiments, receiving a request to perform a secure operation includes detecting the activation of an affordance (e.g., 710 and / or 750) within a virtual user interface object (e.g., 702, 712, 736, 738, 748, and / or 752) containing information about a secure operation (e.g., 706, 708, 710, 716a, 716b, 744a, and / or 744b) by the user (e.g., displayed via a display generation component (e.g., 120 and / or 700a)) (e.g., the user (e.g., 704) activates an affordance labeled "Allow Payment" or the user (e.g., 704) activates an affordance labeled "Allow Auto-Fill"). Displaying information about secure actions when requesting permission improves security and privacy by ensuring that users have the opportunity to review information related to secure actions before authorizing them to proceed.

[0219] In some embodiments, each criterion includes an eye-opening criterion, which is met when one or more of the user's (e.g., 704) eyes (e.g., one eye and / or both eyes) are not continuously closed beyond a threshold period (e.g., non-zero) (e.g., the continuity criterion is not met when the user's (e.g., 704) eyes are closed beyond the threshold period), and the eye-opening criterion is not met when one or more of the user's (e.g., 704) eyes (e.g., one eye and / or both eyes) are continuously closed beyond a threshold period (e.g., non-zero) (e.g., the continuity criterion is not met when the user's (e.g., 704) eyes are closed beyond the threshold period). Determining that bio-continuity is not met when eyes are detected as closed results in improved security / privacy by ensuring informed consent regarding the operation.

[0220] In some embodiments, each criterion is based on biometric features available for analysis regardless of whether the user's (e.g., 704) eyes are closed or not (e.g., regardless of the duration for which the user's (e.g., 704) eyes are closed) (e.g., the continuity criterion may be met even when the user (e.g., 704) has their eyes closed for an extended period). Determining that biometric continuity is met when eyes are detected as closed provides an improvement in security / privacy by improving authentication when the features used for initial authentication are unavailable for continuity verification.

[0221] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchangeable, substituted, and / or added to among these methods. For example, the determination in method 900 of whether the user has met individual (e.g., continuity) criteria may optionally be used as part of method 800 to determine whether to enable user authorization. For brevity, those details will not be repeated here.

[0222] Additional information regarding Figures 7A to 7N is provided below with reference to Method 1000 described with respect to Figures 7A to 7N.

[0223] Figure 10 is a flowchart illustrating Method 1000, an exemplary method for authentication continuity for secure operation, according to several embodiments. In some embodiments, Method 1000 is executed in a computer system (e.g., computer system 101 in Figure 1 (e.g., a smartphone, smartwatch, tablet, and / or wearable device)) which includes display generating components (e.g., display generating components 120 in Figures 1, 3, and 4 (e.g., display controllers, touch-sensitive display systems, displays (e.g., embedded and / or connected), 3D displays, transparent displays, projectors, and / or head-up displays)) (e.g., head-up displays, displays, touchscreens, projectors, etc.) and / or one or more input devices. In some embodiments, Method 1000 is governed by a plurality of instructions, which are stored in a non-temporary (or temporary) computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control 110 in Figure 1). Some operations of method 1000 are combined as optional choices, and / or the order of some operations is changed as optional choices.

[0224] As described later, Method 1000 provides an intuitive method for allowing secure operation through an accessibility interface. This method reduces the cognitive load on the user while performing secure operation through the accessibility interface, thereby creating a more efficient human-machine interface. In the case of battery-powered computing devices, faster and more efficient permission of secure operation through the accessibility interface saves power and extends the interval between battery charges.

[0225] The system receives requests via one or more input devices (e.g., 125 and / or 150) to display user interfaces related to the execution of secure operations (e.g., 702, 712, 736, 738, 748, and / or 752) (e.g., requests to execute autofill procedures and / or requests to execute payment transactions) (1002). In response to a request to display a user interface related to the execution of a secure operation (e.g., 702, 712, 736, 738, 748, and / or 752), a first user interface (e.g., 702, 712, 736, 738, 748, and / or 752) is displayed via a display generating component (e.g., 120 and / or 700a) (1004) (e.g., an interface displayed in response to a request to execute an autofill procedure and / or an interface displayed in response to a request to execute a payment transaction).

[0226] In accordance with the determination that the computer system (e.g., 700) is operating in a first mode (e.g., a standard mode in which accessibility features are not enabled, and / or a mode that allows secure operation using a physical input mechanism (e.g., 722 and / or 758)), individual prompts are first prompts (e.g., 722, 734 and / or 758) for providing physical input (e.g., 720 and / or 756) (1006) (e.g., the first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) for the user (e.g., 704) to allow the user to provide physical input (e.g., 722 and / or 758) (e.g., physical input mechanism (e.g., The prompt indicates that the user should provide input using a pushable button (e.g., 722 and / or 758), press, and / or a series of presses of the button (e.g., 722 and / or 758) (for example, the prompt (e.g., 720, 734 and / or 756) indicates that if the user (e.g., 704) intends to authorize an autofill procedure, the user (e.g., 704) should provide physical input (e.g., 722 and / or 758), and / or the prompt (e.g., 720, 734 and / or 756) indicates that if the user (e.g., 704) intends to authorize a purchase, the user (e.g., 704) should provide physical input (e.g., 722 and / or 758)).

[0227] In accordance with the determination that the computer system (e.g., 700) is operating in a second mode different from the first mode (e.g., an accessibility mode and / or a mode that does not require input from physical input mechanisms (e.g., 722 and / or 758) to allow secure operation), the individual prompts (e.g., 720, 734 and / or 756) are second prompts to display the second user interface (e.g., 742) (for example, the first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) does not prompt the user (e.g., 704) that the user should provide physical input (e.g., 722 and / or 758) if the user (e.g., 704) intends to allow the autofill procedure, and / or the first user interface (e.g., 702, 712, 736, 738, 748 and / or 752) does not prompt the user (For example, 704) does not prompt the user (for example, 704) that the user should provide physical input (for example, 722 and / or 758) if the user (for example, 704) intends to authorize a purchase), in this case the second user interface (for example, 742) enables user authorization for secure operation without using physical input (for example, 722 and / or 758) (1008) (for example, enabling authorization for secure operation using first level (for example, full or high fidelity) iris feature validation (for example, first level iris feature validation is performed while the user (for example, 704) is activating the affordance (for example, 744), and / or first level iris feature validation is performed while the user (for example, 704) is gazing at the affordance (for example, 744), and / or enables authorization for secure operation using first level (for example, full or high fidelity) fingerprint validation). Allowing secure operation without physical input improves security and privacy when the device is used with one or more alternative input devices.

[0228] In some embodiments, a first separate affordance (e.g., 740 and / or 750) is provided within a second prompt (e.g., 736 and / or 738) (e.g., the affordance includes an indication (e.g., 734) that the authorization of a secure action is performed using a secondary input type different from the primary input type (e.g., "Autofill" is displayed next to "Confirm with AssistiveTouch" or an affordance for "Pay" is provided next to "Confirm with AssistiveTouch" (e.g., displayed via a display generation component (e.g., 120 and / or 700a))), then a user selection of the affordance (e.g., 740 and / or 750) is received (e.g., user (For example, 704) selects an "autofill" affordance (e.g., 740) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input, or a user (e.g., 704) selects a "pay" affordance (e.g., 750) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input). In some embodiments, the user (e.g., 704) gazes at the affordances (e.g., 740 and / or 750), and the affordances (e.g., 740 and / or 750) are activated according to the determination that the user (e.g., 704) gazed at the affordances (e.g., 740 and / or 750) over a threshold period.In some embodiments, the user (e.g., 704) activates an affordance (e.g., 740 and / or 750) by controlling an adaptive accessory (e.g., a switch control) to select the affordance (e.g., 740 and / or 750) (for example, the user (e.g., 704) taps the switch control once to start displaying a sliding vertical line that slides back and forth across the display (e.g., 700a); the user (e.g., 704) taps the switch control a second time when the vertical line appears over the affordance (e.g., 740 and / or 750); in response to the second tap, a horizontal line that slides up and down across the display (e.g., 700a) is displayed; and the user (e.g., 704) taps the switch control a third time when the horizontal line appears over the affordance (e.g., 740 and / or 750)). In some embodiments, a user (e.g., 704) controls an adaptive accessory to select an affordance (e.g., 740 and / or 750), and the affordance (e.g., 740 and / or 750) is activated according to the determination that the user (e.g., 704) is gazing at the affordance (e.g., 740 and / or 750) while the user (e.g., 704) is selecting the affordance (e.g., 740 and / or 750). An adaptive accessory is an accessory that allows a user (e.g., 704) to provide input via alternative means compared to a typical usage scenario of the device. An example of an adaptive accessory is a switch control that provides a limited number of selection states (e.g., between 1 and 5) used to make a selection among options provided by the device (e.g., 700).An example of switch control is a device that can be activated by pressing, pulling, blinking, gripping, and / or sucking / blowing through a straw, allowing users with cognitive or physical accessibility needs to interact with the device (e.g., 700) when a user (e.g., 704) cannot easily interact with the device (e.g., 700) using standard input methods (e.g., touchscreen, physical buttons / knobs, and / or air gestures).

[0229] In some embodiments, upon receiving a user selection of an affordance, a second user interface (e.g., 742) is displayed, which includes a representation of multiple functions (e.g., an accessibility menu (e.g., 742) containing multiple affordances representing device functions), which includes a first distinct function (e.g., the accessibility menu (e.g., 742) includes options such as "Home," "Control Center," "Device," "Notifications," or "Custom") from among the multiple functions, which is called depending on a tertiary input type different from the primary input type (e.g., 744) (e.g., the accessibility menu (e.g., 742) includes a new option "Allow Autofill" or a new option "Payment"). In some embodiments, an accessibility menu (e.g., 742) includes a grid of multiple affordances and a contour line enclosing one of those affordances (e.g., 744), where the contour line periodically moves (e.g., every 2 or 5 seconds) to enclose another affordance (e.g., 744), and the user (e.g., 704) can provide input (e.g., input to a switch control and / or gaze input directed to the enclosed affordance) to select the currently contoured affordance (e.g., 744). Displaying an accessibility interface with alternative input options provides improved security / privacy when the device is used with one or more alternative input devices.

[0230] In some embodiments, a secondary input type simulates a primary input type input (e.g., calling "Home" in response to a swipe up from a screen button), and a tertiary input type simulates a primary input type input (e.g., calling "Home" in response to a custom input (e.g., long press) via an accessibility menu (e.g., 742)). Enabling simulated inputs instead of conventional input types provides improved security / privacy when the device is used with one or more alternative input devices.

[0231] In some embodiments, while the computer system (e.g., 700) is operating in a second mode, affordances associated with individual prompts (e.g., 740 and / or 750) are displayed (e.g., the affordance for "autofill" is displayed with the text "Confirm with AssistiveTouch" or the affordance for "payment" is displayed with the text "Confirm with AssistiveTouch"), and in accordance with the receipt of user input associated with the individual prompts (e.g., the user (e.g., 704) selects the affordance for "autofill" (e.g., 740) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input, or the user (e.g., 704) selects the affordance for "payment" (e.g., 750) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input), a second user interface (e.g., 742) is displayed, and the second user - The interface (e.g., 742) includes affordances (e.g., 744) to allow secure actions (e.g., displaying an accessibility menu (e.g., 742) with a new option "Allow Autofill" or a new option "Pay"), and the affordances (e.g., 744) to allow secure actions are invoked depending on a secondary input type different from the primary input type (e.g., the option "Allow Autofill" is invoked depending on a custom input (e.g., long press) via the accessibility menu (e.g., 742) instead of a primary input type (e.g., double-pressing a side button (e.g., 722 and / or 758)), or the option "Pay" is invoked depending on a custom input (e.g., long press) via the accessibility menu (e.g., 742) instead of a primary input type (e.g., double-pressing a side button (e.g., 722 and / or 758)). Displaying an accessibility interface that includes affordances to allow secure actions provides security / privacy improvements when the device is used with one or more alternative input devices.

[0232] In some embodiments, details related to the execution of secure operations (e.g., the interface includes details for executing an autofill procedure (e.g., username and password fields), or the interface includes details for executing a payment transaction (e.g., product name, product description, and product price)) are displayed within the user interface related to the execution of secure operations (e.g., 702, 712, 736, 738, 748, and / or 752). In some embodiments, while the computer system (e.g., 700) is operating in a second mode (e.g., accessibility mode, and / or a mode that does not require input from physical input mechanisms (e.g., 722 and / or 758) to allow secure operations), affordances related to individual prompts (e.g., 740 and / or 750) are displayed (e.g., the affordance for "autofill" is displayed with the text "Please confirm with AssistiveTouch", or the affordance for "payment" is displayed with the text "Please confirm with AssistiveTouch").In some embodiments, user input related to individual prompts is received (for example, the user (e.g., 704) selects the "autofill" affordance (e.g., 740) using gaze, gaze + adaptive accessory input, and / or adaptive accessory input, or the user (e.g., 704) selects the "pay" affordance (e.g., 750) using gaze, gaze and adaptive accessory input, and / or adaptive accessory input), and in response to a determination that the user (e.g., 704) is gazing at a user interface related to performing a secure action while user input related to individual prompts is received (for example, the user (e.g., 704) selects the "autofill" affordance (e.g., 740) while gazing at an interface containing details for performing an autofill procedure, or the user (e.g., 704) selects the "pay" affordance (e.g., 750) while gazing at an interface containing details for performing a payment transaction), A second user interface (e.g., 742) is displayed, and the second user interface (e.g., 742) includes affordances (e.g., 744) to allow secure actions (e.g., displaying an accessibility menu (e.g., 742) that includes a new option "Allow Autofill" or a new option "Pay"). The affordances (e.g., 744) are displayed depending on a secondary input type different from the primary input type (e.g., the option "Allow Autofill" is invoked via a custom input (e.g., long press) through the accessibility menu (e.g., 742) instead of a primary input type (e.g., double press of a side button (e.g., 722 and / or 758)), or the option "Pay" is invoked via a custom input (e.g., long press) through the accessibility menu (e.g., 742) instead of a primary input type (e.g., double press of a side button (e.g., 722 and / or 758)).Displaying accessibility interfaces based on a combination of gaze and affordance activation provides improved security and privacy when a device is used with one or more alternative input devices.

[0233] In some embodiments, while a computer system (e.g., 700) is operating in a first mode (e.g., a standard mode in which accessibility features are not enabled, and / or a mode that allows secure operation using a physical input mechanism), user activation of physical input (e.g., 722 and / or 758) is received while a first prompt (e.g., 720, 734, and / or 756) is displayed (e.g., a user (e.g., 704) performs a predetermined activation pattern of hardware input affordances, such as double-clicking a hardware button (e.g., 722 and / or 758)). In some embodiments, the predetermined activation pattern includes a single press, a long press, a single press followed by a long press, a press followed by a rotation, a rotation followed by a press, and / or a series of consecutive presses (e.g., a double press or a triple press). In some embodiments, a secure operation is permitted (for example, allowing an autofill procedure in response to the activation of a hardware button (e.g., 722 and / or 758) or a payment procedure in response to the activation of a hardware button (e.g., 722 and / or 758)) in response to receiving user activation of a physical input (e.g., 722 and / or 758) while a first prompt (e.g., 720, 734, and / or 756) is displayed, upon determination that the user authentication criteria are met (e.g., the user (e.g., 704) is staring at the prompt (e.g., 720, 734, and / or 756)). In some embodiments, if it is determined that the user authentication criteria are not met (for example, the user (e.g., 704) is not paying attention to the prompt (e.g., 720, 734, and / or 756)), the permission for secure operation is revoked (for example, interfering with an autofill procedure in response to a double-click of a hardware button (e.g., 722 and / or 758), or interfering with a payment procedure in response to a double-click of a hardware button (e.g., 722 and / or 758)).Utilizing accessible and inaccessible operating modes provides enhanced security and privacy when the device is used with one or more alternative input devices.

[0234] In some embodiments, the authentication criteria include criteria that are met when the user (e.g., 704) provides specific biometric movements (e.g., gazing at a specific part of an authentication guidance user interface (e.g., 724), providing facial rotation along a specific path, and / or moving a finger along a specific path). Requiring specific biometric movements while the physical input mechanism is activated results in improved security / privacy by ensuring informed consent for secure operation.

[0235] In some embodiments, input is received from the user (e.g., 704) to allow secure operation (e.g., during standard mode, the user (e.g., 704) performs the activation of a hardware button (e.g., 722 and / or 758)) (e.g., single press, long press, long press after single press, rotation after press, press after rotation, multiple consecutive presses (e.g., double press, triple press), or during accessibility mode, the option "Allow Autofill" is invoked in response to custom input via the accessibility menu (e.g., 742) or the accessibility menu The option “Payment” is invoked in response to custom input via Nu (e.g., 742), and user authentication is performed in response to receiving input to authorize a secure operation (e.g., a first level (e.g., full or high fidelity) iris verification is performed, a second level (e.g., continuity) iris verification is performed, or passcode verification is performed), and the secure operation is authorized according to the determination that the user (e.g., 704) is authenticated (e.g., it determines whether authentication continuity has been true since the last full or high fidelity biometric authentication, performs full or high fidelity biometric authentication, and / or performs passcode authentication). In some embodiments, the secure operation is not authorized according to the determination that the user (e.g., 704) is not authenticated (e.g., the device revokes authorization for the secure operation). Requiring user authentication in addition to user authorization for a secure operation results in improved security / privacy by adding an additional verification layer specific to the user requesting authorization.

[0236] In some embodiments, performing user authentication includes performing a second type of biometric authentication (e.g., performing iris feature verification at a second level (e.g., tracking eye information for continuity)) (e.g., performing authentication every two frames (e.g., each frame includes a still image or set of images acquired from the device's sensors (e.g., biosensors)), every three frames, every four frames, every five frames, every six frames, every seven frames, every eight frames, every nine frames, or every ten frames), the second type of biometric authentication includes receiving an input from the user (e.g., 704) containing a first individual biometric (e.g., acquiring eye information from the current frame), and comparing the received input with a previously received input containing a second individual biometric (e.g., comparing eye information from the current frame with eye information from the previous frame). Utilizing stored authentication information in addition to user authorization for secure operation provides improved security / privacy by verifying that the same user has been using the device since the initial authentication.

[0237] In some embodiments, performing user authentication includes performing a first type of biometric authentication, the first type of biometric authentication includes detecting the presence of a first type of biometric feature (e.g., a first level (e.g., full or high fidelity) verification based on eye features, or a first level (e.g., full or high fidelity) verification based on facial features). Requiring full or high fidelity biometric authentication in addition to user authorization for secure operation provides an improvement in security / privacy by verifying that an authorized user is requesting the secure operation.

[0238] In some embodiments, performing user authentication includes providing a prompt (e.g., 728) requesting a passcode from the user (e.g., 704); receiving a passcode from the user (e.g., 704) in response to the prompt (e.g., 728) (e.g., receiving multiple digit inputs (e.g., 0-9)); determining that the user (e.g., 704) is authenticated based on the determination that the received passcode matches a stored passcode; and allowing secure action based on the determination that the user (e.g., 704) is authenticated (e.g., the user interface is not grayed out, a button is marked as active, or the user interface provides text (e.g., "Proceed with Request") indicating that the user (e.g., 704) can request to perform secure action). In some embodiments, determining that the user (e.g., 704) is not authenticated based on the determination that the received passcode does not match a stored passcode, and secure action is not allowed. Requiring passcode authentication in addition to user permission for secure action results in improved security / privacy by providing an additional method for verifying the user's identity.

[0239] In some embodiments, an input for allowing a secure operation is detected, and in response to the detection of an input for allowing a secure operation, a determination is made as to whether the input is a physical input for allowing a secure operation (e.g., 722 and / or 758) (e.g., an activation procedure for a physical button (e.g., 722 and / or 758) for allowing autofill, or an activation procedure for a physical button (e.g., 722 and / or 758) for allowing payment). In some embodiments, the activation procedure includes a single press, a long press, a long press after a single press, a rotation after a press, a press after a rotation, and multiple consecutive presses (e.g., a double press, a triple press). In some embodiments, authentication information is retrieved, which is related to a second type of biometric authentication that is repeatedly performed (e.g., retrieving continuity information relating to a previously performed second level (e.g., tracking eye information for continuity)). In some embodiments, as to whether the input is related to a second user interface (e.g., 742) as to whether the input is related to a second user interface (e.g., 742) as to whether the input is a physical input for allowing a secure operation. In some embodiments, a secure operation is permitted based on the determination that the user (e.g., 704) is authenticated based on a second type of biometric authentication (e.g., allowing autofill if a second level of iris feature verification (e.g., tracking eye information for continuity) is successful, or allowing payment if a second level of iris feature verification (e.g., tracking eye information for continuity) is successful). In some embodiments, a secure operation is not permitted based on the determination that the user (e.g., 704) is not authenticated based on a second type of biometric authentication (e.g., tracked eye information indicates that the user's (e.g., 704) eyes were closed for a certain period of time (e.g., 5 seconds, 7 seconds, or 10 seconds) during interaction with the device).

[0240] In some embodiments, if an input associated with a second user interface (e.g., 742) enables user authorization for secure actions that do not use physical input (e.g., if the detected input is associated with the option "autofill" or the new option "pay"), a first type of biometric authentication is performed according to the determination that the input is associated with a second user interface (e.g., 742) (e.g., the input is detected via the accessibility menu), and a secure action is permitted according to the determination that the user (e.g., 704) has been authenticated based on the first type of biometric authentication (e.g., a first level (e.g., full or high fidelity) iris feature verification is performed), and a secure action is permitted according to the determination that the user (e.g., 704) has been authenticated based on the first type of biometric authentication (e.g., autofill is permitted if the first level (e.g., full or high fidelity) iris feature verification is successful, or payment is permitted if the first level (e.g., full or high fidelity) iris feature verification is successful). In some embodiments, a secure action is not permitted according to the determination that the user (e.g., 704) has not been authenticated based on the first type of biometric authentication. In some embodiments, the determination of whether an input is an additional input unrelated to the physical inputs (e.g., 722 and / or 758) for allowing secure operation is made based on the determination that the input is not an input related to a second user interface (e.g., 742). Requiring full or high-fidelity biometric authentication when the device operates in accessible mode provides security / privacy improvements by verifying that an authorized user is requesting secure operation.

[0241] In some embodiments, aspects / operations of methods 800, 900, and 1000 may be interchangeable, substituted, and / or added to among these methods. For example, the determination in method 900 of whether the user has met individual (e.g., continuity) criteria may optionally be used as part of method 1000 to allow secure operation through the accessibility interface. For brevity, those details will not be repeated here.

[0242] The above is described in relation to specific embodiments for illustrative purposes. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the invention and its practical applications, thereby enabling other persons skilled in the art to best use the invention and the various described embodiments with various modifications suitable for specific applications that may be conceived.

[0243] As described above, one aspect of this technology involves collecting and using data available from various sources to improve the authorization of secure operation. This disclosure considers that in some cases, such collected data may include personal information data that uniquely identifies a particular person, or personal information data that can be used to contact a particular person or locate them. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, addresses, data or records relating to a user's (e.g., 704) health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0244] This disclosure acknowledges that such use of personal data in the technology may be for the benefit of the user. For example, personal data may be used to enable secure operation. Furthermore, other uses of personal data that benefit the user are also intended by this disclosure. For example, health and fitness data may be used to provide insights into the user's overall wellness, or as positive feedback to individuals using the technology to pursue wellness goals.

[0245] This disclosure assumes that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal data will adhere to a robust privacy policy and / or privacy practice. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for the strict confidentiality of personal data. Such policies should be readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected for the lawful and legitimate use of the entity and should not be shared or sold for any other purpose. Furthermore, such collection / sharing should be carried out only after informing and obtaining the user's consent. In addition, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others with access to personal data faithfully adhere to those privacy policies and procedures. Furthermore, such entities may undergo third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal data collected and / or accessed, and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be subject to federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). On the other hand, health data in other countries may be subject to other regulations and policies and should be addressed accordingly. Therefore, different privacy practices should be maintained in each country with respect to different types of personal data.

[0246] Notwithstanding the foregoing, the Disclosure also envisions embodiments that allow users to selectively prevent the use of or access to personal data. That is, the Disclosure intends that hardware and / or software elements may be provided to prevent or prevent access to such personal data. For example, in the case of allowing secure operations, the Technology may be configured so that users can choose to “opt in” or “opt out” of participating in the collection of personal data during or at any time thereafter when registering for a service. In another example, a user may choose not to provide data for customizing secure operations such as payments and autofill. In yet another embodiment, a user may choose to limit the length of time data is retained or to completely prohibit the deployment of customized services to facilitate secure operations such as payments or autofill. In addition to providing “opt-in” and “opt-out” options, the Disclosure intends to provide notices regarding access to or use of personal data. For example, a user may be notified when downloading an app that will access their personal data, and then again immediately before the app accesses their personal data.

[0247] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. In addition, where applicable in certain health-related applications, data anonymization can be used to protect user privacy. De-identification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of data stored (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across all users), and / or by other means.

[0248] Therefore, while this disclosure broadly covers the use of personal data to implement one or more different disclosed embodiments, it is conceivable that these different embodiments can also be implemented without requiring access to such personal data. That is, the different embodiments of the technology are not rendered inoperable by the absence of all or part of such personal data. For example, enabling secure operation can be facilitated based on non-personal data or a minimum amount of personal information, such as content requested by a user-related device, other non-personal information available to the service, or publicly available information.

Claims

1. In a computer system that communicates with one or more input devices and display generation components, A three-dimensional environment, which includes a virtual user interface object containing information about secure operation, is visible via the display generation component, while a change in the user's current viewpoint is detected via one or more input devices. In response to detecting the change in the user's viewpoint, In accordance with the determination that at least a threshold amount of the virtual user interface objects is visible from the user's perspective and that the user is permitted to perform the secure operation, user permission to perform the secure operation using the virtual user interface objects is enabled. In accordance with the determination that the number of virtual user interface objects visible from the user's viewpoint is less than the threshold amount, the user permission for the secure operation using the virtual user interface objects is deactivated. Methods that include...

2. Receiving input from the user to authorize the execution of the secure operation, and in response to receiving the input to authorize the execution of the secure operation, The execution of the secure operation is initiated in accordance with the determination that one or more execution criteria are met, including a visibility criterion that is satisfied when at least the threshold amount of the virtual user interface objects is visible from the user's perspective. In accordance with the determination that one or more of the aforementioned execution criteria are not met, the start of the secure operation is canceled. The method according to claim 1, including the method described in claim 1.

3. The method according to claim 2, wherein the set of one or more execution criteria includes a user authentication criterion that is satisfied when the user is authenticated.

4. To detect biometric information from the aforementioned user, The determination of whether the user authentication criteria are met includes comparing the biometric information with registered biometric information, and the determination of whether the user authentication criteria are met is based on the comparison between the biometric information and the registered biometric information. The method according to claim 3.

5. The method according to claim 3 or 4, wherein the determination that the user authentication criteria are met is made in response to receiving the input for permitting the execution of the secure operation.

6. The method according to any one of claims 3 to 5, wherein the determination of whether the user authentication criteria are met includes obtaining stored authentication information, wherein the stored authentication information indicates one or more previously performed biometric authentications.

7. The determination of whether the user authentication criteria are met is Performing the first type of biometric authentication, The method includes repeatedly performing a second type of biometric authentication after performing the first type of biometric authentication, wherein the second type of biometric authentication is Receiving input from the user that includes a first individual biometric information, This includes comparing the received input with a previously received input that includes a second individual biometric information, The method according to any one of claims 3 to 6.

8. Performing the first type of biometric authentication includes detecting the presence of a first type of biometric feature, Performing the second type of biometric authentication described above includes detecting the presence of a second biometric type, wherein the first biometric type and the second biometric type correspond to the same biometric type. The method according to claim 7.

9. The method according to any one of claims 3 to 8, wherein the user authentication criteria are satisfied when the user is authenticated based on biometric information of the eyes.

10. The method according to any one of claims 3 to 9, wherein the user authentication criterion is satisfied when the entered passcode matches a registered passcode.

11. In accordance with the determination that the user authentication criteria are not met, the system includes providing a prompt to enable authentication using a passcode. The method according to any one of claims 3 to 10.

12. The method according to any one of claims 1 to 11, wherein user authorization of the secure operation using the virtual user interface object includes user activation of a hardware user interface element.

13. The method according to any one of claims 1 to 12, wherein the secure operation is a payment.

14. The method according to any one of claims 1 to 13, wherein the secure operation includes automatically filling in user credentials.

15. In response to detecting the change in the user's viewpoint, the amount of virtual user interface objects visible to the user is changed. The method according to any one of claims 1 to 14, including the method described in any one of claims 1 to 14.

16. The method according to claim 15, wherein the amount of the visible virtual user interface object changes based on occlusion by a physical object.

17. The method according to claim 15, wherein the amount of the visible virtual user interface object changes based on occlusion by the virtual object.

18. The method according to any one of claims 1 to 17, wherein the amount of the visible virtual user interface objects changes based on the virtual user interface objects moving out of the user's field of view.

19. The method according to any one of claims 1 to 18, wherein the amount of visible virtual user interface objects changes based on the virtual user interface objects moving beyond a threshold distance from the center of the user's field of view.

20. The method according to any one of claims 1 to 19, wherein the amount of the visible virtual user interface objects changes based on occlusion by physical objects.

21. The method according to any one of claims 1 to 20, wherein the determination that the user is permitted to perform the secure operation includes determining whether a gaze criterion is met when the user is gazing at the virtual user interface object.

22. The method according to any one of claims 1 to 21, wherein the determination that the user is permitted to perform the secure operation includes determining whether the eye criterion is met when the user's eyes are open.

23. In accordance with the determination that at least a threshold amount of the virtual user interface objects is visible from the user's viewpoint, the appearance of the virtual user interface objects is modified to indicate to the user that the user can request the execution of the secure operation. In accordance with the determination that the amount of the virtual user interface object visible from the user's viewpoint is less than the threshold amount, the appearance of the virtual user interface object is modified to indicate to the user that the user cannot request the execution of the secure operation. The method according to any one of claims 1 to 22, including the method described in any one of claims 1 to 22.

24. The three-dimensional environment includes virtual objects related to the secure operation, and the method This includes attaching the virtual user interface object to the virtual object related to the secure operation, wherein the movement of the virtual object causes the attached virtual user interface object to move together with the virtual object. The method according to any one of claims 1 to 23.

25. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 24.

26. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system One or more processors, A computer system comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 24.

27. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system A computer system comprising means for performing the method described in any one of claims 1 to 24.

28. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, wherein the one or more programs are A three-dimensional environment, which includes a virtual user interface object containing information about secure operation, is visible via the display generation component, while a change in the user's current viewpoint is detected via one or more input devices. In response to detecting the change in the user's viewpoint, From the user's perspective, at least a threshold amount of the virtual user interface objects is visible, and in accordance with the determination that the user is permitted to perform the secure operation, user permission for the secure operation using the virtual user interface objects is enabled. The command includes, in accordance with the determination that the number of virtual user interface objects visible from the user's viewpoint is less than the threshold amount, a command to deactivate user permission for the secure operation using the virtual user interface objects, Non-temporary computer-readable storage medium.

29. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are A three-dimensional environment, which includes a virtual user interface object containing information about secure operation, is visible via the display generation component, while a change in the user's current viewpoint is detected via one or more input devices. In response to detecting the change in the user's viewpoint, From the user's perspective, at least a threshold amount of the virtual user interface objects is visible, and in accordance with the determination that the user is permitted to perform the secure operation, user permission for the secure operation using the virtual user interface objects is enabled. The command includes, in accordance with the determination that the number of virtual user interface objects visible from the user's viewpoint is less than the threshold amount, a command to deactivate user permission for the secure operation using the virtual user interface objects, Computer system.

30. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system A means for detecting changes in the user's current viewpoint via one or more input devices, while the three-dimensional environment, which includes a virtual user interface object containing information about secure operation, is visible via the display generation component, In response to detecting the change in the user's viewpoint, Means for enabling user permission to perform the secure operation using the virtual user interface objects, based on the determination that at least a threshold amount of the virtual user interface objects is visible from the user's perspective and the user is permitted to perform the secure operation, The system includes means for deactivating user permission for the secure operation using the virtual user interface objects, based on the determination that the number of virtual user interface objects visible from the user's viewpoint is less than the threshold amount. Computer system.

31. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 24.

32. A computer program product comprising a display generation component and one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are A three-dimensional environment, which includes a virtual user interface object containing information about secure operation, is visible via the display generation component, while a change in the user's current viewpoint is detected via one or more input devices. In response to detecting the change in the user's viewpoint, From the user's perspective, at least a threshold amount of the virtual user interface objects is visible, and in accordance with the determination that the user is permitted to perform the secure operation, user permission for the secure operation using the virtual user interface objects is enabled. The command includes, in accordance with the determination that the number of virtual user interface objects visible from the user's viewpoint is less than the threshold amount, a command to deactivate user permission for the secure operation using the virtual user interface objects, Computer program products.

33. In a computer system communicating with one or more input devices, display generation components, and biosensors, At a first time, biometric authentication of the user of the device using the biometric sensor to perform a first type of biometric authentication, Receiving a request to perform a secure operation at a second time after the first time, Upon receiving the request to perform the aforementioned secure operation, The secure operation is performed without performing the first type of biometric authentication after receiving the request to perform the secure operation, based on the determination that the user of the device has met the respective criteria based on sensor measurements taken at multiple intermediate times between the first time and the second time, including sensor measurements taken at a first intermediate time and sensor measurements taken at a second intermediate time, which indicate that the same user was using the device between the first time and the second time, and that the user of the device has met the criteria between the first time and the second time. This includes, in accordance with the determination that the user of the device did not meet each criterion between the first time and the second time, ceasing to perform the secure operation, method.

34. Upon receiving the request to perform the aforementioned secure operation, This includes biometric authentication of the user of the device using the biometric sensor for performing the first type of biometric authentication, in accordance with the determination that the user of the device did not meet each criterion between the first time and the second time, The method according to claim 33.

35. The method according to claim 33 or 34, wherein the first type of biometric authentication is based on the biometric characteristics of the eyes.

36. The method according to any one of claims 33 to 35, wherein the first type of biometric authentication is based on facial biometric features, and the second type of biometric authentication is based on facial biometric features.

37. Each criterion being met by the user of the device includes repeatedly performing a second type of biometric authentication using one or more sensors of the computer system, and the second type of biometric authentication is Receiving input from the user that includes a first individual biometric information, This includes comparing the received input with a previously received input that includes a second individual biometric information, The method according to any one of claims 33 to 36.

38. The method according to any one of claims 33 to 37, wherein the one or more sensors used to perform the second type of biometric authentication include the biosensors used to perform the first type of biometric authentication.

39. The method according to any one of claims 33 to 38, wherein the first type of biometric authentication is based on a first biometric feature, the second type of biometric authentication is based on a second type of biometric feature, and the first biometric feature is different from the second biometric feature.

40. The method according to any one of claims 33 to 38, wherein the first type of biometric authentication is based on a first biometric feature, and the second type of biometric authentication is based on the first type of biometric feature.

41. At the first time, biometric authentication of the user of the device based on the user's first biometric characteristics is performed. The method according to claims 33 to 40, comprising acquiring sensor measurements at a plurality of intermediate time intervals between the first time interval and the second time interval, wherein the acquired sensor measurements include at least one sensor measurement indicating that the first biometric feature of the user was not detected at any of the intermediate time intervals.

42. At the first time, biometric authentication of the user of the device based on the user's first biometric characteristics is performed. This includes periodically acquiring sensor measurements of the user's second biological characteristics at a plurality of intermediate time points between the first time point and the second time point, The acquired sensor measurement values ​​include at least one sensor measurement value indicating that the user's first biological characteristic was not detected at any individual intermediate time point, The acquired sensor measurement values ​​include at least one sensor measurement value indicating that the user's second biological characteristic was detected at the individual intermediate time points, The first biological characteristic is different from the second biological characteristic. The method according to claim 41.

43. The method according to any one of claims 33 to 42, wherein each of the criteria between the first time and the second time includes a continuity criterion that is satisfied when the biological feature is repeatedly detected between the first time and the second time.

44. To perform a first type of biometric authentication, the biometric sensor is used to biometrically authenticate the user of the device. This includes providing the user with an authentication guidance user interface that includes instructions for performing the first type of biometric authentication, wherein the instructions require the user to provide specific movements of the biometric features. The method according to any one of claims 33 to 43.

45. The method according to any one of claims 33 to 44, wherein the secure operation includes providing payment information.

46. The method according to any one of claims 33 to 45, wherein the secure operation includes providing access credentials for accessing the application.

47. The method according to any one of claims 33 to 46, wherein the secure operation includes automatically filling in user credentials.

48. The method according to any one of claims 33 to 47, wherein receiving the request to perform the secure operation includes detecting a physical input to activate a hardware button.

49. The system includes providing the user with a system user interface that includes affordances related to the secure operation, and receiving the request to perform the secure operation includes detecting the activation of the affordances related to the secure operation. The method according to any one of claims 33 to 48.

50. Receiving the request for the execution of the secure operation, This includes detecting user activation of affordances within a virtual user interface object while providing the user with a virtual user interface object containing information about secure operation, The method according to any one of claims 33 to 49.

51. The method according to any one of claims 33 to 50, wherein each of the aforementioned criteria includes an eye-opening criterion that is met when one or more of the user's eyes are not continuously closed beyond a threshold period, and the eye-opening criterion is not met when one or more of the user's eyes are continuously closed beyond the threshold period.

52. The method according to any one of claims 33 to 50, wherein each of the aforementioned criteria is based on a biological characteristic that is available for analysis regardless of whether the user's eyes are closed or not.

53. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, a biosensor, and one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 33 to 52.

54. A computer system communicating with a display generation component, a biosensor, and one or more input devices, wherein the computer system is One or more processors, A computer system comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 33 to 52.

55. A computer system communicating with a display generation component, a biosensor, and one or more input devices, wherein the computer system is A computer system comprising means for performing the method described in any one of claims 33 to 52.

56. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, a biosensor, and one or more input devices, wherein the one or more programs are At a first time, the biometric sensor is used to biometrically authenticate the user of the device in order to perform a first type of biometric authentication. A request to perform a secure operation is received at a second time after the first time, Upon receiving the request to perform the aforementioned secure operation, Sensor measurements taken at multiple intermediate times between the first time and the second time, including sensor measurements taken at a first intermediate time and sensor measurements taken at a second intermediate time, and each of these criteria based on sensor measurements that indicate the same user was using the device between the first time and the second time, are met by the user of the device between the first time and the second time, and the secure operation is performed without performing the first type of biometric authentication after receiving the request to perform the secure operation. The command includes an instruction to discontinue the secure operation in accordance with the determination that the user of the device did not meet each criterion between the first time and the second time, Non-temporary computer-readable storage medium.

57. A computer system communicating with a display generation component, a biosensor, and one or more input devices, wherein the computer system is One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are At a first time, the biometric sensor is used to biometrically authenticate the user of the device in order to perform a first type of biometric authentication. A request to perform a secure operation is received at a second time after the first time, Upon receiving the request to perform the aforementioned secure operation, Sensor measurements taken at multiple intermediate times between the first time and the second time, including sensor measurements taken at a first intermediate time and sensor measurements taken at a second intermediate time, and each of these criteria based on sensor measurements that indicate the same user was using the device between the first time and the second time, are met by the user of the device between the first time and the second time, and the secure operation is performed without performing the first type of biometric authentication after receiving the request to perform the secure operation. The command includes an instruction to discontinue the secure operation in accordance with the determination that the user of the device did not meet each criterion between the first time and the second time, Computer system.

58. A computer system communicating with a display generation component, a biosensor, and one or more input devices, wherein the computer system is A means for biometrically authenticating the user of the device using the biometric sensor to perform a first type of biometric authentication at a first time, A means for receiving a request to perform a secure operation after the first time at a second time, Upon receiving the request to perform the aforementioned secure operation, Means for performing the secure operation without performing the first type of biometric authentication after receiving the request to perform the secure operation, based on the determination that the user of the device met the respective criteria based on the sensor measurements taken at multiple intermediate times between the first time and the second time, including sensor measurements taken at a first intermediate time and sensor measurements taken at a second intermediate time, which indicate that the same user was using the device between the first time and the second time, The system includes means for ceasing to perform the secure operation if it is determined that the user of the device did not meet each criterion between the first time and the second time, Computer system.

59. A computer program product comprising a display generation component, a biosensor, and one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 33 to 52.

60. A computer program product comprising a display generation component, a biosensor, and one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are At a first time, the biometric sensor is used to biometrically authenticate the user of the device in order to perform a first type of biometric authentication. A request to perform a secure operation is received at a second time after the first time, Upon receiving the request to perform the aforementioned secure operation, Sensor measurements taken at multiple intermediate times between the first time and the second time, including sensor measurements taken at a first intermediate time and sensor measurements taken at a second intermediate time, and each of these criteria based on sensor measurements that indicate the same user was using the device between the first time and the second time, are met by the user of the device between the first time and the second time, and the secure operation is performed without performing the first type of biometric authentication after receiving the request to perform the secure operation. The command includes an instruction to cancel the execution of the secure operation in accordance with the determination that the user of the device did not meet the respective criteria between the first time and the second time, Computer program products.

61. In a computer system that communicates with one or more input devices and display generation components, Receiving a request via one or more input devices for the display of a user interface related to performing secure operations, In response to the request for the display of the user interface relating to performing the secure operation, the display generation component includes displaying a first user interface via the display generation component, which includes separate prompts for providing additional input to the device to authorize the device to perform the secure operation. In accordance with the determination that the computer system is operating in a first mode, the individual prompt is a first prompt for providing physical input to permit the secure operation, In accordance with the determination that the computer system is operating in a second mode different from the first mode, the individual prompt is a second prompt that displays a second user interface which enables user authorization for the secure operation without using the physical input. method.

62. The second prompt includes providing the first individual affordance, Receiving the user selection of the aforementioned affordance, The process includes, upon receiving the user selection of the aforementioned affordance, displaying the second user interface which includes a representation of multiple functions, and the multiple functions being Among the aforementioned multiple functions, a first individual function is called according to a secondary input type different from the primary input type, The above-mentioned multiple functions include a second individual function which is called according to a tertiary input type different from the primary input type and the secondary input type, The method according to claim 61.

63. The method according to claim 62, wherein the secondary input type simulates the input of the primary input type, and the tertiary input type simulates the input of the primary input type.

64. While the computer system is operating in the second mode, Displaying affordances related to the individual prompts, The process includes, upon receiving user input related to the individual prompts, displaying the second user interface, wherein the second user interface includes affordances for enabling the secure operation, the affordances being invoked in accordance with a secondary input type different from the primary input type. The method according to any one of claims 61 to 63.

65. The user interface related to performing the secure operation displays details related to the execution of the secure operation, While the computer system is operating in the second mode, Displaying affordances related to the individual prompts, Receiving user input related to the individual prompts, The process includes displaying the second user interface in accordance with the determination that the user is looking at the user interface related to the execution of the secure operation while user input related to the individual prompt is being received, The second user interface includes affordances for allowing the secure operation, The affordance for allowing the secure operation is displayed according to a secondary input type different from the primary input type. The method according to claims 61 to 64.

66. While the computer system is operating in the first mode, Receiving user activation of the physical input while the first prompt is displayed, In response to receiving user activation of the physical input while the first prompt is displayed, The secure operation is permitted in accordance with the determination that the user authentication criteria are met, This includes, in accordance with the determination that the user authentication criteria are not met, ceasing to permit the secure operation, The method according to claims 61 to 65.

67. The method according to claim 66, wherein the authentication criteria include criteria that are met when the user provides specific movements of biometric characteristics.

68. Receiving input from the user to authorize the aforementioned secure operation, Upon receiving the input for granting the secure operation, Performing user authentication and In accordance with the determination that the user is authenticated, the secure operation is permitted, The method according to claims 61 to 67, including the method described in claims 61 to 67.

69. Performing user authentication includes performing a second type of biometric authentication, and the second type of biometric authentication is Receiving input from the user that includes a first individual biometric information, This includes comparing the received input with a previously received input that includes a second individual biometric information, The method according to claim 68.

70. The method according to claim 68 or 69, wherein performing user authentication includes performing a first type of biometric authentication, and the first type of biometric authentication includes detecting the presence of a first type of biometric feature.

71. Performing user authentication is To provide a prompt requesting a passcode from the user, In response to the aforementioned prompt, receive a passcode from the user, In accordance with the determination that the received passcode matches the stored passcode, If it is determined that the aforementioned user is authenticated, This includes, in accordance with the determination that the user is authenticated, allowing the secure operation, The method according to any one of claims 68 to 70.

72. To detect the input necessary to permit the aforementioned secure operation, In response to detecting the input for permitting the secure operation, In accordance with the determination that the input is a physical input for permitting the secure operation, The secure operation is permitted in accordance with the determination that the user is authenticated based on the second type of biometric authentication. In accordance with the determination that the input is an input related to the second user interface that enables user authorization of the secure operation without using the physical input, Performing the first type of biometric authentication, This includes, in accordance with the determination that the user is authenticated based on the first type of biometric authentication, allowing the secure operation. The method according to any one of claims 61 to 71.

73. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 61 to 72.

74. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system One or more processors, A computer system comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 61 to 72.

75. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system A computer system comprising means for performing the method described in any one of claims 61 to 72.

76. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, wherein the one or more programs are A request for the display of a user interface related to performing secure operations is received via one or more input devices. In response to the request for the display of the user interface relating to performing the secure operation, the command includes a command to display a first user interface via the display generation component, which includes separate prompts for providing additional input to the device to authorize the performance of the secure operation, In accordance with the determination that the computer system is operating in a first mode, the individual prompt is a first prompt for providing physical input to permit the secure operation, A non-temporary computer-readable storage medium in which, upon determination that the computer system is operating in a second mode different from the first mode, the individual prompt is a second prompt that displays a second user interface which enables user authorization of the secure operation without using the physical input.

77. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are A request for the display of a user interface related to performing secure operations is received via one or more input devices. In response to the request for the display of the user interface relating to performing the secure operation, the command includes a command to display a first user interface via the display generation component, which includes separate prompts for providing additional input to the device to authorize the performance of the secure operation, In accordance with the determination that the computer system is operating in a first mode, the individual prompt is a first prompt for providing physical input to permit the secure operation, In accordance with the determination that the computer system is operating in a second mode different from the first mode, the individual prompt is a second prompt that displays a second user interface which enables user authorization for the secure operation without using the physical input. Computer system.

78. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system Means for receiving, via one or more input devices, a request for the display of a user interface related to performing secure operations, The system includes means for displaying a first user interface via the display generation component, which includes separate prompts for providing additional input to the device to authorize the device to perform the secure operation, in response to the request for the display of the user interface related to performing the secure operation, In accordance with the determination that the computer system is operating in a first mode, the means wherein the individual prompt is a first prompt for providing physical input to permit the secure operation, In accordance with the determination that the computer system is operating in a second mode different from the first mode, the means is that the individual prompt is a second prompt which displays a second user interface that enables user authorization of the secure operation without using the physical input, Computer system.

79. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 61 to 72.

80. A computer program product comprising a display generation component and one or more programs configured to be executed by one or more processors of a computer system communicating with one or more input devices, wherein the one or more programs are A request for the display of a user interface related to performing secure operations is received via one or more input devices. In response to the request for the display of the user interface relating to performing the secure operation, the command includes a command to display a first user interface via the display generation component, which includes separate prompts for providing additional input to the device to authorize the performance of the secure operation, In accordance with the determination that the computer system is operating in a first mode, the individual prompt is a first prompt for providing physical input to permit the secure operation, In accordance with the determination that the computer system is operating in a second mode different from the first mode, the individual prompt is a second prompt that displays a second user interface which enables user authorization for the secure operation without using the physical input. Computer program products.

Images