Multi-mode posture sequence password unlocking device for head-mounted display

Abstract

The invention relates to a multi-mode posture sequence password unlocking device for a head-mounted display. Example apparatuses, computer-implemented methods, and computer program products are provided for unlocking a locking feature of a head-mounted display with a plurality of sensors on the head-mounted display. An example apparatus includes a head-mounted display, a plurality of sensors mounted on the head-mounted display and configured to detect a pose of a user of the head-mounted display, and a multi-modal unlocking model. The multi-modal unlocking model also includes one or more processors and one or more storage devices, the one or more storage devices storing instructions, the instructions, when executed by the one or more processors, are operable to cause the one or more processors to detect a gesture sequence password and unlock a locking feature of the head mounted display based on the gesture sequence password, the gesture sequence cryptographic includes one or more cryptographic gestures of a series of cryptographic gestures defined by physical interaction with the head mounted display.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to unlocking a locking feature on a head-mounted display, and more particularly to using multi-modal gesture sequence password detection to unlock a locking feature on a head-mounted display. Background Art

[0002] Locking mechanisms are an important aspect of mobile phones, smart glasses, watches, computers, and other similar wearable electronic devices. As the amount of personal information accessible on these devices increases, it becomes increasingly important to lock and unlock such electronic devices. Head-mounted displays such as virtual reality headsets and smart glasses pose unique problems related to locking and unlocking mechanisms.

[0003] The applicant has identified many technical challenges and difficulties associated with locking and unlocking a head-mounted display. Through applied effort, ingenuity, and innovation, the applicant has solved the problems related to locking and unlocking a head-mounted display by developing the solutions embodied in the present disclosure and described in detail below. Summary of the Invention

[0004] Various embodiments are directed to example apparatuses, computer-implemented methods, and computer program products for unlocking a locking feature of a head-mounted display using multiple sensors on the head-mounted display. The example apparatus may include a head-mounted display, a plurality of sensors mounted on the head-mounted display and configured to detect gestures of a user of the head-mounted display, and a multi-modal unlocking model. The multi-modal unlocking model may also include one or more processors and one or more storage devices, the one or more storage devices storing instructions that, when executed by the one or more processors, operate to cause the one or more processors to detect a gesture sequence password and unlock the locking feature of the head-mounted display based on the gesture sequence password, the gesture sequence password including one or more password gestures from a series of password gestures defined by physical interaction with the head-mounted display.

[0005] In some embodiments, the multi-modal unlocking model is configured to detect a user's head orientation as at least one password gesture among one or more password gestures including the gesture sequence password in the series of password gestures.

[0006] In some embodiments, the multi-modal unlocking model is configured to detect the eye state of each eye of the user as at least one password gesture among one or more password gestures including the gesture sequence password in the series of password gestures.

[0007] In some embodiments, the eye state includes open or closed.

[0008] In some embodiments, the multimodal unlocking model is configured to detect the user's heading as at least one of one or more cryptographic gestures in a series of cryptographic gestures including a gesture sequence password.

[0009] In some embodiments, the user's heading is associated with a relative heading with respect to an initial heading.

[0010] In some embodiments, the multimodal unlocking model is configured to detect the user's hand gesture relative to the head-mounted display as at least one of one or more cryptographic gestures in a series of cryptographic gestures including a gesture sequence password.

[0011] In some embodiments, the hand gesture includes one or more taps on the head-mounted display.

[0012] In some embodiments, the hand gesture includes detecting the user's directional contact on the surface of the head-mounted display.

[0013] In some embodiments, the hand gesture includes positioning the hand near a predetermined position on the head-mounted display.

[0014] In some embodiments, the multimodal unlocking model is configured to detect the user's face gesture of the head-mounted display as at least one of one or more cryptographic gestures in a series of cryptographic gestures including a gesture sequence password.

[0015] In some embodiments, the physical interaction includes physical contact between the user's hand and the head-mounted display.

[0016] In some embodiments, the physical interaction includes one or more taps on the head-mounted display, or the user's directional contact on the surface of the head-mounted display.

[0017] In some embodiments, the plurality of sensors includes at least a head orientation sensor, a charge change sensor, a proximity sensor, or an ambient light sensor.

[0018] In some embodiments, the head orientation sensor includes an inertial measurement unit.

[0019] In some embodiments, the multimodal unlocking model is further configured to detect a deletion gesture and delete at least one of one or more cryptographic gestures in the series of cryptographic gestures based on the deletion gesture.

[0020] In some embodiments, at least one of one or more cryptographic gestures in the series of cryptographic gestures includes a plurality of gesture states.

[0021] In some embodiments, the multimodal unlocking model is configured to detect a start cryptographic gesture that indicates the start of a gesture sequence password.

[0022] A computer-implemented method for unlocking a locking feature of a head-mounted display using multiple sensors on the head-mounted display is also provided. In some embodiments, the computer-implemented method may include: receiving, at a multimodal unlocking model, one or more electrical signals, the one or more electrical signals representing user interaction with at least one of the multiple sensors mounted on the head-mounted display. The computer-implemented method further includes detecting a gesture sequence password, the gesture sequence password including one or more password gestures from a series of password gestures defined by physical interaction with the head-mounted display. The computer-implemented method further includes unlocking the locking feature of the head-mounted display based on the gesture sequence password.

[0023] A computer program product for unlocking a locking feature of a head-mounted display using multiple sensors on the head-mounted display is also provided. The computer program product includes at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions including executable portions configured to receive, at a multimodal unlocking model, one or more electrical signals, the one or more electrical signals representing user interaction with at least one of the multiple sensors mounted on the head-mounted display. The executable portion is further configured to detect a gesture sequence password, the gesture sequence password including one or more password gestures from a series of password gestures defined by physical interaction with the head-mounted display. The executable portion is further configured to unlock the locking feature of the head-mounted display based on the gesture sequence password. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Reference will now be made to the drawings. The components illustrated in the drawings may or may not be present in certain embodiments described herein. Some embodiments may include fewer (or more) components than those shown in the drawings for example embodiments according to the present disclosure.

[0025] Figure 1 A block diagram of an example head-mounted multimodal unlocking device according to an example embodiment of the present disclosure is depicted.

[0026] Figure 2 A block diagram of an example device that may be specifically configured according to an example embodiment of the present disclosure is illustrated.

[0027] Figure 3 A perspective view of a head-mounted multimodal unlocking device according to an example embodiment of the present disclosure is depicted.

[0028] Figure 4 A front view of a head-mounted multimodal unlocking device positioned on a user according to an example embodiment of the present disclosure is depicted.

[0029] Figure 5 FIG. Figure 5 illustrates an example block diagram of a head - mounted multi - modal unlocking device including a plurality of sensors according to an example embodiment of the present disclosure.

[0030] Figure 6 FIG. Figure 6 depicts an example head orientation of a user wearing a head - mounted multi - modal unlocking device according to an example embodiment of the present disclosure.

[0031] Figure 7 FIG. Figure 7 illustrates an example heading of a user wearing a head - mounted multi - modal unlocking device according to an example embodiment of the present disclosure.

[0032] Figure 8 FIG. Figure 8 illustrates an example eye pose of a user wearing a head - mounted multi - modal unlocking device according to an example embodiment of the present disclosure.

[0033] Figure 9 FIG. Figure 9 illustrates an example hand pose of a user relative to a head - mounted multi - modal unlocking device according to an example embodiment of the present disclosure.

[0034] Figure 10 FIG. Figure 10 illustrates an additional example hand pose of a user relative to a head - mounted multi - modal unlocking device according to an example embodiment of the present disclosure.

[0035] Figure 11 FIG. Figure 11 illustrates an example pose - sequence password according to an example embodiment of the present disclosure.

[0036] Figure 12 FIG. Figure 12 illustrates an example pose - sequence password according to an example embodiment of the present disclosure, where the example pose - sequence password includes multiple pose states in each password pose.

[0037] Figure 13 FIG. Figure 13 illustrates a process of an example operation for unlocking a head - mounted display using a multi - modal unlocking model according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] Embodiments will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments of the invention of the present disclosure are shown. In fact, the embodiments of the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals always refer to like elements.

[0039] OVERVIEW

[0040] Various example embodiments solve the technical problems associated with locking and unlocking a head-mounted display without using a text input device, a fingerprint scanner, a retina scanner, or similar biometric authentication devices. As understood, there are many example scenarios where a user may need to unlock a head-mounted display without access to a text input device (such as a keyboard or touch screen) or other biometric authentication devices.

[0041] Locking mechanisms are an important aspect of mobile phones, smart glasses, watches, computers, and other similar personal electronic devices. As the amount of personal information accessible on these personal electronic devices increases, it becomes increasingly important to lock and unlock these personal electronic devices. Personal electronic devices often use locking mechanisms such as passwords, personal identification number (PIN) codes, fingerprint scanners, facial recognition, and even retina scanners to restrict access to digital content on the electronic device to authorized users.

[0042] Head-mounted displays such as virtual reality headsets and smart glasses are becoming increasingly common. Many of these head-mounted displays enable the execution of many functions that are performed on a smart phone. For example, the head-mounted device may enable a user to send and receive messages, receive notifications, access the Internet, access personal accounts, capture and view photos and videos, etc. Therefore, the need for a digital locking mechanism on the head-mounted device increases.

[0043] Many traditional locking mechanisms, such as passwords and PIN codes, require an input mechanism such as a touch screen, keyboard, controller, etc. to enter the password or PIN code. In addition, other traditional locking mechanisms can perform facial recognition, fingerprint scanning, or retina scanning to identify the user. These mechanisms may be impossible or cumbersome in a head-mounted device. For example, the head-mounted device may not include a touch screen, controller, or other input device. In addition, sensors such as optical sensors, fingerprint scanners, and retina scanners may take up space, increase weight, and increase power consumption. For wearable devices such as head-mounted displays, manufacturers want to reduce the size and weight of the electronic components while extending the battery life. Biometric sensors also carry some risks of privacy invasion and identity theft. In addition, personal identification features (such as retina and fingerprint scans) may make sharing these devices difficult.

[0044] Therefore, an unlocking mechanism that utilizes the available sensors of a head-mounted device is required. The unlocking mechanism provides sufficient security to allow only authorized users to access the device and protects the privacy of the user. Utilizing the available sensors reduces the space and power resources dedicated to unlocking the wearable device. In addition, an unlocking mechanism that uses non-biometric sensors for identification can protect the privacy of the user. Furthermore, benefits can be obtained through an unlocking mechanism that can be shared with others.

[0045] The various example embodiments described herein provide a multimodal, gesture-based unlocking mechanism for a head-mounted display. The multimodal, gesture-based unlocking mechanism utilizes multiple sensors to identify passcode gestures as part of a gesture sequence passcode.

[0046] A head-mounted display may include a number of sensors to support the various functions of the head-mounted display. For example, the head-mounted display may include any combination of an inertial measurement unit (IMU), accelerometers, gyroscopes, magnetometers, proximity sensors, ambient light sensors, charge variation sensors, etc. Each device is configured to sense various physical characteristics of the surrounding environment. For example, a charge variation sensor may be configured to detect a change in charge of a nearby surface. The IMU may be used to detect the specific force of the head-mounted display, the angular rate of the head-mounted display, or even the orientation of the head-mounted display. A magnetometer may be able to detect the magnetic field of the head-mounted display relative to the Earth and / or the heading relative to a previous heading. An ambient light sensor may be able to detect the amount of light in the surrounding environment. A proximity sensor may detect the presence and / or movement of an object near the head-mounted display.

[0047] The various embodiments described herein may utilize the sensors of the head-mounted display to detect gestures performed by a user on the head-mounted display. A gesture is any movement of a part of the user's body (e.g., head, hand, face, eyes, etc.). The various sensors of the head-mounted display may be configured to identify the user's gestures. For example, a charge variation sensor may be used to detect face gestures such as blinking, raising eyebrows, squinting, and even facial expressions such as smiling and frowning. The IMU may use gyroscopes and accelerometers to determine the orientation of the user's head. The magnetometer in combination with the IMU may be used to determine the direction in which the user's head is facing. Hand gestures may also be detected by ambient light sensors, proximity sensors, and even charge variation sensors.

[0048] One or more gestures can be detected by a multimodal unlocking model and recognized as passcode gestures. In some embodiments, a user can register a sequence of passcode gestures as a gesture sequence passcode. In the case where the head-mounted display is locked, the user can perform a sequence of passcode gestures to unlock one or more locking features of the head-mounted display. Recognizing the sequence of passcode gestures as matching a registered gesture sequence code provides the authentication required to unlock one or more locking features.

[0049] The multimodal unlocking model can also be configured to recognize physical interaction delimiters between passcode gestures. For example, the user can tap the head-mounted display or lift the head-mounted display to distinguish passcode gestures. Additionally, in some embodiments, a starting passcode gesture can be detected to indicate the start of a gesture sequence passcode. In some embodiments, multiple gesture states can be registered as passcode gestures. For example, the passcode gestures in a gesture sequence passcode can include the user sliding a hand along the frame of the head-mounted display and blinking both eyes.

[0050] As a result of the example embodiments described herein and in some examples, the secure operation of the head-mounted display can be significantly improved. Using multiple sensors to recognize the various gestures in a gesture sequence code can improve the security of the head-mounted display. For example, many types of gestures (including hand gestures, head orientation gestures, face gestures, etc.) can be combined in any order or combination, resulting in millions of possible combinations, thus making the gesture sequence code almost impossible to guess. Additionally, the gestures must be physically performed, making it difficult for an automated system to attempt a brute force attack on the head-mounted display. Further, the combination of gesture states and the discrete nature of many gestures makes it difficult for onlookers to observe and replicate the passcode.

[0051] Additionally, using the existing sensors on the head-mounted display can reduce the size and weight of the head-mounted display while extending battery life. Many sensors are provided on the head-mounted display to support the numerous functions of the device. Additionally, common authentication devices (such as fingerprint scanners, retina scanners, and cameras for face detection) can be bulky and utilize a large amount of power resources. By using low-size and power sensors with additional functions available on the head-mounted display, a significant improvement in battery utilization and size reduction of the head-mounted display can be achieved while still supporting security.

[0052] Furthermore, a gesture sequence passcode including passcode gestures can be shared with trusted users. The gesture-based passcode sequence does not rely on a personally unique biometric. Thus, the gesture sequence passcode can be taught and shared with other users of the head-mounted display, thereby supporting more than one individual to collaborate on the wearable device.

[0053] Example Devices

[0054] Referring now to Figure 1 , there is provided a block diagram of an example head-mounted multi-modal unlocking device 100. As Figure 1 depicted, the multi-modal unlocking model 102 is communicatively coupled to a plurality of head-mounted sensors 104. The depicted multi-modal unlocking model 102 includes an attitude detection circuitry 106, an attitude sequence identification circuitry 108, and an attitude sequence authentication circuitry 110.

[0055] As Figure 1 depicted, the example head-mounted multi-modal unlocking device includes a plurality of head-mounted sensors 104. A head-mounted sensor 104 is any electrical, mechanical, and / or electromechanical device that is mounted on or within the surface of a head-mounted display and is configured to measure the properties of a physical environment and transmit an electrical signal corresponding to the measured physical property. For example, the head-mounted sensor 104 may include an accelerometer (e.g., the accelerometer 534 described with respect to Figure 5 ), a gyroscope (e.g., the gyroscope 535 described with respect to Figure 5 ), a magnetometer (e.g., the magnetometer 536 described with respect to Figure 5 ), an IMU (e.g., the inertial measurement unit 550 described with respect to Figure 5 ), a charge change sensor (e.g., the charge change sensor 533 described with respect to Figure 5 ), an ambient light sensor (e.g., the ambient light sensor 544 described with respect to Figure 5 ), a proximity sensor (e.g., the proximity sensor 542 described with respect to Figure 5 ), etc.

[0056] As also depicted in Figure 1 , the example head-mounted multi-modal unlocking device includes a multi-modal unlocking model 102. A multi-modal unlocking model 102 is any hardware, software, and / or combination thereof that is configured to receive electrical signals from a plurality of head-mounted sensors 104, detect an attitude based on the electrical signals, and authenticate an attitude sequence password based on the detected attitude. As Figure 1 depicted, the example multi-modal unlocking model 102 includes an attitude detection circuitry 106, an attitude sequence identification circuitry 108, and an attitude sequence authentication circuitry 110.

[0057] The pose detection circuit device 106 is any hardware, software, and / or any combination thereof configured to detect the pose of a user of the head-mounted display based on electrical signals received from one or more head-mounted sensors 104. The pose detection circuit device 106 can utilize any mechanism including a machine learning mechanism to identify the user pose based on the electrical signals of the one or more head-mounted sensors 104. For example, the pose detection circuit device 106 can identify head orientation poses, such as the user's head being down, up, left, or right. Additional head orientation poses can include tilting the head left or right. Regarding Figure 6 example head orientation poses are described. In some embodiments, the pose detection circuit device 106 can also detect the direction heading of a user wearing the head-mounted display relative to the earth's magnetic force. For example, the pose detection circuit device 106 can determine the rotational movement, positioning, or heading of the head-mounted display in degrees relative to an initial orientation (e.g., 0 degrees, 45 degrees, -45 degrees, 90 degrees, etc.). Example head-mounted display headings are described regarding Figure 7 The pose detection circuit device 106 can also be configured to detect movement poses, such as nodding, shaking the head, tilting the head, etc.

[0058] In some embodiments, the pose detection circuit device 106 can be configured to identify hand poses based on electrical signals from one or more head-mounted sensors 104. For example, the pose detection circuit device 106 can identify the proximity of the hand to the head-mounted display, the orientation of the hand (e.g., open hand, clenched fist, spread fingers, etc.), the tapping of the hand on the head-mounted display, the directional contact of the user with the head-mounted display (e.g., sliding a finger forward or backward along the surface of the head-mounted display), etc. Example hand poses are described regarding Figure 9 and Figure 10 are described.

[0059] In some embodiments, the pose detection circuit device 106 can be configured to identify face poses based on electrical signals from one or more head-mounted sensors 104. For example, the pose detection circuit device 106 can identify eye poses (e.g., right eye blink, left eye blink, blink) and face poses (e.g., raise eyebrows, smile, frown, etc.). Example face poses are described regarding Figure 8 are described.

[0060] As also shown in Figure 1As depicted in [description], the exemplary multimodal unlock model 102 includes a gesture sequence identification circuitry 108. The gesture sequence identification circuitry 108 is any hardware, software, and / or combination thereof configured to detect a gesture sequence password based on the gestures identified by the gesture detection circuitry 106. The gesture sequence password includes a string of one or more password gestures. A password gesture is any gesture recognized by the gesture detection circuitry 106 and performed by a user of the head-mounted display. For example, the password gesture may include one or more head orientation gestures, one or more hand gestures, one or more face gestures, or any combination thereof.

[0061] The multimodal unlock model 102 can utilize two modes: a registration mode and an operation mode. During the registration mode, the user can register a string of password gestures as a gesture sequence password according to the prompts of the head-mounted display. During the operation mode, the gesture sequence password can be used to unlock the lock feature of the head-mounted display. For example, when the head-mounted display has not been used for a predetermined period of time or has been removed (e.g., placed on a charging station, etc.), the head-mounted display can enter a locked state. Before a user wearing the head-mounted display can view certain information, the user may need to perform the gesture sequence password registered during the registration phase.

[0062] In some embodiments, the execution of the gesture sequence password can be preceded by a start password gesture. The start password gesture is any gesture performed by the user to indicate the start of a new gesture sequence password. The start password gesture can be, for example, a double tap on the head-mounted display, a forward slide of a finger on the frame of the head-mounted display, covering a predetermined portion of the head-mounted display, or other similar gestures. Based on receiving the start password gesture, the gesture sequence identification circuitry 108 can process subsequent gestures as potential password gestures of the attempted gesture sequence password.

[0063] The password gestures of the gesture sequence password are further defined by physical interaction delimiters. A physical interaction delimiter is any interaction between the user of the head-mounted display and the head-mounted display. The physical interaction can include hand gestures, such as one or more taps on the head-mounted display, covering a portion of the head-mounted display, sliding a finger along the surface of the head-mounted display in a directional contact manner, etc. The physical interaction delimiter can also include non-contact gestures, such as a double blink. The physical interaction delimiter differentiates the password gestures in the series of password gestures. For example, if a tap on the head-mounted display is the physical interaction delimiter, the user can perform a first password gesture, then tap the head-mounted display, perform a second password gesture, then tap the head-mounted display, and so on, until the complete gesture sequence password is entered. Based on receiving the physical interaction delimiter, the gesture sequence identification circuitry 108 can store the delimited gestures as potential password gestures in the attempted gesture sequence password.

[0064] In some embodiments, a deletion gesture may be performed by a user of the head-mounted display to delete a previous password gesture from the attempted gesture sequence password. A deletion gesture is any gesture performed by the user to indicate deletion of a previous password gesture from the attempted gesture sequence password. The deletion gesture may be, for example, a double tap on the head-mounted display, a backward slide of a finger on the frame of the head-mounted display, a head shake, or other similar gesture. Based on receiving the deletion gesture, the gesture sequence identification circuitry 108 may delete a previously received gesture that was a potential password gesture from the attempted gesture sequence password.

[0065] As also depicted in Figure 1 the multimodal unlock model 102 includes gesture sequence authentication circuitry 110. The gesture sequence authentication circuitry 110 is any hardware, software, and / or combination thereof configured to store a gesture sequence password and compare the attempted gesture sequence password to the stored gesture sequence password. During the registration mode of the head-mounted multimodal unlock device 100, the gesture sequence authentication circuitry 110 stores the captured gesture sequence password. During the operation mode, an attempt is made to compare the executed gesture sequence password to the stored gesture sequence password. In the case where the executed gesture sequence password matches the stored gesture sequence password, the gesture sequence authentication circuitry 110 indicates to the head-mounted display to unlock one or more locked features based on the input gesture sequence password. In the case where the executed gesture sequence password does not match the stored gesture sequence password, the locked features remain locked.

[0066] Now referring to Figure 2 , Figure 2 FIG. illustrates a block diagram of an example device 200 that may be specifically configured in accordance with at least one example embodiment of the present disclosure. Specifically, Figure 2 FIG. illustrates a multimodal unlock model 102 in accordance with at least one example embodiment of the present disclosure. The multimodal unlock model 102 includes a processor 222, a data storage medium 226, input / output circuitry 224, communication circuitry 228, and a head-mounted sensor interface circuitry 221. In some embodiments, the multimodal unlock model 102 is configured to perform and implement one or more of the operations described herein using one or more of the sets of circuitry 221, 222, 224, 226, and / or 228.

[0067] Generally speaking, the terms computing entity (or "entity" other than a user), device, system, and / or similar terms that may be used interchangeably herein may refer to, for example, one or more computers, computing entities, desktop computers, mobile phones, tablet computers, phablets, notebook computers, laptop computers, distributed systems, item / devices, terminals, servers or server networks, blade computers, gateways, switches, processing devices, processing entities, set-top boxes, relay devices, routers, network access points, base stations, etc., and / or any combination of devices or entities suitable for performing the functions, operations, and / or processes described herein. Such functions, operations, and / or processes may include, for example, transmitting, receiving, operating, processing, displaying, storing, determining, creating / generating, monitoring, evaluating, comparing, and / or similar terms that may be used interchangeably herein. In one embodiment, these functions, operations, and / or processes may be performed on data, content, information, and / or similar terms that are used interchangeably herein. In this regard, the multimodal unlocking model 102 embodies a specific, specially configured computing entity that has been transformed to enable the specific operations described herein and to provide the specific advantages associated therewith as described herein.

[0068] Although components are described in terms of functional limitations, it should be understood that a particular implementation must include the use of specific computing hardware. It should also be understood that in some embodiments, certain components described herein include similar or common hardware. For example, in some embodiments, both sets of circuitry utilize the same (one or more) processors, (one or more) network interfaces, (one or more) storage media, etc. to perform their associated functions, such that each set of circuitry does not require duplicate hardware. The term "circuitry" as used herein with respect to the components of the devices described herein should therefore be understood to include the specific hardware configured to perform the functions associated with the specific circuitry described herein.

[0069] Specifically, the term "circuitry" should be understood broadly to include hardware and, in some embodiments, software for configuring the hardware. For example, in some embodiments, "circuitry" includes processing circuitry, storage media, network interfaces, input / output devices, etc. Alternatively or additionally, in some embodiments, other elements of the multimodal unlocking model 102 provide or supplement the functions of another specific set of circuitry. For example, in some embodiments, the processor 222 provides processing functions for any set of circuitry, the data storage medium 226 provides storage functions for any set of circuitry, the communication circuitry 228 provides network interface functions for any set of circuitry, etc.

[0070] In some embodiments, the processor 222 (and / or a coprocessor or any other processing circuitry that assists or otherwise is associated with the processor) communicates via a bus with a data storage medium 226 for passing information between components of the multimodal unlock model 102. For example, in some embodiments, the data storage medium 226 is non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, in some embodiments, the data storage medium 226 comprises or embodies an electronic storage device (e.g., a computer-readable storage medium). In some embodiments, the data storage medium 226 is configured to store information, data, content, applications, instructions, etc. for enabling the multimodal unlock model 102 to perform various functions in accordance with example embodiments of the present disclosure.

[0071] The processor 222 may be embodied in a number of different ways. For example, in some example embodiments, the processor 222 includes one or more processing devices configured to execute independently. Additionally or alternatively, in some embodiments, the processor 222 includes one or more processors configured in series via a bus to enable independent execution of instructions, pipelining, and / or multithreading. The use of the terms "processor" and "processing circuitry" should be understood to include single-core processors, multi-core processors, multiple processors within the multimodal unlock model 102, and / or one or more remote or "cloud" processors external to the multimodal unlock model 102.

[0072] In example embodiments, the processor 222 is configured to execute instructions stored in the data storage medium 226 or otherwise accessible to the processor. Alternatively or additionally, in some embodiments, the processor 222 is configured to execute hard-coded functions. Thus, whether configured by hardware or software methods, or by a combination thereof, the processor 222 represents an entity (e.g., physically embodied in circuitry) capable of performing operations in accordance with embodiments of the present disclosure when correspondingly configured. Alternatively or additionally, as another example in some example embodiments, when the processor 222 is embodied as an executor of software instructions, the instructions specifically configure the processor 222 to perform the algorithms embodied in the specific operations described herein when such instructions are executed. In some embodiments, the processor 222 includes a CPU, a microprocessor, and / or the like encoded by computer instructions stored, for example, via the non-transitory data storage medium 226, or is embodied by a CPU, a microprocessor, and / or the like encoded by computer instructions stored, for example, via the non-transitory data storage medium 226.

[0073] In some embodiments, the multimodal unlocking model 102 includes input / output circuitry 224 that provides output to a user and, in some embodiments, is used to receive indications of user input. In some embodiments, the input / output circuitry 224 communicates with the processor 222 to provide such functionality. The input / output circuitry 224 may include one or more user interfaces and, in some embodiments, includes a display that includes interfaces presented as an electronic interface, a web user interface, an application user interface, a user device, a backend system, etc. In some embodiments, the input / output circuitry 224 also includes a keyboard, a mouse, a joystick, a touch screen, a touch area, soft keys, a microphone, a speaker, or other input / output mechanisms. The processor 222 and / or the input / output circuitry 224 including the processor may be configured to control one or more functions of one or more user interface elements by means of computer program instructions (such as software and / or firmware) stored on a memory accessible to the processor (such as the data storage medium 226, etc.). In some embodiments, the input / output circuitry 224 includes or utilizes a user-facing application to provide input / output functionality to a client device and / or other displays associated with the user. In some embodiments, the input / output circuitry 224 includes hardware, software, firmware, and / or combinations thereof that facilitate simultaneously displaying specific data via multiple different devices.

[0074] In some embodiments, the multimodal unlocking model 102 includes a communication circuitry 228. The communication circuitry 228 includes any component configured to receive data from and / or transmit data to a network and / or any other device, circuitry, or module that communicates with the multimodal unlocking model 102, such as a device or circuitry embodied in hardware or a combination of hardware and software. In this regard, in some embodiments, the communication circuitry 228 includes, for example, a network interface for enabling communication with a wired or wireless communication network. Additionally or alternatively, in some embodiments, the communication circuitry 228 includes one or more network interface cards, antennas, buses, switches, routers, modems, and supporting hardware, firmware, and / or software, or any other device suitable for implementing communication via one or more communication networks. Additionally or alternatively, the communication circuitry 228 includes circuitry for interacting with the antennas and / or other hardware or software to transmit signals via the antennas or process signals received via the antennas. In some embodiments, the communication circuitry 228 enables the transmission of data to and / or the receipt of data from client devices, capture devices, and / or other external computing devices that communicate with the multimodal unlocking model 102.

[0075] In some embodiments, the multimodal unlocking model 102 includes a head-mounted sensor interface circuitry 221. The head-mounted sensor interface circuitry 221 includes components configured to receive data from and / or transmit data to one or more head-mounted sensors, such as a device or circuitry embodied in hardware or a combination of hardware and software. In this regard, in some embodiments, the head-mounted sensor interface circuitry 221 includes, for example, a communication interface for enabling communication with one or more head-mounted sensors according to a wired or wireless communication protocol. The head-mounted sensor interface circuitry 221 may also include components for receiving analog or digital electrical signals and converting the electrical signals required for use throughout the multimodal unlocking model 102. For example, in some embodiments, the head-mounted sensor interface circuitry 221 may receive an analog electrical signal representing a physical characteristic observed by one or more head-mounted sensors and convert the electrical signal into a digital representation of the physical characteristic.

[0076] Additionally or alternatively, in some embodiments, two or more of the sets of circuitry 221–228 are combinable. Alternatively or additionally, in some embodiments, one or more of the sets of circuitry perform some or all of the functions described in association with another component. For example, in some embodiments, two or more of the sets of circuitry 221–228 are combined into a single module, which is embodied in hardware, software, firmware, and / or combinations thereof. Similarly, in some embodiments, one or more of the sets of circuitry are combined with processor 222 such that processor 222 performs one or more of the operations described above for each of these sets of circuitry 226–228.

[0077] Example System

[0078] Now referring Figure 3 , a head-mounted display 330 is provided that includes an example embodiment of a head-mounted multimodal unlocking device 300. As Figure 3 depicted, the head-mounted display 330 includes a frame 338 and lenses 331 configured to display a display interface 332 on one or more of the lenses of lenses 331. As also depicted in Figure 3 , the head-mounted multimodal unlocking device is incorporated with the head-mounted display 330. The head-mounted multimodal unlocking device 300 includes a multimodal unlocking model 302 and a plurality of head-mounted sensors mounted or otherwise disposed within or on the frame 338 of the head-mounted display 330. Figure 3 The head-mounted sensors depicted in Figure 3 include a charge change sensor 333, an accelerometer 334, a gyroscope 335, and a magnetometer 336. As also depicted in

[0079] As Figure 3 depicted, the head-mounted display 330 is any electronic device configured to present a display interface 332 within the user's field of view and worn on the user's head. In the depicted embodiment, the head-mounted display 330 includes a frame 338 that includes two temple arms, hinges, and a front frame with edges that includes two lenses 331 designed to resemble a pair of glasses. The display interface 332 is presented on one or more of the lenses of lenses 331 such that the user can view the display interface 332 while viewing the surrounding environment. Although primarily described as an extended reality display in the depicted embodiment, the head-mounted display 330 can include a virtual reality headset, an augmented reality headset, an extended reality headset, a mixed reality headset, smart glasses, a heads-up display, or other similar head-mounted visual display devices.

[0080] As also shown in Figure 3 depicted in, the head-mounted multi-modal unlocking device 300 of the head-mounted display 330 includes a plurality of sensors (e.g., a charge change sensor 333, an accelerometer 334, a gyroscope 335, and a magnetometer 336) and a multi-modal unlocking model 302. Although mainly described as being connected to the frame, the plurality of sensors may be encapsulated within the frame 338, attached to one of the lenses 331, incorporated into the structure of the frame 338, or otherwise incorporated into the design of the head-mounted display 330.

[0081] As Figure 3 depicted, the charge change sensor 333 is positioned on the left temple of the frame 338 of the head-mounted display 330. As positioned, the charge change sensor 333 can detect the directional contact of the user on the surface of the left temple of the frame 338. Although depicted in the left temple of the frame 338, the charge change sensor 333 can be positioned at any location above or within the frame 338 of the head-mounted display 330 such that the directional contact, the face posture, or other postures can be detected by the charge change sensor 333.

[0082] Now referring to Figure 4 , a front view of a head-mounted display 430 according to an example embodiment of the present disclosure is provided. As Figure 4 depicted, the head-mounted display 430 includes a frame 438 and two lenses 431 and is positioned on the head of a user 440. In addition, Figure 4 the head-mounted display 430 of

[0083] As Figure 4 depicted, the example head-mounted display 430 includes a proximity sensor 442 incorporated into the frame 438 of the head-mounted display 430 such that the proximity sensor 442 is positioned within the proximity of the frame 438 of the head-mounted display 430 to sense an object and / or movement near the right eye of the user 440. Although depicted in the frame 438 near the right eye of the user 440, the proximity sensor 442 can be positioned at any location above or within the frame 438 of the head-mounted display 430 such that the proximity or movement of an object (such as a hand) can be detected at another surface of the frame 438.

[0084] As also shown in Figure 4As depicted in, example head - mounted display 430 includes an ambient light sensor 444. The ambient light sensor 444 is incorporated into the frame 438 of the head - mounted display 430 such that the ambient light sensor 444 is positioned near the left eye of the user 440 to sense the ambient light received at the frame 438 of the head - mounted display 430. Although depicted in the frame 438 near the left eye of the user 440, the ambient light sensor 444 can be positioned anywhere on or within the frame 438 of the head - mounted display 430 such that the ambient light can be detected at another surface of the frame 438.

[0085] Now referring to Figure 5 , a block diagram of an example embodiment of a head - mounted multi - modal unlocking device 500 is provided. As Figure 5 depicted, example head - mounted multi - modal unlocking device 500 includes a multi - modal unlocking model 502 communicatively connected to a plurality of sensors. The plurality of sensors includes a charge - change sensor 533, a proximity sensor 542, an ambient light sensor 544, and an inertial measurement unit (IMU) 550. The inertial measurement unit (IMU) includes an accelerometer 534, a gyroscope 535, and a magnetometer 536.

[0086] As Figure 5 depicted, example head - mounted multi - modal unlocking device 500 includes a charge - change sensor 533. The charge - change sensor 533 is any electrical, mechanical, and / or electromechanical device configured to measure the static charge of a nearby surface and transmit an electrical signal corresponding to the surface static charge generated due to the triboelectric effect. When placed on a head - mounted display, the charge - change sensor 533 can be used to measure various postures of the user, including face postures, eye postures, and hand postures.

[0087] For example, the charge - change sensor 533 can be placed near the face of a user of a head - mounted display. As the shape of the surface (e.g., the user's face) changes, the static charge measured by the charge - change sensor 533 also changes. The charge - change sensor 533 placed near the user's face can be used to detect facial expressions, such as raising eyebrows, squinting, smiling, and frowning, based on the shape change of the nearby surface.

[0088] In addition, different surfaces of the face exhibit different charges. For example, the surface of the eyeball can exhibit a different charge from the surface of the eyelid. Thus, eye postures such as squinting and blinking can be detected by the charge - change sensor 533 placed near the user's eyes.

[0089] In addition, the charge change sensor 533 can be configured to detect movement across or near the surface of the charge change sensor 533. For example, a charge change sensor 533 placed on the surface of a head-mounted display can detect a directed contact by a user across the surface of the head-mounted display where the charge change sensor 533 is placed. The directed contact can include a finger sliding forward or backward along the side or temple of the head-mounted display; a finger sliding left or right across the front or bridge of the head-mounted display; or other similar gestures.

[0090] As also depicted in Figure 5 the example head-mounted multimodal unlocking device 500 includes an ambient light sensor 544. The ambient light sensor 544 is any device, sensor, photodiode, semiconductor device, or other structure that generates a current corresponding to the intensity of light received at the ambient light sensor 544. In some embodiments, the ambient light sensor 544 can be a photosensitive semiconductor diode that generates a current proportional to the intensity of light received at the ambient light sensor 544. When placed on a head-mounted display, the ambient light sensor 544 can be used to measure various gestures of the user, including hand gestures.

[0091] For example, the ambient light sensor 544 can be placed on the surface of the head-mounted display facing away from the user, such as the front surface, bridge, or acetate of the head-mounted display. In addition to detecting ambient light for functional purposes (such as adjusting the brightness of the display interface), the ambient light sensor 544 can also detect hand gestures near the ambient light sensor 544. For example, a user can cover the ambient light sensor 544 by placing a hand on the surface where the ambient light sensor 544 is provided. In addition, the ambient light sensor 544 can be configured to identify the orientation of the hand based on the amount of light received at the ambient light sensor 544. A fist near the ambient light sensor 544 can result in an electrical output from the ambient light sensor 544 that is different from that of an open hand. In addition, an open hand with outstretched fingers can result in an electrical output from the ambient light sensor 544 that is different from that of an open hand with closed fingers.

[0092] As also depicted in Figure 5 the example head-mounted multimodal unlocking device 500 includes a proximity sensor 542. The proximity sensor 542 is any electrical, mechanical, and / or electromechanical device configured to output an electrical signal representing the proximity of one or more objects to the proximity sensor 542. In some embodiments, the proximity of an object can be measured by generating pulsed or continuous-wave ranging light radiation, receiving the pulsed or continuous-wave ranging light radiation reflected from the surface of the object, and using the time of flight of the pulsed or continuous-wave ranging light radiation to determine distances, movements, and / or other similar characteristics related to the position of an object near the proximity sensor 542.

[0093] In some embodiments, the proximity sensor 542 may include an array of sensing elements, such as, for example, a multi-zone and / or multi-pixel time-of-flight sensor. A multi-zone and / or multi-pixel time-of-flight sensor including a plurality of sensing elements may enable the detection of an attitude relative to the proximity sensor 542. For example, the proximity sensor 542 may be configured to detect a sliding motion of a hand in front of the proximity sensor 542 (e.g., a sliding motion from right to left, a sliding motion from top to bottom, etc.). When placed on the surface of a head-mounted display, the proximity sensor 542 may be used to measure various postures of a user, including hand postures. For example, a user may cover the proximity sensor 542 by placing a hand on the surface provided with the proximity sensor 542. The user may also wave and / or slide a hand near the proximity sensor 542.

[0094] As also shown in Figure 5 the example head-mounted multimodal unlocking device 500 includes an accelerometer 534, a gyroscope 535, and a magnetometer 536 in the IMU 550. Although shown together as including the IMU 550, the head-mounted multimodal unlocking device 500 may include one or more accelerometers 534, gyroscopes 535, and / or magnetometers 536 deployed for separate data capture. For example, the accelerometer 534, the gyroscope 535, and / or the magnetometer 536 may be individually configured to detect certain motions of the head-mounted device or certain motions of a user wearing the head-mounted device, including certain head postures, head orientations, hand postures, and other similar movements.

[0095] As Figure 5 depicted, the accelerometer 534, the gyroscope 535, and the magnetometer 536 may together form the IMU 550. The IMU 550 is any device configured to determine the specific force, angular rate, and / or orientation of a head-mounted display. In some embodiments, the IMU 550 may be configured to perform the function of a head orientation sensor. A head orientation sensor is any head-mounted sensor configured to determine the orientation of the head of a user wearing a device including the head-mounted sensor. The head orientation sensor may be configured to determine certain head postures, head positions, the heading of the user, or other similar characteristics related to the orientation of the user's head. The IMU 550 and / or one or more of the unified components in the unified component may be individually configured to further determine the hand postures performed by a user of the head-mounted display.

[0096] The head orientation sensor, such as Figure 5The IMU 550 can be configured to determine the orientation of a user's head. By mounting the IMU 550 on or within a head-mounted display, the IMU 550 can determine the attitude related to the head orientation. The head orientation attitude can include face down, face up, face left, face right, or other similar head orientation postures.

[0097] The head orientation sensor or the IMU 550 can also be configured to determine the head attitude based on the movement of the head. For example, the IMU 550 placed on or within the head-mounted display can be configured to detect head shaking, nodding, and other movements of the head.

[0098] The head orientation sensor or the IMU 550 can also be configured to determine the heading based on the orientation of the head-mounted display. For example, the IMU 550 placed on or within the head-mounted display can be configured to determine the rotation of the head-mounted display relative to the previous orientation of the head-mounted display. For example, in some embodiments, the rotation determination can be represented by degrees relative to the initial orientation (e.g., 0 degrees, 45 degrees, -45 degrees, 90 degrees, -90 degrees, etc.).

[0099] The IMU 550 can also be configured to determine hand gestures. For example, the IMU 550 can be configured to determine certain physical interactions with the head-mounted display including the IMU 550, such as tapping, raising or lowering the head-mounted display, or similar interactions. For example, the IMU 550 can use the accelerometer 534, gyroscope 535, and / or magnetometer 536 together or separately to identify taps, double taps, etc. on the surface of the head-mounted display. The IMU 550 can also use the accelerometer 534, gyroscope 535, and / or magnetometer 536 together or separately to identify additional physical interactions, such as shaking the head-mounted display, raising the head-mounted display, lowering the head-mounted display, etc.

[0100] Now refer to Figure 6 , a series of example head orientations 600 are provided. As Figure 6 depicted, a user 640 wearing a head-mounted display 630 can perform various head postures by changing the orientation of the head. For example, the user 640 can look down 660 (e.g., head down), look up 662 (e.g., head up), the user can tilt their head to the left 664, the user can tilt their head to the right 666, etc. Although only four head orientations 600 are depicted in Figure 6 , any number of head orientations 600 can be identified through the head-mounted display 630 using the head-mounted multimodal unlocking device described herein.

[0101] Now refer to Figure 7, an example series of headings 700 of a user 740 wearing a head-mounted display 730 is provided. The heading relative to the initial orientation of the user 740 can be determined based on the heading. For example, the user can turn left, turn right, or turn in a circle. Based on the initial heading, one or more head-mounted sensors can determine subsequent relative headings. As Figure 7 depicted, the user 740 wearing the head-mounted display 730 can change the heading of their head relative to the initial heading to perform various head postures. For example, the user 740 can turn their head and the attached head-mounted display 730 to a 45-degree position 770 relative to the initial orientation, turn their head to a -45-degree position 772 relative to the initial orientation, turn to a 90-degree position 774 relative to the initial orientation, or turn to a -90-degree position 776 relative to the initial orientation. Although only four example headings 700 are depicted in Figure 7 , any number of headings 700 can be identified via the head-mounted display 730 using the head-mounted multimodal unlocking device as described herein.

[0102] In addition to Figure 7 the head orientation postures depicted in (e.g., the head orientation 600 depicted in Figure 6 ) and the heading postures (e.g., the headings 700 in Figure 7 ), various head movement postures, such as shaking the head, nodding, rotating, turning in a circle, etc., can also be detected by one or more head-mounted sensors of the head-mounted multimodal unlocking device.

[0103] Now referring to Figure 8 , an example series of face postures 800 that can be detected by one or more head-mounted sensors of the head-mounted multimodal unlocking device is provided. As Figure 8 depicted, the user 840 wearing the head-mounted display 830 can perform face postures, such as eye postures. The head-mounted multimodal unlocking device can be configured to determine one or more eye states of each eye of the user 840. The eye states can include eyes open 883, eyes closed 881, squinting, or other similar eye postures. Based on the detected eye states, the user 840 can perform one or more eye postures, e.g., eyes open 880, right eye blink 882, left eye blink 884, and eyes closed 886 as depicted in Figure 8 . Although only four example face postures 800 are depicted in Figure 8 , any number of face postures 800 can be identified via the head-mounted display 830 using the head-mounted multimodal unlocking device as described herein.

[0104] Now referring to Figure 9, a series of example hand gestures 900 that can be detected by one or more head-mounted sensors of a head-mounted multimodal unlocking device are provided. The head-mounted multimodal unlocking device can be configured to detect one or more physical interactions with a head-mounted display 930. For example, the head-mounted multimodal unlocking device can be configured to determine a directional contact of a user's hand or finger 996 along the surface of the head-mounted display 930. The directional contact can include a backward sliding motion 990 along the surface of the head-mounted display 930, a forward sliding motion 992 along the surface of the head-mounted display 930, a vertical sliding up or down along the surface of the head-mounted display 930, 993, or other similar motions. Other physical interactions with the head-mounted display can include a tap 994 on the surface of the head-mounted display 930 as depicted in Figure 9 Although only four example hand gestures 900 are depicted in Figure 9 , any number of hand gestures 900 can be identified via the head-mounted display 930 using the head-mounted multimodal unlocking device described herein.

[0105] Now referring to Figure 10 , a series of additional example hand gestures 1000 that can be detected by one or more head-mounted sensors of a head-mounted multimodal unlocking device are provided. As depicted in Figure 10 , the head-mounted display 1030 includes a proximity sensor 1042 positioned on the frame of the head-mounted display 1030 near the user 1040's right eye, and an ambient light sensor 1044 positioned on the frame of the head-mounted display 1030 near the user 1040's left eye. The user 1040 can position their hand above or near the head-mounted sensors (e.g., proximity sensor 1042, ambient light sensor 1044) to perform hand gestures (and / or physically interact with the head-mounted display 1030). The hand gestures can include a hands-free gesture as depicted in hand gesture 1002, placing the hand 1096 near the right eye as depicted in hand gesture 1004, placing the hand 1096 near the left eye as depicted in hand gesture 1008, or placing one hand 1096 near the right eye and one hand 1096 near the left eye as depicted in hand gesture 1008. For example, for the purpose of determining a passcode gesture delimiter, such an interaction can be detected as a physical interaction with the head-mounted display 1030. Although only four example additional hand gestures 1000 are depicted in Figure 10 , any number of hand gestures 1000 can be identified via the head-mounted display 1030 using the head-mounted multimodal unlocking device as described herein. For example, as described herein, the head-mounted multimodal unlocking device can also determine the orientation of the hand (e.g., a fist, an open palm, outstretched fingers, etc.).

[0106] Now referring to Figure 11, an example gesture sequence password 1102 is provided. The example gesture sequence password 1102 includes a series of example password gestures 1102a–1102d. As Figure 11 depicted, the example gesture sequence password 1102 includes a series of four password gestures 1102a–1102d, each password gesture being defined by physical interaction delimiters 1106a–1106c. Additionally, a start password gesture 1104 is depicted as indicating the start of the gesture sequence password 1102.

[0107] As Figure 11 depicted, a head-mounted display can be configured with a head-mounted multimodal unlocking device. The head-mounted multimodal unlocking device includes a multimodal unlocking model and a plurality of head-mounted sensors to detect the gesture sequence password 1102. The gesture sequence password 1102 is any series of password gestures 1102a–1102d that can be detected by the multimodal unlocking model and are defined by the physical interaction delimiters 1106a–1106c. The password gestures 1102a–1102d are used to lock one or more locking features of the head-mounted display. During the registration of the gesture sequence password 1102, a user can perform the series of password gestures 1102a–1102d defined by the physical interaction delimiters 1106a–1106c to be stored as the saved gesture sequence password 1102. During the operation of the head-mounted display, a user of the head-mounted display can perform the series of password gestures 1102a–1102d defined by the physical interaction delimiters 1106a–1106c to attempt to unlock one or more locking features of the head-mounted display. In the case where the series of password gestures 1102a–1102d matches the saved gesture sequence password 1102, the locking features are unlocked. However, in the case where the gesture sequence password 1102 does not match the saved gesture sequence password 1102, the locking features of the head-mounted display remain locked. Although Figure 11 is depicted as including four password gestures 1102a–1102d, the gesture sequence password 1102 can be configured to include any number of password gestures (e.g., 4, 8, 16, etc.).

[0108] As also depicted in Figure 11 , the gesture sequence password 1102 includes a plurality of password gestures 1102a–1102d. The password gestures 1102a–1102d are any gestures performed by a user of the head-mounted display that are recognized by the multimodal unlocking model. The password gestures can include one or more head orientation gestures, one or more hand gestures, one or more face gestures, or any combination thereof. Figure 11The first cryptographic gesture 1102a in the depicted gesture sequence password 1102 includes a head orientation gesture in which the user looks down. The second cryptographic gesture 1102b in the depicted gesture sequence password 1102 includes an eye gesture in which the user blinks the left eye. The third cryptographic gesture 1102c in the depicted gesture sequence password 1102 includes a first hand gesture in which the user slides a finger forward on the temple of the head-mounted display. The fourth cryptographic gesture 1102d in the depicted gesture sequence password 1102 includes a second hand gesture in which the user covers a portion of the head-mounted display associated with one or more head-mounted sensors.

[0109] As also depicted in Figure 11 The gesture sequence password 1102 includes physical interaction delimiters 1106a–1106c, which indicate the division between respective cryptographic gestures 1102a–1102d during the execution of the gesture sequence password 1102. The physical interaction delimiter is any interaction of the user of the head-mounted display with the head-mounted display. The physical interaction can include hand gestures such as one or more taps on the head-mounted display, covering a portion of the head-mounted display, sliding a finger in a directed contact along the surface of the head-mounted display, and the like. Figure 11 The physical interaction delimiters 1106a–1106 depicted in

[0110] In some embodiments, the multimodal unlocking model of the head-mounted display can also be configured to recognize one or more deletion gestures. The deletion gesture can be performed by the user to indicate the removal of a previous cryptographic gesture 1102a–1102d from the attempted gesture sequence password 1102. For example, the deletion gesture can be a double tap on the head-mounted display, sliding a finger backward on the frame of the head-mounted display, shaking the head, or other similar gestures. Based on receiving the deletion gesture, the multimodal unlocking model can remove the previously received gesture that was a potential cryptographic gesture from the attempted gesture sequence password.

[0111] As also depicted in Figure 11 The gesture sequence password 1102 includes a starting cryptographic gesture 1104. The starting cryptographic gesture 1104 is any gesture performed by the user to indicate the start of a new gesture sequence password 1102. For example, the starting cryptographic gesture can be a double tap on the head-mounted display, sliding a finger forward on the frame of the head-mounted display, covering a predetermined portion of the head-mounted display, or other similar gestures. As Figure 11As depicted, the start gesture 1104 is a double tap on the surface of the head-mounted display. The multimodal unlocking model can be configured to identify subsequent gestures as potential password gestures of the attempted gesture sequence password 1102 when the start gesture 1104 is recognized.

[0112] Now refer to Figure 12 , an example gesture sequence password 1202 is provided. The example gesture sequence password 1202 includes four password gestures 1202a–1202d, and each password gesture represents a plurality of gesture states 1212a–1212n. As also depicted in Figure 12 , the generation of the sequence password 1202 is initiated by a start gesture 1204, and each password gesture 1202a–1202d is delimited by physical interaction delimiters 1206a–1206c.

[0113] As Figure 12 depicted, each password gesture 1202a–1202d includes a plurality of gesture states 1212a–1212n. In the embodiment depicted in Figure 12 , the password gestures 1202a–1202d may include a plurality of gestures that are performed simultaneously and delimited by physical interaction delimiters 1206a–1206c. The gesture states 1212a–1212n are any values that represent the state of a set of gestures or a specific gesture. In a specific example, a gesture may include two states. For example, the gesture state of the user's right eye may be open or closed. When the user's right eye is open, the gesture state for a specific gesture may be set to 1, and when the user's right eye is closed, the gesture state may be set to 0. In another example, the eye gesture may include four states. For example, 0 indicates that both of the user's eyes are closed; 1 indicates that the user's right eye is open and the user's left eye is closed; 2 indicates that the user's left eye is open and the user's right eye is closed; and 3 indicates that both of the user's eyes are open. Thus, the password gestures 1202a–1202d may store the states 1212a–1212n for n gestures, where n is the number of simultaneous gestures recorded in the password gestures 1202a–1202d.

[0114] In a specific illustrative example, the password gesture 1202a–1202d may include three gesture states 1212a–1212c. State 1212a may represent the state of the user's eyes (e.g., 0 – both eyes closed, 1 – right eye open, left eye closed, 2 – right eye closed, left eye open, 3 – both eyes open). State 1212b may represent the rotation of the user's head (e.g., 0 – 45 degrees, 1 – -45 degrees, 2 – 90 degrees, 3 – -90 degrees). State 1212c may represent the placement of the user's hand (e.g., 0 – no hand covering the proximity sensor or ambient light sensor, 1 – one hand covering the proximity sensor, 2 – one hand covering the ambient light sensor, 3 – one hand covering the proximity sensor and one hand covering the ambient light sensor). Thus, the states for each password gesture 1202a–1202d can be recorded. For example, during password gesture 1202a, the multiple gesture states at 1212a–1212c may be 111, during password gesture 1202b, the multiple states 1212a–1212c may be 212, during password gesture 1202c, the multiple states 1212a–1212c may be 132, and during password gesture 1202d, the multiple states 1212a–1212c may be 333.

[0115] To correctly execute the gesture sequence password 1202, the gesture sequence password 1202 includes the states: 1202a–111; 1202b–212; 1202c–132; 1202d–333; the user may perform the following password gestures. First, the user may perform a start password gesture 1204 (e.g., double-tap the head-mounted display). Next, the user may perform password gesture 1202a (111) by simultaneously performing an eye gesture of right eye open and left eye closed, turning the head 45 degrees, and covering the proximity sensor; followed by a physical interaction delimiter. Next, the user may perform password gesture 1202b (212) by simultaneously performing an eye gesture of right eye closed and left eye open, turning the head 45 degrees, and covering the ambient light sensor; then a physical interaction delimiter (e.g., both eyes blink). Next, the user may perform password gesture 1202c (132) by simultaneously performing an eye gesture of right eye open and left eye closed, turning the head -90 degrees, and covering the ambient light sensor; then a physical interaction delimiter. Finally, the user may perform password gesture 1202d (333) by simultaneously performing an eye gesture of both eyes open, turning the head -90 degrees, and covering both the proximity sensor and the ambient light sensor.

[0116] By simultaneously enabling password gestures representing multiple gesture states, the strength of the gesture sequence password 1202 can be significantly increased. In addition, especially in cases where certain gestures (e.g., eye gestures) are hidden from view, it may be difficult for onlookers to observe and imitate the simultaneous performance of multiple gestures.

[0117] Example method

[0118] Now refer to Figure 13 and an example process 1300 is provided for unlocking one or more locking features of a head-mounted display using multiple head-mounted sensors. At block 1302, a multimodal unlocking model (e.g., multimodal unlocking model 102, 502) receives one or more electrical signals that represent user interaction (e.g., pose) with at least one of a plurality of sensors (e.g., head-mounted sensor 104; charge change sensors 333, 533; accelerometers 334, 534; gyroscopes 335, 535; magnetometers 336, 536; proximity sensors 442, 542; ambient light sensors 444, 544; inertial measurement unit 550) mounted on a head-mounted display (e.g., head-mounted displays 330, 430, 630, 730, 830, 930, 1030). As depicted herein, the multimodal unlocking model can be electrically coupled to the plurality of head-mounted sensors. In some embodiments, the plurality of head-mounted sensors can be present on the head-mounted display to provide existing functionality. However, the plurality of head-mounted sensors can also sense user interaction with the head-mounted display and the surrounding environment. For example, the plurality of head-mounted sensors can sense poses performed by a user of the head-mounted display.

[0119] At block 1304, the multimodal unlocking model detects a pose sequence password (e.g., pose sequence passwords 1102, 1202), the pose sequence password including one or more password poses (e.g., 1102a–1102d; 1202a–1202d) from a series of password poses delimited by physical interaction with the head-mounted display (e.g., physical interaction delimiters 1106a–1106c; 1206a–1206c). As described herein, the multimodal unlocking model can be configured to detect a wide range of poses based on the electrical signals received from the plurality of head-mounted sensors, the wide range of poses including head orientation poses, hand poses, face poses, etc. Each password pose performed by the user can be delimited by a physical interaction such as tapping on the head-mounted display, directed contact along the surface of the head-mounted display, covering a portion of the head-mounted display, or a similar physical interaction.

[0120] At block 1306, the multimodal unlocking model unlocks the locking features of the head-mounted display based on the gesture sequence password. The multimodal unlocking model can compare the series of gestures performed with the password gestures saved as part of the gesture sequence password saved during the registration process. In the case where the series of gestures matches the gesture sequence password, one or more locking features of the head-mounted display can be unlocked. The locking features can include certain notifications displayed on the display interface of the head-mounted display, available applications on the head-mounted display, and the like.

[0121] Using the multimodal unlocking model as an unlocking mechanism for the head-mounted display can greatly improve the security and usability of the head-mounted display. For example, using various gestures and even gesture combinations in the gesture sequence password can provide a large number of possible combinations in the gesture sequence password, making the gesture sequence password difficult to crack. In addition, many of the gestures in the password sequence can be hidden or not obvious, making it difficult for bystanders to imitate the series of gestures. In addition, the gesture-based password can be easily shared with trusted individuals. In addition, using the existing sensors on the head-mounted display to identify the password gestures enables the size and weight of the head-mounted display to be reduced while increasing the battery life.

[0122] Although this detailed description has set forth some embodiments of the present invention, the appended claims cover other embodiments of the present invention, which are different from the described embodiments according to various modifications and improvements. For example, those skilled in the art will recognize that such principles can be applied to any electronic device that may benefit from a low-power unlocking mechanism that relies on a multifunctional sensing device for gesture recognition rather than standard input devices such as keyboards, touchscreens, and biometric authentication devices. For example, extended reality headsets, virtual reality headsets, augmented reality headsets, mixed reality headsets, smart glasses, smart watches, and other wearable electronic devices.

[0123] Within the appended claims, unless the specific terms "means for..." or "step for..." are used in a given claim, the claim is not intended to be interpreted under paragraph 6 of 35 U.S.C. 112.

[0124] The use of broad terms such as "comprising," "including," and "having" should be understood to provide support for narrow terms such as "consisting of," "consisting essentially of," and "substantially consisting of." The use of terms such as "optionally," "can," "may," "possibly," etc. with respect to any element of an embodiment means that the element is not required, or alternatively, the element is required, and both alternatives are within the scope of the (multiple) embodiments. In addition, the reference to examples is for illustrative purposes only and is not intended to be exclusive.

Claims

1. A device, comprising: A head-mounted display; A plurality of sensors mounted on the head-mounted display and configured to detect the posture of a user of the head-mounted display; And A multimodal unlocking model, the multimodal unlocking model including one or more processors and one or more storage devices, the one or more storage devices storing instructions that can be operated, when executed by the one or more processors, to cause the one or more processors to: Detect a posture sequence password, the posture sequence password including one or more password postures in a series of password postures defined by physical interaction with the head-mounted display; and Unlock the locking feature of the head-mounted display based on the posture sequence password.

2. The device according to claim 1, wherein the multimodal unlocking model is configured to detect the head orientation of the user as at least one password posture among the one or more password postures including the posture sequence password in the series of password postures.

3. The device according to claim 1, wherein the multimodal unlocking model is configured to detect the eye state of each eye of the user as at least one password posture among the one or more password postures including the posture sequence password in the series of password postures.

4. The device according to claim 3, wherein the eye state includes open or closed.

5. The device according to claim 1, wherein the multimodal unlocking model is configured to detect the heading of the user as at least one password posture among the one or more password postures including the posture sequence password in the series of password postures.

6. The device according to claim 5, wherein the heading of the user is associated with a relative heading with respect to an initial heading.

7. The device according to claim 1, wherein the multimodal unlocking model is configured to detect the hand posture of the user relative to the head-mounted display as at least one password posture among the one or more password postures including the posture sequence password in the series of password postures.

8. The device according to claim 7, wherein the hand posture includes one or more taps on the head-mounted display.

9. The device according to claim 7, wherein the hand posture includes detecting the directional contact of the user on the surface of the head-mounted display.

10. The device according to claim 7, wherein the hand posture includes positioning the hand near a predetermined position on the head-mounted display.

11. The device according to claim 1, wherein the multimodal unlocking model is configured to detect the face posture of the user of the head-mounted display as at least one password posture among the one or more password postures including the posture sequence password in the series of password postures.

12. The device according to claim 1, wherein the physical interaction includes physical contact between the user's hand and the head-mounted display.

13. The device according to claim 12, wherein the physical interaction includes one or more taps on the head-mounted display, or directional contact of the user on the surface of the head-mounted display.

14. The device according to claim 1, wherein the plurality of sensors includes at least a head orientation sensor, a charge change sensor, a proximity sensor, or an ambient light sensor.

15. The device according to claim 14, wherein the head orientation sensor includes an inertial measurement unit.

16. The device according to claim 1, the multimodal unlocking model is further configured to: detect a deletion gesture; and delete at least one of the one or more password gestures in the series of password gestures based on the deletion gesture.

17. The device according to claim 1, wherein at least one of the one or more password gestures in the series of password gestures includes a plurality of gesture states.

18. The device according to claim 1, wherein the multimodal unlocking model is configured to detect a start password gesture, the start password gesture indicating the start of the gesture sequence password.

19. A computer-implemented method, comprising: receiving, at a multimodal unlocking model, one or more electrical signals, the one or more electrical signals representing user interaction with at least one of a plurality of sensors mounted on a head-mounted display; detecting a gesture sequence password, the gesture sequence password including one or more password gestures in a series of password gestures defined by physical interaction with the head-mounted display; and unlocking the locking feature of the head-mounted display based on the gesture sequence password.

20. A computer program product, comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions including executable portions configured to: receive, at a multimodal unlocking model, one or more electrical signals, the one or more electrical signals representing user interaction with at least one of a plurality of sensors mounted on a head-mounted display; detect a gesture sequence password, the gesture sequence password including one or more password gestures in a series of password gestures defined by physical interaction with the head-mounted display; and unlock the locking feature of the head-mounted display based on the gesture sequence password. ​

Images