Multi-factor user authentication method and system based on eye tracking and gesture interaction in a metaverse environment

Abstract

A kind of multi-factor user authentication method and system based on eye tracking and gesture interaction under meta-universe environment, its method includes: (1) explicit authentication method and data acquisition;(2) pre-processing eye tracking data and gesture data;(3) eye tracking and gesture feature extraction;(4) model training and implicit authentication.The present application also provides a kind of multi-factor user authentication system based on eye tracking and gesture interaction under meta-universe environment, in turn include the following modules: (1) explicit authentication method module;(2) data preprocessing module;(3) feature extraction and model training module;(4) implicit authentication method module.The present application effectively solves the shoulder-surfing attack problem existing in the user authentication process under meta-universe environment, increases security.

Description

Technical Field

[0001] The present invention relates to the fields of human-computer interaction, metaverse, virtual reality, and more particularly to a multi-factor user authentication method and system based on eye tracking and gesture interaction in a metaverse environment. Background Art

[0002] The development of the metaverse has sparked growing interest in it. However, new interaction methods present new opportunities for attackers, making secure user authentication methods crucial in the metaverse. Traditional explicit authentication methods, such as PIN codes and graphical passwords, while convenient, are vulnerable to shoulder surfing attacks. This is especially true when users are immersed in VR environments, where they are often unable to observe their surroundings. Therefore, they cannot meet the security requirements of the metaverse. In recent years, implicit authentication methods based on biometrics have also been applied to various systems.

[0003] Most of the currently widely used implicit biometric authentication methods use physiological characteristics, such as fingerprint recognition and facial recognition. However, physiological characteristics are easy to obtain and forge. The largest recorded biometric information leak involved a database containing 27.8 million records, including fingerprint and facial recognition data. Compared with traditional passwords, if physiological characteristics are obtained and forged, users cannot use different physiological characteristics to reset them, thus posing a significant security risk. Behavioral-based authentication refers to authentication based on characteristics such as the user's typing style and reading behavior. Such characteristics are difficult to imitate and forge, resulting in the recent emergence of some applications that use behavioral characteristics for authentication. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a multi-factor user authentication method and system based on eye tracking and gesture interaction in a metaverse environment.

[0005] Users are authenticated through a multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment. At the same time, eye movement data and gesture data are collected in the background, and fusion features are used for implicit authentication. Users must pass both explicit and implicit authentication to be identified as legitimate users.

[0006] The present invention provides a multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment, which mainly includes the following steps:

[0007] (1) Explicit authentication and data collection through eye movement and gestures: The user wears a VR helmet with a built-in eye movement module for authentication, and the Leap Motion device is fixed in front of the VR helmet. Explicit authentication is completed through two interactive methods: eye movement tracking and gestures. At the same time, the background collects eye movement and gesture data through the eye movement module and Leap Motion.

[0008] (2) Preprocessing eye tracking data and gesture data; removing obvious noise data and classifying eye movement data into fixation and saccade data using an LSTM-based method;

[0009] (3) Extracting eye tracking and gesture features; extracting eye movement features and gesture features from the processed data;

[0010] (4) Use eye movement and gesture features to train the model and perform implicit authentication; use the biometric features extracted in step (3) to classify the random forest classifier to obtain a classification model to implicitly authenticate the user.

[0011] Wherein, step (1) specifically includes:

[0012] In the display authentication method, the user operates a cube dice marked with numbers 1 to 6 and enters a digital password according to the corresponding prompt object to authenticate the user; there are four prompt objects in the metaverse environment, each prompt object requires the user to enter a specific password; the cube dice is not only marked with numbers, but also with color information. The final input password requires not only the number to be correct, but also the color must meet the requirements of the prompt object; only when the number and color match correctly can the user pass the explicit authentication; the following steps are included: (11) the user wears the VR helmet and completes the five-point eye movement calibration; (12) the user enters the password through the cube dice The user can select a number by rolling the dice. During this process, the user can get different numbers by grabbing the dice, and the number facing up is the number selected by the user; (13) Six coloring modules are set up in the metaverse environment of explicit authentication, each module corresponds to a color, and the user changes the color of the dice by grabbing the dice and colliding them with the coloring module through this interaction; (14) The user selects the prompt object through eye movement interaction. When the gaze point is on the prompt object and the user uses the left hand to make an "OK" gesture, the password is entered; (15) The password entered by the user is verified. The user can pass the verification only when the number and color match correctly.

[0013] In step (1), the password space size for explicit authentication is (6x6) 4=1,679,616. During explicit authentication, the user's eye movement and gesture data are collected in the background through the eye movement module and Leap Motion. The raw eye movement data includes x-axis coordinates, y-axis coordinates, and timestamps, while the gesture data includes the Frame object recorded by Leap Motion and timestamps.

[0014] Wherein, step (4) specifically includes:

[0015] (41) The extracted gesture features are used to train a gesture random forest classifier. Multiple decision trees are constructed by randomly extracting multiple different subsets from the original training dataset. The final classification decision is then made based on the voting results of these decision trees. The classification decision function is as follows:

[0016]

[0017] Among them, I(g k (x)=Y) is the characteristic function, g k (x) is the classification model of a single decision tree, and Y is the target variable;

[0018] (42) The extracted eye movement features are used to train the eye movement random forest classifier, where the eye movement features are divided into fixation and saccade features. The two features are fused in the decision layer, and the fixation weight and saccade weight are 0.5 and 0.5 respectively;

[0019] (43) The eye movement features and gesture features are fused at the decision layer, and the classification results of the eye movement and gesture random forest classifiers are fused by weighted averaging, where the eye movement weight and gesture weight are 0.5 and 0.5 respectively; the classification results are used to determine whether the user has passed the implicit authentication;

[0020] (44) If a user passes both explicit and implicit authentication, he or she will be judged as a legitimate user.

[0021] The present invention also provides a multi-factor user authentication system based on eye tracking and gesture interaction in a metaverse environment, which includes the following modules in sequence:

[0022] (1) Explicit authentication method module, used for explicit authentication in the metaverse;

[0023] (2) Data preprocessing module, which preprocesses the collected data for subsequent model training;

[0024] (3) Feature extraction and model training module, which extracts eye movement features and gesture features from the processed data and uses biometric features to train a random forest classifier;

[0025] (4) Implicit authentication method module, which uses biometrics for implicit authentication in the metaverse.

[0026] The present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment of the present invention is implemented.

[0027] The present invention further provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the present invention is implemented.

[0028] The explicit authentication method of the present invention interacts through eye movements and gestures. Since the VR helmet blocks the user's eyes and the user's gestures are difficult to understand, it is difficult for an attacker to guess the password by observing the user's movements. This can effectively deal with shoulder surfing attacks. The present invention also includes an implicit authentication method, which uses the biometrics collected when the user performs explicit authentication to train a random forest classifier to authenticate the user in the background. Even if the attacker successfully guesses the password, he or she will be identified as an illegal user because the biometrics are different from the user, greatly increasing the security of the metaverse.

[0029] The advantage of the present invention is that by combining explicit authentication with implicit authentication, it effectively solves the shoulder surfing attack problem in metaverse user authentication, increases security, and uses eye tracking and gestures for authentication without destroying the user's immersion in the metaverse. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 It is a schematic flow chart of the method of the present invention.

[0032] Figure 2 It is the LSTM eye movement event classification model of the present invention.

[0033] Figure 3 Schematic diagram of the framework of the user classification method based on eye movement of the present invention.

[0034] Figure 4 Schematic diagram of the framework of the gesture-based user classification method of the present invention.

[0035] Figure 5 It is a schematic diagram of the basic logical structure of the system of the present invention. DETAILED DESCRIPTION

[0036] The following, in conjunction with the accompanying drawings and embodiments, provides a detailed and complete description of a multi-factor user authentication method and system based on eye tracking and gesture interaction in a metaverse environment. Obviously, the embodiments described are only a portion of the embodiments of the present invention, not all of them, and should not be construed as limiting the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort fall within the scope of protection of the present invention.

[0037] Example 1

[0038] See Figure 1 , an embodiment of the present invention provides a flowchart of a multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment, and specifically describes the steps therein:

[0039] (1) Explicit authentication and data collection through eye movements and gestures;

[0040] After completing eye movement calibration using the five-point calibration method, users authenticate by rolling a cube dice marked with numbers 1 to 6 and entering a numeric password according to the corresponding prompt object. The Metaverse environment contains four prompt objects, each requiring the user to enter a specific password. The cube dice are not only marked with numbers but also with color information. Users can change the color of the dice by interacting with the coloring module in the environment. The Metaverse environment for explicit authentication features six coloring modules, each corresponding to a specific color. Users use the cube dice to select a number. By grabbing the dice, users can obtain different numbers. After selecting a number, they collide the dice with a coloring module to change the color. Users select the prompt object through eye movement interaction. Password entry is completed when gaze is focused on the prompt object and the user makes an "OK" gesture with their left hand. The final password must not only be correct in number but also in color according to the prompt object's requirements. Only when both the number and color match correctly can the user pass authentication. The password space for explicit authentication is (6x6)4 = 1,679,616. During explicit authentication, the eye tracking module and Leap Motion collect the user's eye movement and gesture data in the background. The raw eye movement data includes x-axis coordinates, y-axis coordinates, and timestamps, while the gesture data includes the Frame object recorded by the Leap Motion, timestamps, and other data.

[0041] (2) Preprocessing eye tracking data and gesture data;

[0042] First, we remove some obvious noise, such as coordinates out of range, and then process the original eye movement data into the form of x-axis velocity + y-axis velocity. Since the speed of eye movement during fixation and saccade is quite different, global normalization can easily compress the slower fixation to close to 0, losing most of the eye movement information. Therefore, we use an LSTM-based eye movement event classification algorithm to classify the original eye movement data into fixation data and saccade data. At the same time, we can also remove some noise data such as blinking in the original data. Figure 2 The figure shows an LSTM eye movement event classification model, consisting of two LSTM hidden layers with 128 and 64 nodes, respectively. Reluctant Lu (ReLU) activation function was used, and dropout was added after each LSTM layer to prevent overfitting. Dropout values ​​were set to 0.2. During model training, the dataset was split into a test set (60%), a validation set (20%), and a test set (20%). Gesture data was obtained by calling members of each Frame object recorded by the Leap Motion. The Hand class retrieves data such as palm width and velocity for both hands, while the Finger class retrieves data such as the width and length of each finger on both hands. Due to the high accuracy of the Leap Motion device, physiological data such as finger width in the dataset can be used after removing any significant noise.

[0043] (3) Extract eye tracking and gesture features;

[0044] We extracted 16 eye movement features and 26 gesture features from the eye movement and gesture data, respectively. Eye movement features include total authentication duration, gaze duration, average gaze speed, maximum gaze speed, minimum gaze speed, median gaze speed, standard deviation of gaze speed, glance duration, average glance speed, maximum glance speed, minimum glance speed, median glance speed, standard deviation of glance speed, maximum pupil size, minimum pupil size, and average pupil size. Gesture features include total authentication duration, input interval duration, average left palm speed, average right palm speed, left hand five-finger width, right hand five-finger width, left hand five-finger length, right hand five-finger length, left hand palm width, and right hand palm width. These features encompass both behavioral and physiological characteristics.

[0045] (4) Use eye movement and gesture features to train the model and perform implicit verification;

[0046] The decision layer fusion is selected to fuse the eye movement features and gesture features. First, the eye movement classifier is trained using the eye movement features alone, and the gesture classifier is trained using the gesture features alone. Then, the classification results are fused by weighted averaging, where the eye movement weight and gesture weight are 0.5 and 0.5 respectively. Figure 3 、 Figure 4These are the frameworks for eye movement user classification and gesture user classification. For eye movement classification, gaze and saccade features are extracted and fused at the decision layer through weighted averaging, with gaze and saccade weights of 0.5 and 0.5, respectively. For gesture classification, physiological and behavioral features are fused at the feature layer. Both classifiers are random forest classifiers, which construct multiple decision trees by randomly extracting multiple subsets from the original training dataset. The final classification decision is then made based on the voting results of these decision trees. The classification decision function is as follows:

[0047]

[0048] Among them, I(g k (x)=Y) is the characteristic function, g k (x) is the classification model of a single decision tree, and Y is the target variable.

[0049] The classification results are used to determine whether the user has passed implicit authentication. If the user has passed both explicit and implicit authentication, he or she will be judged as a legitimate user.

[0050] Example 2

[0051] like Figure 5 Figure 2 shows the basic logical structure of a multi-factor user authentication system based on eye tracking and gesture interaction in a metaverse environment, as provided by an embodiment of the present invention. For ease of illustration, only the parts relevant to the embodiment of the present invention are shown. The functional modules / units in this system can be hardware modules / units or software modules / units, and mainly include:

[0052] (1) Explicit authentication method module, used for explicit authentication in the metaverse;

[0053] (2) Data preprocessing module, which preprocesses the collected data for subsequent model training;

[0054] (3) Feature extraction and model training module, which extracts eye movement features and gesture features from the processed data and uses biometric features to train a random forest classifier;

[0055] (4) Implicit authentication method module, which uses biometrics for implicit authentication in the metaverse.

[0056] Example 3

[0057] A computer-readable storage medium, characterized in that a program is stored thereon, which, when executed by a processor, implements the multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment described in Example 1.

[0058] Example 4

[0059] A computing device includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in Example 1 is implemented.

[0060] The various modules in the embodiments of the present invention may be integrated, deployed separately, or further divided into multiple sub-modules. The various modules may be distributed in the system of the embodiment as described in the embodiment, or may be located in one or more systems different from the embodiment of the present invention with corresponding changes.

[0061] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0062] The above disclosure is merely a specific embodiment of the present invention, but the present invention is not limited thereto. Those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Obviously, such modifications and variations should fall within the scope of protection claimed by the present invention.

Claims

1. A multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment, characterized in that: The following steps are involved: (1) Explicit authentication and data collection through eye movements and gestures; The user wears a VR headset with a built-in eye tracking module for authentication. The Leap Motion device is fixed in front of the VR headset. Explicit authentication is completed through eye tracking and gesture interaction. At the same time, the backend collects eye movement and gesture data through the eye tracking module and Leap Motion. (2) Preprocessing eye movement data and gesture data; Eliminate obvious noise data and classify eye movement data into fixations and saccades using an LSTM-based method; (3) Extract eye movement features and gesture features; Extract eye movement features and gesture features from the processed data; (4) Use eye movement and gesture features to train the model and perform implicit verification; The random forest classifier is used to classify the biometric features extracted in step (3) to obtain a classification model to implicitly authenticate the user.

2. The multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment according to claim 1 is characterized in that: Step (1) specifically includes: In the display authentication method, the user operates a cube dice marked with numbers 1 to 6 and enters a digital password according to the corresponding prompt object to authenticate the user; there are four prompt objects in the metaverse environment, each prompt object requires the user to enter a specific password; the cube dice is not only marked with numbers, but also with color information. The final input password requires not only the number to be correct, but also the color must meet the requirements of the prompt object; only when the number and color match correctly can the user pass the explicit authentication; the following steps are included: (11) the user wears the VR helmet and completes the five-point eye movement calibration; (12) the user enters the password through the cube dice The user can select a number by rolling the dice. During this process, the user can get different numbers by grabbing the dice, and the number facing up is the number selected by the user; (13) Six coloring modules are set up in the metaverse environment of explicit authentication, each module corresponds to a color. The user grabs the dice and collides with the coloring module to change the color of the dice through this interaction; (14) The user selects the prompt object through eye movement interaction. When the gaze point is on the prompt object and the user uses the left hand to make an "OK" gesture, the password is entered; (15) The password entered by the user is verified. The user can pass the verification only when the number and color match correctly.

3. The multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment according to claim 2, characterized in that: In step (1), the password space size for explicit authentication is (6x6) 4 =1,679,616; During the explicit authentication process, the user's eye movement data and gesture data will be collected in the background through the eye movement module and Leap Motion. The raw data of the eye movement data includes the x-axis coordinate, y-axis coordinate, and timestamp data, and the gesture data includes the Frame object recorded by Leap Motion and timestamp data.

4. The multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment according to claim 1 is characterized in that: Step (4) specifically includes: (41) The extracted gesture features are used to train a gesture random forest classifier. Multiple decision trees are constructed by randomly extracting multiple different subsets from the original training dataset. The final classification decision is then made based on the voting results of these decision trees. The classification decision function is as follows: Among them, I(g k (x)=Y) is the characteristic function, g k (x) is the classification model of a single decision tree, and Y is the target variable; (42) The extracted eye movement features are used to train the eye movement random forest classifier, where the eye movement features are divided into fixation and saccade features. The two features are fused in the decision layer, and the fixation weight and saccade weight are 0.5 and 0.5 respectively; (43) The eye movement features and gesture features are fused at the decision layer, and the classification results of the eye movement and gesture random forest classifiers are fused by weighted averaging, where the eye movement weight and gesture weight are 0.5 and 0.5 respectively; the classification results are used to determine whether the user has passed the implicit authentication; (44) If a user passes both explicit and implicit authentication, he or she will be judged as a legitimate user.

5. A multi-factor user authentication system based on eye tracking and gesture interaction in a metaverse environment, applying the multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment as claimed in claim 1, characterized in that include: Explicit authentication method module, used for explicit authentication in the metaverse; Data preprocessing module, which preprocesses the collected data for subsequent model training; Feature extraction and model training module, which extracts eye movement features and gesture features from the processed data and uses biometric features to train a random forest classifier; Implicit authentication method module, which uses biometrics for implicit authentication in the metaverse.

6. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, it implements the multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment as described in any one of claims 1 to 4.

7. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 4 is implemented.

Images