Multi-factor user authentication method and system based on eye movement tracking and gesture interaction in element universe environment

Abstract

The invention discloses a multi-factor user authentication method and system based on eye movement tracking and gesture interaction in a meta-universe environment. The method comprises the following steps: (1) an explicit authentication method and data acquisition; (2) preprocessing eye movement tracking data and gesture data; (3) eye movement tracking and gesture feature extraction; and (4) model training and implicit authentication. The invention further provides a multi-factor user authentication system based on eye movement tracking and gesture interaction in the meta-universe environment. The system sequentially comprises the following modules: (1) an explicit authentication method module; (2) a data preprocessing module; (3) a feature extraction and model training module; and (4) an implicit authentication method module. According to the method, the problem of shoulder peeping attack in the user authentication process in the meta-universe environment is effectively solved, and the security is improved.

Description

Technical Field

[0001] The present invention relates to the fields of human-computer interaction, metaverse, virtual reality, and more particularly to a multi-factor user authentication method and system based on eye tracking and gesture interaction in a metaverse environment. Background Art

[0002] The development of the Metaverse has led to an increasing interest in it. However, new ways of interaction provide new opportunities for attackers. Secure user authentication methods are essential in the Metaverse. Although traditional explicit authentication methods such as PIN codes and graphic passwords are relatively convenient, they are vulnerable to shoulder surfing attacks, especially when users are immersed in a VR environment and usually cannot observe the surrounding environment. Therefore, they cannot meet the security requirements of the Metaverse. In recent years, implicit authentication methods based on biometrics have also been applied to various systems.

[0003] Most of the currently widely used implicit authentication methods based on biometrics use physiological features, such as fingerprint recognition and facial recognition. However, physiological features are easy to obtain and forge. The largest biometric information leakage incident currently recorded involved a database containing 27.8 million records, which included fingerprint and facial recognition data. Compared with traditional passwords, if physiological features are obtained and forged, users have no way to reset them using different physiological features, so there are also greater security risks. Authentication based on behavioral features refers to authentication using features such as the user's typing style and reading behavior. This feature is difficult to imitate and forge, so some applications that use behavioral features for authentication have recently appeared. Summary of the invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a multi-factor user authentication method and system based on eye tracking and gesture interaction in a metaverse environment.

[0005] The user's identity is authenticated through a multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment. At the same time, eye movement data and gesture data are collected in the background, and fusion features are used for implicit authentication. The user must pass both explicit authentication and implicit authentication to be identified as a legitimate user.

[0006] The present invention provides a multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment, which mainly includes the following steps:

[0007] (1) Explicit authentication and data collection through eye movement and gestures: The user wears a VR helmet with a built-in eye movement module for authentication, and the Leap Motion device is fixed in front of the VR helmet. Explicit authentication is completed through eye movement tracking and gesture interaction. At the same time, the background collects eye movement and gesture data through the eye movement module and Leap Motion.

[0008] (2) Preprocessing eye tracking data and gesture data; removing obvious noise data and classifying eye movement data into fixation and saccade data using an LSTM-based method;

[0009] (3) Extracting eye movement tracking and gesture features: Extracting eye movement features and gesture features from the processed data;

[0010] (4) Use eye movement and gesture features to train the model and perform implicit authentication; use the biometric features extracted in step (3) to classify the random forest classifier to obtain a classification model to implicitly authenticate the user.

[0011] Wherein, step (1) specifically includes:

[0012] In the display authentication method, the user operates a cube dice marked with numbers 1 to 6 and enters a digital password according to the corresponding prompt object to authenticate the user; there are four prompt objects in the metaverse environment, each of which requires the user to enter a specific password; the cube dice is not only marked with numbers, but also has color information. The final input password requires not only the number to be correct, but also the color must meet the requirements of the prompt object; only when the number and color match correctly can the user pass the explicit authentication; the method includes the following steps: (11) the user puts on the VR helmet and completes the five-point eye movement calibration; (12) the user enters the password through the cube The user can select a number by rolling the dice. During this process, the user can get different numbers by grabbing the dice, and the number that the dice faces up is the number selected by the user; (13) Six coloring modules are set up in the metaverse environment of explicit authentication, and each module corresponds to a color. The user grabs the dice and collides with the coloring module to change the color of the dice through this interaction; (14) The user selects the prompt object through eye movement interaction. When the gaze point is on the prompt object and the user uses the left hand to make an "OK" gesture to complete the password input; (15) The password entered by the user is verified. The user can only pass the verification if the number and color match correctly.

[0013] In step (1), the size of the password space for explicit authentication is (6x6) 4=1,679,616. During the explicit authentication process, the user's eye movement data and gesture data will be collected in the background through the eye movement module and Leap Motion. The raw data of the eye movement data includes x-axis coordinates, y-axis coordinates, timestamps and other data, and the gesture data includes the Frame object recorded by Leap Motion, timestamps and other data.

[0014] Wherein, step (4) specifically includes:

[0015] (41) The extracted gesture features are used to train the gesture random forest classifier. Multiple decision trees are constructed by randomly extracting multiple different subsets from the original training data set. The final classification decision is then made based on the voting results of these decision trees. The classification decision function is as follows:

[0016]

[0017] Among them, I(g k (x) = Y) is the indicative function, g k (x) is the classification model of a single decision tree, and Y is the target variable;

[0018] (42) The extracted eye movement features are used to train the eye movement random forest classifier, where the eye movement features are divided into fixation and saccade features. The two features are fused at the decision layer, and the fixation weight and saccade weight are 0.5 and 0.5 respectively;

[0019] (43) The eye movement features and gesture features are fused at the decision layer, and the classification results of the eye movement and gesture random forest classifiers are fused by weighted averaging, where the eye movement weight and gesture weight are 0.5 and 0.5 respectively; the classification results are used to determine whether the user has passed the implicit authentication;

[0020] (44) If a user passes both explicit authentication and implicit authentication, he or she will be judged as a legitimate user.

[0021] The present invention also provides a multi-factor user authentication system based on eye tracking and gesture interaction in a metaverse environment, which comprises the following modules in sequence:

[0022] (1) Explicit authentication method module, used for explicit authentication in the Metaverse;

[0023] (2) Data preprocessing module, which preprocesses the collected data for subsequent model training;

[0024] (3) Feature extraction and model training module, which extracts eye movement features and gesture features from the processed data and uses biometric features to train a random forest classifier;

[0025] (4) Implicit authentication method module, which uses biometrics for implicit authentication in the metaverse.

[0026] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment of the present invention is implemented.

[0027] The present invention further provides a computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method of the present invention is implemented.

[0028] The explicit authentication method of the present invention interacts through eye movements and gestures. Since the VR helmet blocks the user's eyes and the user's gestures are difficult to understand, it is difficult for an attacker to guess the password by observing the user's movements. This method can effectively deal with shoulder surfing attacks. The present invention also includes an implicit authentication method, which trains a random forest classifier using the biometrics collected when the user performs explicit authentication to authenticate the user in the background. Even if the attacker successfully guesses the password, he or she will be identified as an illegal user because the biometrics are different from those of the user, greatly increasing the security of the metaverse.

[0029] The advantage of the present invention is that, by combining explicit authentication with implicit authentication, the shoulder surfing attack problem existing in the metaverse user authentication is well solved, security is increased, and the use of eye tracking and gestures for authentication will not destroy the user's immersion in the metaverse. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 It is a schematic diagram of a flow chart of the method of the present invention.

[0032] Figure 2 It is the LSTM eye movement event classification model of the present invention.

[0033] Figure 3 It is a schematic diagram of the framework of the user classification method based on eye movement of the present invention.

[0034] Figure 4 It is a schematic diagram of the framework of the gesture-based user classification method of the present invention.

[0035] Figure 5 It is a basic logical structure diagram of the system of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the accompanying drawings and embodiments to provide a detailed and complete description of a multi-factor user authentication method and system based on eye tracking and gesture interaction in a metaverse environment of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limitations of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example 1

[0038] See also Figure 1 , a flowchart of a multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment provided by an embodiment of the present invention is provided, and the steps thereof are described in detail:

[0039] (1) Explicit authentication and data collection through eye movements and gestures;

[0040] After completing eye movement calibration using the five-point calibration method, the user authenticates by operating a cube dice marked with numbers 1 to 6 and entering a digital password according to the corresponding prompt object. There are four prompt objects in the metaverse environment, each of which requires the user to enter a specific password. The cube dice is not only marked with numbers, but also with color information. Users can change the color of the dice by interacting with the coloring module in the environment. Six coloring modules are set in the metaverse environment of explicit authentication, each corresponding to a color. Users use cube dice to select numbers. During this process, users can get different numbers by grabbing the dice. After selecting a number, users need to collide the dice with the coloring module to change the color of the dice through this interaction. Users select the prompt object through eye movement interaction. When the gaze point is on the prompt object and the user uses the left hand to make an "OK" gesture, the password is entered. The final input password requires not only the number to be correct, but also the color must meet the requirements of the prompt object. Only when the number and color match correctly can the user pass the verification. The size of the password space for explicit authentication is (6x6)4=1,679,616. During the explicit authentication process, the user's eye movement data and gesture data will be collected in the background through the eye movement module and Leap Motion. The raw data of the eye movement data includes x-axis coordinates, y-axis coordinates, timestamps and other data, and the gesture data includes the Frame object recorded by Leap Motion, timestamps and other data.

[0041] (2) Preprocessing eye tracking data and gesture data;

[0042] First, remove some obvious noise such as coordinates out of range, and then process the original eye movement data into the form of x-axis speed + y-axis speed. Since the speed of eye movement during fixation and saccade movement is quite different, global normalization can easily compress the slower fixation to close to 0, losing most of the eye movement information. Therefore, an LSTM-based eye movement event classification algorithm is used to classify the original eye movement data into fixation data and saccade data, and at the same time, some noise data such as blinking in the original data can be removed. Figure 2 The figure shows the LSTM eye movement event classification model, which mainly includes two LSTM hidden layers with 128 and 64 nodes respectively. ReLu is used as the activation function, and Dropout is added after each LSTM layer to prevent the model from overfitting. The values ​​are all set to 0.2. When training the model, the data set is divided into a test set, a validation set, and a test set at 60%, 20%, and 20%. The gesture data is obtained by calling the members of each Frame object recorded by Leap Motion. The data such as the palm width and palm speed of the left and right hands are obtained in the Hand class, and the data such as the width and length of each finger of the left and right hands are obtained in the Finger class. Due to the high accuracy of the LeapMotion device, physiological data such as finger width in the data set can be used by removing some obvious noise data.

[0043] (3) Extract eye tracking and gesture features;

[0044] 16 eye movement features and 26 gesture features were extracted from the eye movement data and gesture data respectively. The eye movement features include total authentication time, gaze time, average gaze speed, maximum gaze speed, minimum gaze speed, median gaze speed, standard deviation of gaze speed, glance time, average glance speed, maximum glance speed, minimum glance speed, median glance speed, standard deviation of glance speed, maximum pupil size, minimum pupil size, and average pupil size. The gesture features include total authentication time, input interval time, average speed of left palm, average speed of right palm, width of five fingers of left hand, width of five fingers of right hand, length of five fingers of left hand, length of five fingers of right hand, width of left palm, and width of right palm. These features include behavioral characteristics and physiological characteristics.

[0045] (4) Use eye movement and gesture features to train the model and perform implicit verification;

[0046] Select decision layer fusion to fuse eye movement features and gesture features. First, use eye movement features alone to train the eye movement classifier, and use gesture features alone to train the gesture classifier. Then, the respective classification results are fused by weighted averaging, where the eye movement weight and gesture weight are 0.5 and 0.5 respectively. Figure 3 , Figure 4These are the method frameworks for eye movement user classification and gesture user classification. When classifying eye movements, gaze and scan features are extracted respectively, and fused at the decision layer through weighted average. The gaze weight and scan weight are 0.5 and 0.5 respectively. When classifying gestures, physiological features and behavioral features are fused at the feature layer. The classifiers are all random forest classifiers, which construct multiple decision trees by randomly extracting multiple different subsets from the original training data set, and then make the final classification decision based on the voting results of these decision trees. The classification decision function is as follows:

[0047]

[0048] Among them, I(g k (x) = Y) is the indicative function, g k (x) is the classification model of a single decision tree, and Y is the target variable.

[0049] The classification results are used to determine whether the user has passed implicit authentication. If the user has passed both explicit and implicit authentication, he or she will be judged as a legitimate user.

[0050] Example 2

[0051] like Figure 5 As shown, a basic logical structure diagram of a multi-factor user authentication system based on eye tracking and gesture interaction in a metaverse environment provided by an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown. The functional modules / units in the system can be hardware modules / units or software modules / units, and mainly include:

[0052] (1) Explicit authentication method module, used for explicit authentication in the Metaverse;

[0053] (2) Data preprocessing module, which preprocesses the collected data for subsequent model training;

[0054] (3) Feature extraction and model training module, which extracts eye movement features and gesture features from the processed data and uses biometric features to train a random forest classifier;

[0055] (4) Implicit authentication method module, which uses biometrics for implicit authentication in the metaverse.

[0056] Example 3

[0057] A computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, the multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment described in Example 1 is implemented.

[0058] Example 4

[0059] A computing device includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in Example 1 is implemented.

[0060] In the embodiments of the present invention, each module can be integrated into one, or can be deployed separately, or can be further split into multiple sub-modules. Each module can be distributed in the system of the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more systems different from the embodiments of the present invention.

[0061] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0062] The above disclosure is only a specific embodiment of the present invention, but the present invention is not limited thereto. Those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Obviously, these changes and modifications should all fall within the scope of protection required by the present invention.

Claims

1. A multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment, characterized in that: The following steps are involved: (1) Explicit authentication and data collection through eye movements and gestures; The user wears a VR helmet with a built-in eye-tracking module for authentication. The Leap Motion device is fixed in front of the VR helmet. The explicit authentication is completed through eye-tracking and gesture interaction. At the same time, the eye-tracking module and Leap Motion are used to collect eye-tracking and gesture data in the background. (2) Preprocessing eye tracking data and gesture data; Remove obvious noise data and classify eye movement data into fixation and saccade data using an LSTM-based method; (3) Extract eye tracking and gesture features; Extract eye movement features and gesture features from the processed data; (4) Use eye movement and gesture features to train the model and perform implicit verification; The biometric features extracted in step (3) are used to classify the random forest classifier to obtain a classification model to implicitly authenticate the user.

2. The multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment according to claim 1 is characterized in that: Step (1) specifically includes: In the display authentication method, the user operates a cube dice marked with numbers 1 to 6 and enters a digital password according to the corresponding prompt object to authenticate the user; there are four prompt objects in the metaverse environment, each of which requires the user to enter a specific password; the cube dice is not only marked with numbers, but also has color information. The final input password requires not only the number to be correct, but also the color must meet the requirements of the prompt object; only when the number and color match correctly can the user pass the explicit authentication; the method includes the following steps: (11) the user puts on the VR helmet and completes the five-point eye movement calibration; (12) the user enters the password through the cube The user can select a number by rolling the dice. During this process, the user can get different numbers by grabbing the dice, and the number that the dice faces up is the number selected by the user; (13) Six coloring modules are set up in the metaverse environment of explicit authentication, and each module corresponds to a color. The user grabs the dice and collides with the coloring module to change the color of the dice through this interaction; (14) The user selects the prompt object through eye movement interaction. When the gaze point is on the prompt object and the user uses the left hand to make an "OK" gesture to complete the password input; (15) The password entered by the user is verified. The user can only pass the verification if the number and color match correctly.

3. The multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment as claimed in claim 2, characterized in that: In step (1), the size of the password space for explicit authentication is (6x6) 4 =1,679,616. During the explicit authentication process, the user's eye movement data and gesture data will be collected in the background through the eye movement module and Leap Motion. The raw data of the eye movement data includes x-axis coordinates, y-axis coordinates, timestamps and other data, and the gesture data includes the Frame object recorded by Leap Motion, timestamps and other data.

4. The multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment according to claim 1 is characterized in that: Step (4) specifically includes: (41) The extracted gesture features are used to train the gesture random forest classifier. Multiple decision trees are constructed by randomly extracting multiple different subsets from the original training data set. The final classification decision is then made based on the voting results of these decision trees. The classification decision function is as follows: Among them, I(g k (x) = Y) is the indicative function, g k (x) is the classification model of a single decision tree, and Y is the target variable; (42) The extracted eye movement features are used to train the eye movement random forest classifier, where the eye movement features are divided into fixation and saccade features. The two features are fused at the decision layer, and the fixation weight and saccade weight are 0.5 and 0.5 respectively; (43) The eye movement features and gesture features are fused at the decision layer, and the classification results of the eye movement and gesture random forest classifiers are fused by weighted averaging, where the eye movement weight and gesture weight are 0.5 and 0.5 respectively; the classification results are used to determine whether the user has passed the implicit authentication; (44) If a user passes both explicit authentication and implicit authentication, he or she will be judged as a legitimate user.

5. A multi-factor user authentication system based on eye tracking and gesture interaction in a metaverse environment, characterized in that include: Explicit authentication method module, used for explicit authentication in the Metaverse; Data preprocessing module, which preprocesses the collected data for subsequent model training; Feature extraction and model training module, which extracts eye movement features and gesture features from the processed data and uses biometric features to train a random forest classifier; Implicit authentication method module, using biometrics for implicit authentication in the Metaverse.

6. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a multi-factor user authentication method based on eye tracking and gesture interaction in a metaverse environment as described in the claims is implemented.

7. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method described in the claims is implemented.

Images