A secure face authentication method based on eye movement
Through the secure face authentication method based on eye movement, the identity is verified by ceiving points and periophthalmic features, the security risks of the face authentication system in the face of video playback, 3D masks and glasses attacks are solved, and high security is achieved and low-cost face authentication for mobile devices.
Patent Information
- Application Number
- CN202111541072.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-12-16
AI Technical Summary
The existing face authentication system has security risks when facing video playback attacks, 3D mask attacks and glasses attacks. The highly secure face authentication system has high requirements for hardware, which limits its general promotion on smartphones. .
Using a safe face authentication method based on eye movement, eye movement is detected through ordinary cameras, gaze point sequence and periophthalmic features are used for live detection, and a gaze point estimation model and periophthalmic feature extraction model are constructed, combining the gaze point and periophthalmic features. Similarity verifies user identity and resists attacks.
It realizes that without relying on additional hardware, improves the security and accuracy of the face authentication system, defends against multiple attacks, reduces hardware costs, and is suitable for mobile devices such as smartphones.
Smart Images

Figure CN114299569B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of face authentication security enhancement, and in particular relates to a secure face authentication method based on eye movement. Background Art
[0002] With the development of face recognition technology, face recognition systems have become popular due to their high efficiency and convenience, and have gradually become the mainstream identity authentication method on mobile devices. They are widely used in daily life scenarios such as device unlocking and online payment. At the same time, the widespread use of face recognition, especially in key scenarios involving personal privacy and property security such as mobile authentication and mobile payment, has also brought many privacy risks and security issues. On the one hand, with the popularity of social software, a large number of face photos and videos are exposed on social networks, and face information is within reach. Attackers can use advanced machine learning technology to forge face recognition videos, and even imitate various facial expressions in real time and apply them to the victim's 3D face model, thereby bypassing the face anti-spoofing mechanism based on simple responses (such as opening the mouth, blinking, etc.) and 3D features. On the other hand, using adversarial sample attack methods, attackers can carefully design facial accessories (such as glasses) or makeup that are not easy to cause alarm, so that the neural network of the face recognition system will identify the attacker as the target user, or directly evade the detection of the face recognition system. In addition, existing highly secure face recognition systems often have high requirements for devices, such as iPhone's FaceID, which uses depth sensors to provide secure 3D face recognition. However, additional hardware consumes screen space and greatly increases hardware costs, limiting the widespread promotion of high-security face recognition systems on smartphones. Human eye movements are agile and spontaneous, and cannot be forged in real time. Advanced eye tracking technology can be used to track the position of a person's gaze point using the front camera of a smartphone without relying on additional hardware, thereby establishing a faster and more practical response mechanism to assist the face recognition system in completing liveness detection. At the same time, the individual uniqueness of eye movement biometrics can be used to verify the user's identity, preventing attackers from forging identities by wearing masks or counterfeit glasses, making face recognition systems on mobile devices more secure and reliable.
[0003] Eye tracking is a technology that detects changes in a person's line of sight or gaze position, and its core is gaze point estimation. Through gaze point estimation, the gaze point position of a person's eyes on the mobile phone screen can be roughly predicted. Therefore, liveness detection can be performed by comparing the consistency between the gaze point position of the person's eyes and the random point position (gaze point sequence) displayed on the screen. In addition, eye movement is also an agile and triggerable biometric feature. In addition to fine features such as eye movement speed and reaction time, the periocular features that change with eye movement also contain rich personal identity information. Although the biological soft features contained in the periocular area are not sufficient as a basis for identity authentication alone, they can be used as a basis for verifying whether the identity corresponding to the periocular area is consistent with the identity corresponding to the full face. Moreover, the morphological changes in the periocular area during eye movement can be captured by the front camera of a smartphone, so the dynamic changes of periocular features can be used for periocular authentication.
[0004] 3D mask attacks and adversarial glasses attacks can use the attacker's real-time eye movements to forge the victim's eye movement video, but the attacker must wear the victim's mask or carefully designed adversarial glasses, which will change the victim's peri-ocular features. Therefore, during eye tracking, it is necessary to pay real-time attention to and compare the authenticator's peri-ocular features to assist eye tracking in completing liveness detection.
[0005] Without any hardware assistance (using only the front camera of the mobile phone), the accuracy of existing eye tracking solutions is poor, which greatly affects the accuracy of liveness detection using eye tracking. Therefore, how to improve the performance of eye tracking on mobile devices is an important issue. In addition, the trade-off between the diversity of the combination of random point sequences (fixation point sequences) on the screen and the accuracy of eye tracking also needs to be considered. Summary of the invention
[0006] In order to solve the above technical problems, the present invention provides a secure face authentication method based on eye movement.
[0007] The present invention is committed to using ordinary cameras to detect eye movements, taking eye movements as a liveness detection method for face authentication, and researching and exploring a secure face authentication method based on eye movements, which can resist existing video replay attacks, 3D mask attacks, and anti-glasses attacks against face authentication systems, and improve the security of face authentication systems on mobile devices such as smartphones.
[0008] The technical solution adopted by the present invention is: a safe face recognition method based on eye movement, comprising the following steps:
[0009] Step 1, generating a fixation point sequence, i.e. randomly selecting a fixation point sequence within a preset fixation point sequence range and displaying them on the screen in sequence;
[0010] Step 2: video frame acquisition and screening, and the screened video frames are used as images to be processed;
[0011] Step 3: face image processing, including removing the video transition frames in the non-attention state, extracting the face key points from the processed video frames, and cropping the human eye image according to the needs of subsequent authentication;
[0012] Step 4, gaze point feature extraction, including constructing an input sequence containing gaze information, building and training a gaze point estimation model, and then using the gaze point estimation model to extract gaze point features;
[0013] Step 5, extracting periocular features, including constructing an image input sequence containing periocular features, building and training a periocular feature extraction model, and then extracting periocular features using the periocular feature extraction model;
[0014] Step 6, user legitimacy verification, includes comparing the similarity of gaze point features and periorbital features in each identity authentication stage and user registration stage to verify the legitimacy of the user. If and only if the user passes both gaze point authentication and periorbital authentication, he or she will be considered a legitimate user.
[0015] Furthermore, the specific implementation process of step 1 includes the following sub-steps:
[0016] Step 1.1, selection of the range of fixation point sequence: select a random point from a number of points evenly distributed on the screen, denoted as P = {p1, p2, ..., p n},p i =(x i ,y i ), the different random points displayed on the screen are kept at a certain interval, where n represents the number of random points selected, and x i is the vertical distance from the corresponding point i to the left edge of the screen, y i is the vertical distance from the corresponding point i to the upper edge of the screen, in pixels;
[0017] Step 1.2, randomly select a fixation point sequence; randomly select K fixation point sequences from P to form a fixation point sequence C = {c1, c2, ..., c K},c i ∈P, where K is the length of the fixation sequence;
[0018] Step 1.3, display the fixation point sequence; display the fixation point sequence generated in step 1.2 on the screen in sequence, with each point lasting for a certain period of time.
[0019] Furthermore, the specific implementation process of step 2 is as follows;
[0020] Use the front camera of an ordinary mobile phone to shoot a face authentication video, and select stable and high-quality video frames as the images to be processed. A random sequence of K gaze points will be displayed on the smartphone screen, and the user needs to look at the points that appear on the screen in time; at the same time, call the front camera to shoot and save the video frame frame of the user tracking the points on the screen i , and record the coordinates of the random points displayed on the screen (x i ,y i ), and temporarily store the generated video frames in a buffer for processing; in order to improve efficiency and reduce cache consumption, while collecting video frames, the device acceleration value corresponding to each frame of the image is recorded It is used to reflect the stability of the device when shooting the frame. Before the previous frame is processed, the frame with the smallest corresponding acceleration value is selected from all the collected frames, that is, the most stable frame is added to the buffer and used as the image to be processed in the next frame.
[0021] Whenever the processing of the previous frame of image is completed, the frame in the buffer is read for subsequent processing.
[0022] Furthermore, the specific implementation process of step 3 is as follows;
[0023] Step 3.1: Remove the transition image of the human eye. For each original face image captured by the front camera of the mobile phone, calculate the pupil center O p (x p ,y p ) and inner corner of eye O c (x c ,y c ) distance:
[0024]
[0025] Remove the gap between two random points on the screen at a distance d. pc Mutated face images;
[0026] Step 3.2, image cropping, extract facial feature points from the face image processed in step 3.1 to obtain several facial feature points, and crop the left eye image I with a size of N1×N1 based on these facial feature points l and right eye image I r , record the eye corner coordinates L of the left and right eyes. The eye corner coordinates L of the left and right eyes are an eight-dimensional vector, including the inner and outer eye corner coordinates of the left and right eyes of a person, recorded as L = ((x r1 ,y r1 ),(x r2 ,y r2 ),(x l1 ,y l1 ),(x l2 ,yl2 )), used for gaze feature point extraction, where (x r1 ,y r1 ) represents the pixel coordinates of the outer corner of the right eye in the original face image. Similarly, (x r2 ,y r2 )、(x l1 ,y l1 )、(x l2 ,y l2 ) correspond to the inner corner coordinates of the right eye, the inner corner coordinates of the left eye, and the outer corner coordinates of the left eye; crop the left eye periphery image E of size N2×N3 l and right eye periorbital image E r , used for periocular feature extraction.
[0027] Furthermore, the specific implementation process of step 4 is as follows;
[0028] Step 4.1: Construct the input sequence. The input sequence of the gaze point estimation model is denoted as sequence, which includes: left eye image I l , right eye image I r , the eye grid G and the eye corner coordinates L of the left and right eyes; where the left eye image I l and right eye image I r Provide the eye position information of both eyes to the gaze point estimation model. The eye grid G is a binary mask with a size of 25×25, which provides the neural network with the position information of the human eye relative to the camera. The generation process of the eye grid G is: first, the original face image is mapped into a 25×25 mask area through affine transformation, and then the position of the human eyes in the image is mapped to the mask area and marked as 1, and the rest of the non-human eye area is marked as 0; the left and right eye corner coordinates L are an eight-dimensional vector, including the inner and outer eye corner coordinates of the left and right eyes, denoted as L=((x r1 ,y r1 ),(x r2 ,y r2 ),(x l1 ,y l1 ),(x l2 ,y l2 )), where (x r1 ,y r1 ) represents the pixel coordinates of the outer corner of the right eye in the original face image. Similarly, (x r2 ,y r2 )、(x l1 ,y l1 )、(x l2 ,y l2 ) correspond to the inner corner coordinates of the right eye, the inner corner coordinates of the left eye, and the outer corner coordinates of the left eye, respectively, which provides the gaze point estimation model with the head posture information of the person;
[0029] The input sequence of the overall gaze estimation model is represented as sequence(I l ,I r ,G,L);
[0030] Step 4.2: Build a gaze point estimation model. For the left eye image I in the input sequence sequence l With the right eye image I r , after the first convolutional layer CONV1, the maximum pooling layer, the second convolutional layer CONV2, the maximum pooling layer, the third convolutional layer CONV3, the fourth convolutional layer CONV4, and the fully connected layer FC1, the feature vector fea extracted from the eye image is obtained. I ;
[0031] For the human eye grid G in the input sequence sequence, it passes through two fully connected layers FC2 and FC3 in sequence to obtain the feature vector fea extracted from the human eye grid G ;
[0032] For the eye corner coordinates L of the left and right eyes in the input sequence sequence, the feature vector fea extracted from the eye corner coordinates is obtained by passing through three fully connected layers FC4, FC5, and FC6 in sequence. L ;
[0033] Connect the above feature vectors fea I ,fea G ,fea L , and obtain the intermediate output feature vector of the gaze point estimation model:
[0034] fea IGL =(feat I ,feat G ,feat L )
[0035] For fea IGL , passing through two fully connected layers FC7 and FC8 in sequence, the gaze point coordinates finally estimated by the gaze point estimation model are obtained.
[0036] Step 4.3, train the gaze point estimation model, use the mean square error as the loss function in the model training process, and the mean square error calculation formula is:
[0037] Loss(x pre ,x true )=(x pre -x true ) 2
[0038] where x true is the coordinate of the random point displayed on the screen, xpre The coordinates of the gaze point predicted by the model are train =sequence1,sequence2,…,sequence n Input the model for training, where n is the size of the training set and the test data S val =sequence1,sequence2,…,sequence m Input the model for verification, where m is the size of the verification set, which is used to observe the model training status and adjust the model-related parameters in time;
[0039] Step 4.4, extract gaze features, use the pre-trained gaze point estimation model as the gaze feature extractor E, and extract the point p that the human eye is looking at i The corresponding input sequence sequence i Input model, where p i Belongs to the gaze point set P, and takes the vector output by the model after the fully connected layer FC7 as the gaze point feature in and The fixation point is p i The left and right eye images at The fixation point is p i The human eye grid, The fixation point is p i The coordinates of the left and right eye corners at that time.
[0040] Furthermore, the first convolutional layer CONV1 uses 96 convolution kernels of size 11×11 to perform convolution operations on the input image; the second convolutional layer CONV2 uses 256 convolution kernels of size 5×5; the third convolutional layer CONV3 uses 384 convolution kernels of size 3×3; and the fourth convolutional layer CONV4 uses 64 convolution kernels of size 1×1.
[0041] Furthermore, the specific implementation process of step 5 is as follows:
[0042] Step 5.1, construct the input sequence, using the left and right eye periorbital images eyes(E l ,E r ) as the input of the periocular feature extraction model;
[0043] Step 5.2: Build an eye periphery feature extraction model. For the input left eye image E l and right eye picture E r, sequentially passing through: the first convolution layer CONV11, the second convolution layer CONV12, the third convolution layer CONV13, the fourth convolution layer CONV14, each convolution layer is connected to a maximum pooling layer, and finally the left eye features and the right eye features are obtained, and the peri-eye features are obtained after splicing;
[0044] Step 5.3: Train the periocular feature extraction model. Combine soft-max loss and center loss as loss functions to train the periocular feature extraction model. The calculation formula of soft-max loss is:
[0045]
[0046] Where N represents the total number of classification categories; label i is a 1×N vector, in which the item representing the true label of the sample is 1, and all other items are 0; D j It is the j-th value of the soft-max output vector D, which represents the probability that this sample belongs to the j-th category;
[0047] The calculation formula of Center loss is:
[0048]
[0049] Where N is still the total number of classification categories, M represents the total number of samples, and x i represents the output features, Indicates label i the center of the category represented;
[0050] The final loss function used to train the model is expressed as:
[0051] Loss=Loss softmax +a*Loss_{center}
[0052] Among them, a is a constant, and the training data E train =eyes1,eyes2,…,eyes n Input the model for training, where n is the size of the training set and the test data E val =eyes1,eyes2,…,eyes m Input the model for verification, where m is the size of the verification set, which is used to observe the model training status and adjust the model-related parameters in time;
[0053] Step 5.4, extract the periorbital features, input the periorbital images of the left and right eyes into the periorbital feature extraction model trained in step 5.3, and output the periorbital feature vector as the user's periorbital feature f peri .
[0054] Furthermore, the first convolution layer CONV11 uses 20 4×4 convolution kernels; the second convolution layer CONV12 uses 40 3×3 convolution kernels; the third convolution layer CONV13 uses 60 3×3 convolution kernels; and the fourth convolution layer CONV14 uses 120 2×2 convolution kernels.
[0055] Furthermore, the specific implementation process of step 6 is as follows;
[0056] Step 6.1, determine the judgment threshold, calculate the equal error rate EER of the periocular feature extraction model and the fixation point estimation model respectively, and use the cosine similarity value that can make the positive samples in the test data with a ratio of more than (1-EER) pass the verification as the threshold. This step uses cosine similarity as a measurement indicator for feature comparison. The calculation formula of cosine similarity is as follows:
[0057]
[0058] Where A represents the first N-dimensional vector, B represents the second N-dimensional vector, and the larger the cosine similarity, the closer the two feature vectors are;
[0059] Step 6.2, gaze point feature similarity comparison, according to the gaze point sequence generated in step 1, compare the user's gaze point features obtained in step 4 with the corresponding gaze point features entered when the legitimate user registered, and use the formula to calculate the cosine similarity between the two p i represents the gaze point, and the gaze point authentication threshold selected in step 6.1 is thr E ,if The gaze point authentication is judged to be passed, otherwise it is not passed;
[0060] Step 6.3, similarity comparison of periocular features: compare the periocular features of the user obtained in step 5 with the periocular features entered when the legitimate user registered, and use the cosine similarity formula to calculate cos(θ peri ), the periocular authentication threshold selected in step 6.1 is thre peri , when cos(θ peri )>thre peri If the eye area is not properly identified, the periorbital authentication is considered passed, otherwise it is considered failed.
[0061] Furthermore, in step 3.1, the facial key points extracted by Face++ are used.
[0062] Compared with the prior art, the advantages and beneficial effects of the present invention are: it realizes high-security, low cognitive load, and easy-to-promote secure face authentication suitable for mobile devices such as smartphones. The high security of the present invention is reflected in that it can defend against the most advanced attacks on face authentication such as media-based face forgery (MFF) attacks and 3D mask attacks. The low cognitive load of the present invention is reflected in that the entire process only involves eye movements, and the operation is simple, and there is no need to remember passwords or learn special operations. The easy promotion of the present invention is reflected in that it does not require the support of additional hardware, so the present invention can be used on any ordinary smartphone equipped with a front camera, and the existing response-type authentication (such as shaking the head, opening the mouth) movements are often exaggerated, especially when performing authentication in public places, which can easily cause embarrassment. The eye movements used in the present invention are not easily noticed by others and are suitable for various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 Schematic diagram of the focus point of the present invention. Figure 1 (a) are the fifteen candidate positions of the fixation point; Figure 1 (b) shows an example of a fixation sequence.
[0064] Figure 2 It is a schematic diagram of the dynamic effect of the gaze point of the present invention.
[0065] Figure 3 It is a structural diagram of the gaze point estimation model of the present invention.
[0066] Figure 4 It is a schematic diagram of an embodiment of the present invention. DETAILED DESCRIPTION
[0067] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0068] The present invention is mainly based on the field of face recognition security enhancement. Considering the spontaneity and agility of human eye movements, the present invention compares the consistency of the user's gaze point with the random points on the screen and verifies the user's unique peri-ocular features, thereby assisting the existing face system in resisting a variety of advanced attacks. The present invention proposes a secure face recognition method based on eye movement. The method can be used as a supplement to the existing face recognition liveness detection method to improve the robustness and practicality of the face recognition system.
[0069] The method provided by the present invention can be implemented by computer software technology, see Figure 4 The present invention provides a method for secure face recognition based on eye movement, comprising the following steps:
[0070] Step 1: Generate the fixation point sequence. The specific implementation process is:
[0071] Randomly select a fixation point sequence within the preset fixation point sequence range and display it on the screen in sequence. It includes the following sub-steps:
[0072] Step 1.1: Selection of the range of the gaze point sequence. For existing eye tracking technology, the minimum gaze point estimation error is about 1.4 cm. Therefore, in order to ensure the accuracy of eye tracking, the different random points displayed on the screen are at least 2.8 cm apart. In view of the fact that the current mainstream mobile phone size is 5.4 to 6.7 inches (5.4 inches is equivalent to 13.7 cm in length and 6.42 cm in width), selecting random points from fifteen evenly distributed points on the screen can maximize the combination diversity of the gaze point sequence while ensuring the accuracy of eye tracking. Therefore, the present invention uses the following method: Figure 1 The fifteen fixed positions shown in (a) are used as the selection range of random points, denoted as P = {p1, p2, ..., p 15},p i =(x i ,y i ), where x i is the vertical distance from the corresponding point to the left edge of the screen, y i It is the vertical distance from the corresponding point to the top edge of the screen, in pixels.
[0073] Step 1.2: Randomly select a fixation point sequence. Randomly select K points from 15 fixed positions P to form a fixation point sequence C = {c1, c2, ..., c K},c i ∈P, where K is the length of the fixation sequence, Figure 1 (b) shows an example of a fixation sequence when K=4.
[0074] Step 1.3: Display the fixation point sequence. The fixation point sequence generated in step 1.2 is displayed on the screen in order, with each point lasting for 1 second. In order to reduce the situation where the user fails to fixate on the points on the screen in time due to lack of concentration, the present invention adds dynamic effects to the random points on the screen to attract the user's attention. Figure 2 .
[0075] Step 2: Video frame acquisition and screening. The specific implementation process is:
[0076] Use the front camera of an ordinary mobile phone to shoot a face authentication video, and select stable and high-quality video frames as the images to be processed. A random sequence of K fixation points will be displayed on the smartphone screen, and the user needs to keep an eye on the points that appear on the screen. At the same time, the system calls the front camera to shoot and save the video frame of the user tracking the points on the screen.i , and record the coordinates of the random points displayed on the screen (x i ,y i ). Because the speed of video frame generation is faster than the speed of image processing, the generated video frames need to be temporarily stored in a buffer for processing. In addition, in order to improve efficiency and reduce cache consumption, the present invention acquires the device acceleration value corresponding to each frame of image through the built-in acceleration sensor of the mobile phone while collecting video frames. It reflects the stability of the device when the frame was taken. Before the previous frame is processed, the frame with the smallest corresponding acceleration value is selected from all the collected frames, that is, the most stable frame is added to the buffer as the image to be processed next.
[0077] Whenever the processing of the previous frame of image is completed, the frame in the buffer is read for subsequent processing.
[0078] Step 3: Face image processing. The specific implementation process is:
[0079] Remove the non-attention state video transition frames, extract the facial key points from the processed video frames, and crop the human eye image according to the needs of subsequent authentication. It includes the following sub-steps:
[0080] Step 3.1: Remove transition images of human eyes. According to step 1, the random points on the screen are displayed alternately in sequence. When switching from one point to the next, the user's eyeball is in an unstable transition state, not a gaze state. In order to eliminate the interference of these transition frames on the estimation of the gaze point, it is necessary to detect and remove the transition frames. For each original face image captured by the front camera of the mobile phone, calculate the pupil center O p (x p ,y p ) and inner corner of eye O c (x c ,y c ) distance:
[0081]
[0082] Remove the gap between two random points on the screen at a distance d. pc Mutated face images.
[0083] Step 3.2: Image cropping. Use Face++ to extract key points from the face image processed in step 3.1 and obtain 81 face key points. Face++ is a new generation cloud-based visual service platform that provides basic face detection, face key point location, face attribute analysis and other visual services. Based on the face key points extracted by Face++, crop the left eye image I with a size of 224×224. l and right eye image Ir , record the eye corner coordinates L of the left and right eyes for gaze feature extraction; crop the left eye periphery image E of size 28×56 l and right eye periorbital image E r , used for periocular feature extraction.
[0084] Step 4: Extract gaze point features. The specific implementation process is:
[0085] Construct an input sequence containing gaze information, build and train a gaze point estimation model, and then use the model to extract gaze point features. It includes the following sub-steps:
[0086] Step 4.1: Construct input sequence. The input sequence (denoted as sequence) of the neural network for gaze point estimation includes: left eye image I l , right eye image I r , the eye grid G and the eye corner coordinates L of the left and right eyes. The left eye image I l and right eye image I r The eye position information of both eyes is provided to the neural network. The eye grid G is a binary mask with a size of 25×25, which provides the neural network with the position information of the human eye relative to the camera. The generation process of the eye grid G is: first, the original face image is mapped into a 25×25 mask area through affine transformation, and then the position of the human eyes in the image is mapped to the mask area and marked as 1, and the rest of the non-eye area is marked as 0. The left and right eye corner coordinates L are an eight-dimensional vector, which contains the inner and outer eye corner coordinates of the left and right eyes of the human, denoted as L=((x r1 ,y r1 ),(x r2 ,y r2 ),(x l1 ,y l1 ),(x l2 ,y l2 )), where (x r1 ,y r1 ) represents the pixel coordinates of the outer corner of the right eye in the original face image. Similarly, (x r2 ,y r2 )、(x l1 ,y l1 )、(x l2 ,y l2 ) correspond to the inner corner coordinates of the right eye, the inner corner coordinates of the left eye, and the outer corner coordinates of the left eye. It provides the neural network with information about the person's head posture. For example, when the person's head in the image is tilted to the left, the distance between the left eye corners will be significantly smaller than that between the right eye corners.
[0087] The overall gaze point estimation input sequence can be expressed as sequence(I l ,Ir ,G,L).
[0088] Step 4.2: Build a gaze point estimation model. The structure of the gaze point estimation model is as follows Figure 3 shown.
[0089] For the left eye image I in the input sequence sequence l With the right eye image I r , sequentially passing through: the first convolution layer CONV1, using 96 convolution kernels of size 11×11 to convolve the input image, followed by maximum pooling; the second convolution layer CONV2, using 256 convolution kernels of size 5×5, also with a maximum pooling layer; the third convolution layer CONV3, using 384 convolution kernels of size 3×3, without pooling; the fourth convolution layer CONV4, using 64 convolution kernels of size 1×1. Then passing through a fully connected layer FC1(128), the feature vector fea extracted from the eye image is obtained I .
[0090] For the human eye grid G in the input sequence sequence, it passes through two fully connected layers FC2 (256) and FC3 (128) in sequence to obtain the feature vector fea extracted from the human eye grid G .
[0091] For the eye corner coordinates L of the left and right eyes in the input sequence sequence, they pass through three fully connected layers FC4 (100), FC5 (16), and FC6 (16) in sequence to obtain the feature vector fea extracted from the eye corner coordinates L .
[0092] Connect the above feature vectors fea I ,fea G ,fea L , and get the intermediate output feature vector of the model:
[0093] fea IGL =(feat I ,feat G ,feat L )
[0094] For fea IGL , passing through two fully connected layers FC7(128) and FC8(2) in sequence, the final estimated gaze point coordinates of the model are obtained.
[0095] Step 4.3: Train the gaze point estimation model. The mean square error is used as the loss function in the model training process. The mean square error calculation formula is:
[0096] Loss(x pre,x true )=(x pre -x true ) 2
[0097] where x true is the coordinate of the random point displayed on the screen, x pre is the gaze point coordinate predicted by the model. train =sequence1,sequence2,…,sequence n Input the model for training, where n is the size of the training set. val =sequence1,sequence2,…,sequence m Input the model for verification, where m is the size of the verification set, which is used to observe the model training status and adjust the model-related parameters in time.
[0098] Step 4.4: Extract gaze features. Use the pre-trained gaze point estimation model as the gaze feature extractor E. The point p that the human eye is looking at is i The corresponding input sequence sequence i Feed the model, where p i Belongs to the set P of 15 possible fixation points, and takes the 128-dimensional vector output by the model after the fully connected layer FC7 (128) as the fixation point feature in and The fixation point is p i The left and right eye images at The fixation point is p i The human eye grid, The fixation point is p i The coordinates of the left and right eye corners at that time.
[0099] Step 5: Extract periocular features. The specific implementation process is:
[0100] Construct an image input sequence containing periorbital features, build and train a periorbital feature extraction model, and then use the model to extract periorbital features. This includes the following sub-steps:
[0101] Step 5.1: Construct the input sequence. Use the left and right eye periorbital images eyes(E l ,E r ) is used as the input of the eye-peripheral feature extraction network. It should be noted that the binocular image here is different from the binocular image of gaze point estimation, which only includes the eyes and does not include other areas such as eyebrows. Each image is resized to 28×56.
[0102] Step 5.2: Build an eye periphery feature extraction model. For the input left eye image E of size 28×56 l and right eye picture E r , sequentially passing through: the first convolution layer CONV1, using 20 4×4 convolution kernels; the second convolution layer CONV2, using 40 3×3 convolution kernels; the third convolution layer CONV3, using 60 3×3 convolution kernels; the fourth convolution layer CONV4, using 120 2×2 convolution kernels. Each convolution layer is connected to a maximum pooling layer, and finally 160-dimensional left eye features and right eye features are obtained, and after splicing, a 320-dimensional peri-ocular feature is obtained.
[0103] Step 5.3: Train the periocular feature extraction model. The present invention combines soft-max loss and center loss as loss functions to train the periocular feature extraction model. Soft-max loss is often used in classification tasks and can increase the inter-class distance of the output features. Its calculation formula is:
[0104]
[0105] Where N represents the total number of classification categories; label i is a 1×N vector, in which the item representing the true label of the sample is 1, and all other items are 0; D j It is the j-th value of the soft-max output vector D, which represents the probability that this sample belongs to the j-th category.
[0106] Center loss can reduce the intra-class distance of the output features, and its calculation formula is:
[0107]
[0108] Where N is still the total number of classification categories, M represents the total number of samples, and x i represents the output features, Indicates label i The center of the category represented.
[0109] The final loss function used to train the model can be expressed as:
[0110] Loss=Loss softmax +0.3*Loss_{center}
[0111] The training data E train =eyes1,eyes2,…,eyes n Input the model for training, where n is the size of the training set. val =eyes1,eyes2,…,eyesm Input the model for verification, where m is the size of the verification set, which is used to observe the model training status and adjust the model-related parameters in time.
[0112] Step 5.4: Extract periocular features. Input the periocular images of the left and right eyes into the periocular feature extraction model trained in step 5.3, and output a 320-dimensional periocular feature vector as the user's periocular feature f peri .
[0113] Step 6: User legitimacy verification. The specific implementation process is:
[0114] Compare the similarity of gaze point features and periorbital features in each identity authentication phase and user registration phase to verify the legitimacy of the user. If and only if the user passes both gaze point authentication and periorbital authentication, he or she will be considered a legitimate user. It includes the following sub-steps:
[0115] Step 6.1: Determine the judgment threshold. Calculate the EER of the periocular feature extraction model and the gaze point estimation model respectively, and use the cosine similarity value that can make more than (1-EER) positive samples in the test data pass the verification as the threshold. EER (Equal Error Rate) is the equal error rate. When the false acceptance rate and false rejection rate of the model classification are equal, this common value is the EER of the model. The smaller the EER value, the higher the model classification accuracy. The judgment threshold corresponding to the EER is the judgment threshold that makes the model achieve the best performance. This step uses cosine similarity as a measurement indicator for feature comparison. The formula for calculating cosine similarity is as follows:
[0116]
[0117] Where A represents the first N-dimensional vector, and B represents the second N-dimensional vector. The larger the cosine similarity, the closer the two feature vectors are.
[0118] Step 6.2: Comparison of gaze point feature similarity. According to the gaze point sequence generated in step 1, the user gaze point features obtained in step 4 are compared. C=c1,c2,…,c K ,c i ∈P, the feature of the corresponding gaze point recorded by the legitimate user when registering Let Substitute the formula to calculate the cosine similarity between the two The gaze point authentication threshold selected in step 6.1 is thr E ,if The gaze point authentication is determined to be passed, otherwise it is not passed.
[0119] Step 6.3: Comparison of similarity of periorbital features. Compare the periorbital features of the user obtained in step 5 peri The eye features recorded by the legitimate user during registration peri ' to compare. Let A = f peri , B = f peri ′, and then bring it into the cosine similarity formula to calculate cos(θ peri ). Note that the periocular authentication threshold selected in step 6.1 is thre peri , when cos(θ peri )>thre peri If the eye area is not properly identified, the periorbital authentication is considered passed, otherwise it is considered failed.
[0120] It should be understood that parts not elaborated in detail in this specification belong to the prior art.
[0121] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.
Claims
1. A secure face authentication method based on eye movement, It is characterized in that The following steps are involved: Step 1, generating a fixation point sequence, i.e. randomly selecting a fixation point sequence within a preset fixation point sequence range and displaying them on the screen in sequence; The specific implementation process of step 1 includes the following sub-steps: Step 1.1, selection of the range of fixation point sequence: select a random point from a number of points evenly distributed on the screen, denoted as P = {p 1 , p 2 , ..., p n }, p i =(x i ,y i ), the different random points displayed on the screen are kept at a certain interval, where n represents the number of random points selected, and x i is the vertical distance from the corresponding point i to the left edge of the screen, yi is the vertical distance from the corresponding point i to the upper edge of the screen, in pixels; Step 1.2, randomly select a fixation point sequence; randomly select K fixation point sequences from P to form a fixation point sequence C = {c 1 , c 2 , ..., c K }, c i ∈P, where K is the length of the fixation sequence; Step 1.3, displaying the fixation point sequence; displaying the fixation point sequence generated in step 1.2 on the screen in sequence, with each point lasting for a certain period of time; Step 2: video frame acquisition and screening, and the screened video frames are used as images to be processed; Step 3: face image processing, including removing the video transition frames in the non-attention state, extracting the face key points from the processed video frames, and cropping the human eye image according to the needs of subsequent authentication; Step 4, gaze point feature extraction, including constructing an input sequence containing gaze information, building and training a gaze point estimation model, and then using the gaze point estimation model to extract gaze point features; The specific implementation process of step 4 is as follows; Step 4.1: Construct the input sequence. The input sequence of the gaze point estimation model is denoted as sequence, which includes: left eye image I l , right eye image I r , the eye grid G and the eye corner coordinates L of the left and right eyes; where the left eye image I l and right eye image I r Provide the eye position information of both eyes to the gaze point estimation model. The eye grid G is a binary mask with a size of 25×25, which provides the neural network with the position information of the human eye relative to the camera. The generation process of the eye grid G is: first, the original face image is mapped into a 25×25 mask area through affine transformation, and then the position of the human eyes in the image is mapped to the mask area and marked as 1, and the rest of the non-human eye area is marked as 0; the left and right eye corner coordinates L are an eight-dimensional vector, including the inner and outer eye corner coordinates of the left and right eyes, denoted as L=((x r1 ,y r1 ), (x r2 ,y r2 ), (x l1 ,y l1 ), (x l2 ,y l2 )), where (x r1 ,y r1 ) represents the pixel coordinates of the outer corner of the right eye in the original face image. Similarly, (x r2 ,y r2 )、(x l1 ,y l1 )、(x l2 ,y l2 ) correspond to the inner corner coordinates of the right eye, the inner corner coordinates of the left eye, and the outer corner coordinates of the left eye, respectively, which provides the head posture information of the person to the gaze point estimation model; The input sequence of the overall gaze estimation model is represented as sequence(I l , I r , G, L); Step 4.2: Build a gaze point estimation model. For the left eye image I in the input sequence sequence l With the right eye image I r , after the first convolutional layer CONV1, the maximum pooling layer, the second convolutional layer CONV2, the maximum pooling layer, the third convolutional layer CONV3, the fourth convolutional layer CONV4, and the fully connected layer FC1, the feature vector fea extracted from the eye image is obtained. I ; For the human eye grid G in the input sequence sequence, it passes through two fully connected layers FC2 and FC3 in sequence to obtain the feature vector fea extracted from the human eye grid G ; For the eye corner coordinates L of the left and right eyes in the input sequence sequence, the feature vector fea extracted from the eye corner coordinates is obtained by passing through three fully connected layers FC4, FC5, and FC6 in sequence. L ; Connect the above feature vectors fea I ,fea G ,fea L , and obtain the intermediate output feature vector of the gaze point estimation model: fea IGL =(feat I ,feat G ,feat L ) For fea IGL , passing through two fully connected layers FC7 and FC8 in sequence, the gaze point coordinates finally estimated by the gaze point estimation model are obtained; Step 4.3, train the gaze point estimation model, use the mean square error as the loss function in the model training process, and the mean square error calculation formula is: Loss(x pre ,x true )=(x pre -x true ) 2 where x true is the coordinate of the random point displayed on the screen, x pre The coordinates of the gaze point predicted by the model are train =sequence 1 , sequence 2 , ..., sequence n Input the model for training, where n is the size of the training set and the test data S val =sequence 1 , sequence 2 ,...,sequence m Input the model for verification, where m is the size of the verification set, which is used to observe the model training status and adjust the model-related parameters in time; Step 4.4, extract gaze features, use the pre-trained gaze point estimation model as the gaze feature extractor E, and extract the point p that the human eye is looking at i The corresponding input sequence sequence i Input model, where p i Belongs to the gaze point set P, and takes the vector output by the model after the fully connected layer FC7 as the gaze point feature in and The fixation point is p i The left and right eye images at The fixation point is p i The human eye grid, The fixation point is p i The coordinates of the left and right eye corners at the time; Step 5, extracting periocular features, including constructing an image input sequence containing periocular features, building and training a periocular feature extraction model, and then extracting periocular features using the periocular feature extraction model; Step 6, user legitimacy verification, including comparing the similarity of gaze point features and periorbital features in each identity authentication stage and user registration stage to verify the legitimacy of the user. If and only if the user passes both gaze point authentication and periorbital authentication, he or she will be considered a legitimate user; The specific implementation process of step 6 is as follows; Step 6.1, determine the judgment threshold, calculate the equal error rate EER of the periocular feature extraction model and the fixation point estimation model respectively, and use the cosine similarity value that can make the positive samples in the test data with a ratio of more than (1-EER) pass the verification as the threshold. This step uses cosine similarity as a measurement indicator for feature comparison. The calculation formula of cosine similarity is as follows: Where A represents the first N-dimensional vector, B represents the second N-dimensional vector, and the larger the cosine similarity, the closer the two feature vectors are; Step 6.2, gaze point feature similarity comparison, according to the gaze point sequence generated in step 1, compare the user's gaze point features obtained in step 4 with the corresponding gaze point features entered when the legitimate user registered, and use the formula to calculate the cosine similarity between the two p i represents the gaze point, and the gaze point authentication threshold selected in step 6.1 is thr E ,if The gaze point authentication is judged to be passed, otherwise it is not passed; Step 6.3, similarity comparison of periocular features: compare the periocular features of the user obtained in step 5 with the periocular features entered when the legitimate user registered, and use the cosine similarity formula to calculate cos(θ peri ), the periocular authentication threshold selected in step 6.1 is thre peri , when cos(θ peri )>thre peri If the eye area is not properly identified, the periorbital authentication is considered passed, otherwise it is considered failed.
2. The secure face authentication method based on eye movement according to claim 1, Features: The specific implementation process of step 2 is as follows; Use the front camera of an ordinary mobile phone to shoot a face authentication video, and select stable and high-quality video frames as the images to be processed. A random sequence of K gaze points will be displayed on the smartphone screen, and the user needs to look at the points that appear on the screen in time; at the same time, call the front camera to shoot and save the video frame frame of the user tracking the points on the screen i , and record the coordinates of the random points displayed on the screen (x i ,y i ), and temporarily store the generated video frames in a buffer for processing; in order to improve efficiency and reduce cache consumption, while collecting video frames, the device acceleration value corresponding to each frame of the image is recorded It is used to reflect the stability of the device when shooting the frame. Before the previous frame is processed, the frame with the smallest corresponding acceleration value is selected from all the collected frames, that is, the most stable frame is added to the buffer and used as the image to be processed in the next frame. Whenever the processing of the previous frame of image is completed, the frame in the buffer is read for subsequent processing.
3. The secure face authentication method based on eye movement according to claim 1, Features: The specific implementation process of step 3 is as follows; Step 3.1: Remove the transition image of the human eye. For each original face image captured by the front camera of the mobile phone, calculate the pupil center O p (x p ,y p ) and inner corner of eye O c (x c ,y c ) distance: Remove the gap between two random points on the screen at a distance d. pc Mutated face images; Step 3.2, image cropping, extract facial feature points from the face image processed in step 3.1 to obtain several facial feature points, and crop the left eye image I with a size of N1×N1 based on these facial feature points l and right eye image I r , record the eye corner coordinates L of the left and right eyes. The eye corner coordinates L of the left and right eyes are an eight-dimensional vector, including the inner and outer eye corner coordinates of the left and right eyes of a person, recorded as L = ((x r1 ,y r1 ), (x r2 ,y r2 ), (x l1 ,y l1 ), (x l2 ,y l2 )), used for gaze feature point extraction, where (x r1 ,y r1 ) represents the pixel coordinates of the outer corner of the right eye in the original face image. Similarly, (x r2 ,y r2 )、(x l1 ,y l1 )、(x l2 ,y l2 ) correspond to the inner corner coordinates of the right eye, the inner corner coordinates of the left eye, and the outer corner coordinates of the left eye; crop the left eye periphery image E of size N2×N3 l and right eye periorbital image E r , used for periocular feature extraction.
4. The method for secure face authentication based on eye movement according to claim 1, Features: The first convolutional layer CONV1 uses 96 convolution kernels of size 11×11 to perform convolution operations on the input image; the second convolutional layer CONV2 uses 256 convolution kernels of size 5×5; the third convolutional layer CONV3 uses 384 convolution kernels of size 3×3; the fourth convolutional layer CONV4 uses 64 convolution kernels of size 1×1.
5. The method for secure face recognition based on eye movement according to claim 3, Features: The specific implementation process of step 5 is as follows: Step 5.1, construct the input sequence, using the left and right eye periorbital images eyes(E l , E r ) as the input of the periocular feature extraction model; Step 5.2: Build an eye periphery feature extraction model. For the input left eye image E l and right eye picture E r , sequentially passing through: the first convolution layer CONV11, the second convolution layer CONV12, the third convolution layer CONV13, the fourth convolution layer CONV14, each convolution layer is connected to a maximum pooling layer, and finally the left eye features and the right eye features are obtained, and the peri-eye features are obtained after splicing; Step 5.3: Train the periocular feature extraction model. Combine soft-max loss and center loss as loss functions to train the periocular feature extraction model. The calculation formula of soft-max loss is: Where N represents the total number of classification categories; label i is a 1×N vector, in which the item representing the true label of the sample is 1, and all other items are 0; D j It is the j-th value of the soft-max output vector D, which represents the probability that this sample belongs to the j-th category; The calculation formula of Centerloss is: Where N is still the total number of classification categories, M represents the total number of samples, and x i represents the output features, Indicates label i the center of the category represented; The final loss function used to train the model is expressed as: Loss=Loss softmax +a*Loss center Among them, a is a constant, and the training data E train =eyes 1 , eyes 2 , ..., eyes n Input the model for training, where n is the size of the training set and the test data E val =eyes 1 , eyes 2 , ..., eyes m Input the model for verification, where m is the size of the verification set, which is used to observe the model training status and adjust the model-related parameters in time; Step 5.4, extract the periorbital features, input the periorbital images of the left and right eyes into the periorbital feature extraction model trained in step 5.3, and output the periorbital feature vector as the user's periorbital feature f peri .
6. The method for secure face authentication based on eye movement according to claim 5, Features: The first convolution layer CONV11 uses 20 4×4 convolution kernels; the second convolution layer CONV12 uses 40 3×3 convolution kernels; the third convolution layer CONV13 uses 60 3×3 convolution kernels; the fourth convolution layer CONV14 uses 120 2×2 convolution kernels.
7. The secure face authentication method based on eye movement according to claim 3, Features: In step 3.1, the facial key points are extracted using Face++.
