A face recognition and authentication method based on line of sight features

Abstract

The present invention discloses a facial recognition and authentication method based on line of sight features, which relates to the field of facial recognition technology. The method involves randomly generating and displaying rapidly flashing stimulus points to induce reflex eye movements in the user, thereby capturing a video of the user's face. The captured video is then processed and fed into a gaze calibration model to obtain accurate eye movement trajectories. Eye movement velocity is calculated based on the eye movement trajectories, and based on user gaze trajectory matching and behavior matching indicators, the method detects whether the object corresponding to the user's facial video is alive. The line of sight feature-based facial recognition and authentication method provided by the present invention addresses the technical problem that existing facial authentication systems are susceptible to spoofing attacks, presenting serious security risks.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition, and more particularly to a face recognition and authentication method based on sight line features. Background Art

[0002] With the development of network and mobile communication technologies, the popularity of smartphones, mobile payments, and numerous cloud services, they have become indispensable tools in our daily lives. In this context, security and privacy issues have attracted widespread attention.

[0003] Compared to face recognition, facial authentication technology requires less hardware on the front-end device and is widely used for remote authentication in banking applications, among other applications. Facial authentication systems first require users to register multiple photos taken from different angles, then store their facial biometrics. During verification, they check whether the facial features of the person requesting access match those registered by the registered user. However, facial authentication systems are vulnerable to spoofing attacks, where attackers can impersonate legitimate users simply by displaying an image. Therefore, facial authentication systems require certain defensive measures. During identity verification, facial authentication systems also perform liveness detection to verify the subject's authenticity.

[0004] However, given that the cameras on most smartphones today can only take two-dimensional photos, facial biometrics were initially extracted from two-dimensional images. Current technologies use various anti-spoofing methods to enhance the security of face authentication. Through an additional local camera or other sensing technology, the system can detect more features of 2D or 3D structures for liveness detection. The stimulus-response model is another commonly used liveness detection method. This method only requires the use of a camera, and the user needs to perform specific behaviors (such as turning the head, blinking, opening the mouth, and making facial expressions). However, some of the required behaviors can be complex and require multiple active cooperation, which consumes the user's patience.

[0005] In addition to detecting simple attacks (such as printed or screen-displayed photos), face authentication must also combat various spoofing attacks. For 3D facial authentication systems that measure the three-dimensional structure of the face, attacks include fabricated masks and 3D-printed head models. For motion-based authentication systems, attacks include sequentially playing several images or synthesizing videos.

[0006] Eye tracking technology has developed rapidly in recent decades and has been widely used in various fields, including attention analysis, consumer behavior research, and driving. Traditional eye trackers use pupil-corneal reflection for eye tracking. Infrared light reflected from the cornea of ​​the eye forms a fixed corneal area in the image. The system can use the relative position of the pupil center and the corneal reflection center as a reference position for detecting gaze. However, since this method requires a high-temporal and spatial resolution infrared camera with a corresponding light source, it is currently difficult to deploy on smartphones. Recently, researchers have used machine learning and deep neural networks to make eye tracking feasible on portable devices equipped with front-facing cameras. For critical tasks such as identity verification, low gaze tracking error is essential.

[0007] Therefore, how to provide a face recognition authentication method with high accuracy and fast detection speed is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a face recognition and authentication method based on line of sight features, focusing on fine-grained eye movements that are difficult for attackers to reproduce. By generating a series of random red dot stimuli, displaying them quickly and sequentially on the screen, and analyzing the user's eye movements to perform eye tracking, the present invention solves the technical problem in the prior art that face recognition systems are susceptible to deception attacks and pose serious security risks.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A face recognition and authentication method based on line of sight features comprises the following steps:

[0011] Step 1: Randomly generate and display rapidly flashing stimulus points to induce reflexive eye movements in the user, and capture a video of the user's face;

[0012] Step 2: Process the captured user facial video and feed it into the gaze calibration model to obtain accurate eye movement tracks.

[0013] Step 3: Calculate the eye movement speed based on the eye movement trajectory, and detect whether the object corresponding to the user's facial video is alive based on the user's gaze trajectory matching and behavior matching indicators.

[0014] Preferably, the step 1 specifically includes: randomly generating and displaying rapidly flashing stimulus points including a fixed screen interval delay, generating continuous stimulus points, and the screen interval delay is the same as the reflex eye movement delay of a person.

[0015] Preferably, based on the Fick-gimbal visual angle model, the eye movement angle between the two consecutive stimulation points is [R min ,R max] is uniformly distributed, R min is the minimum scanning angle that the camera can detect, R max It is the maximum viewing angle based on the constrained head movement of the viewing model.

[0016] Preferably, the processing of the captured user facial video in step 2 specifically includes:

[0017] Step 2.1: Use a convolutional neural network to detect faces and facial landmarks. The first layer of the convolutional neural network extracts facial features, and the last fully connected layer concatenates the face bounding box and facial landmarks, selects the face with the largest area, and obtains its facial landmarks.

[0018] Step 2.2: Crop the face and the left and right eyes of the facial landmarks described in step 2.1 from the original user face video and scale them to a fixed resolution.

[0019] Step 2.3: Use the cropped face and the left and right eyes from the facial landmarks in step 2.2 as input and feed them into the gaze detection network model. Then, use the user-customized calibration optimization model to obtain accurate eye movement tracks.

[0020] Preferably, the step 2.3 specifically includes:

[0021] The gaze detection network model shares weights for the left and right eye convolution layers, uses a fully connected layer to connect the convolution layers of the left and right eye images and the convolution layer of the face image, and this fully connected layer and the feature points are input to the last layer, which outputs the two-dimensional screen coordinates of the gaze point; the eye movement trajectory is composed of the position of the two-dimensional screen coordinates of the gaze point output by the last layer of the gaze detection network model, which is the position coordinate of the Cartesian coordinate system in the xy plane;

[0022] During the training phase, the gaze calibration model sets the weights of training samples according to the Euclidean distance between the stimulus point and the center of the screen; and calibrates the eye movement trajectory through the gaze calibration model to obtain an accurate eye movement trajectory.

[0023] Preferably, the behavior matching index in step 3 specifically includes two parts: a viewing angle index and a gaze detection index.

[0024] Preferably, the viewing angle indicator specifically includes:

[0025] (1) The average angular velocity, maximum angular velocity, and maximum angular acceleration of eye movements during the test;

[0026] (2) the maximum peak angular velocity and average angular velocity of eye movements during saccades and gazes;

[0027] (3) Average scanning time and total scanning time.

[0028] Preferably, the gaze detection indicators specifically include:

[0029] (1) Two-dimensional correlation, average distance, and root mean square error of fixation points;

[0030] (2) Successful fixation rate: The successful fixation rate is the ratio of successful fixations to the number of all stimulus points. A successful fixation means that within a fixed time after the stimulus point appears, the user's fixation point is detected inside a circle with a radius of r and a center of the stimulus point.

[0031] (3) The maximum value of the correlation coefficient in the cross-correlation between the gaze point trajectory and the stimulus point trajectory is calculated on the Cartesian coordinate system xy plane as follows:

[0032]

[0033] Among them, P is the user's gaze point trajectory, G is the stimulus point trajectory, k (k ≥ 0) is the lag time, is the original correlation, n is the maximum value of sequence X and sequence Y, Y i The complex conjugate of

[0034] The correlation estimation sequence is as follows:

[0035]

[0036] in, is the normalization coefficient; n T is the number of frames within the maximum delay threshold;

[0037] (4) The ratio of the number of detected user glances to the number of times the stimulus point changes.

[0038] Preferably, during the viewing angle indicator detection process, the scanning and gazing processes specifically include:

[0039] Step 3.1: Convert the recorded user eye movement trajectory into a view sequence based on the Fick-gimbal model;

[0040] Step 3.2: Detect the converted user eye movement trajectory, including filtering, adaptive scanning and gaze detection.

[0041] Preferably, the adaptive glance and gaze detection specifically includes:

[0042] Adaptive saccade and gaze detection consists of saccade threshold calculation and peak search. The saccade threshold is determined based on the average and variance of the eye angular velocity within a fixed time. The minimum number of peak samples considered as a saccade is determined based on the frame rate and eye movement speed of the smartphone camera. If a peak is detected within a fixed time frame exceeding the saccade threshold and the time frame exceeds the minimum number of peak samples, it is detected as a user saccade; if multiple peaks are detected, it is detected as invalid eye movement; the remaining time frames below the saccade threshold are detected as user gaze.

[0043] The above technical solution demonstrates that, compared to existing technologies, the present invention provides a facial recognition and authentication method based on line-of-sight features. The present invention has the following advantages: it can effectively resist physical domain spoofing attacks and authenticate users based on whether they are legitimate; it can be directly applied to devices without requiring additional hardware or complex user actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 The accompanying figure is a flow chart of face recognition authentication provided by the present invention;

[0046] Figure 2 The accompanying drawing is a schematic diagram of the matching of the sight line trajectory and the excitation point trajectory provided by the present invention;

[0047] Figure 3 The accompanying drawing is a schematic diagram of estimated eye angular velocity scan detection provided by the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The embodiment of the present invention discloses a face recognition and authentication method based on sight line features, comprising:

[0050] Step 1: Randomly generate and display rapidly flashing stimulus points to induce reflexive eye movements in the user, and capture a video of the user's face;

[0051] Step 2: Process the captured user facial video and feed it into the gaze calibration model to obtain accurate eye movement tracks.

[0052] Step 3: Calculate the eye movement speed based on the eye movement trajectory, and detect whether the object corresponding to the user's facial video is alive based on the user's gaze trajectory matching and behavior matching indicators.

[0053] In a specific embodiment, step 1 specifically includes: randomly generating and displaying rapidly flashing stimulus points including a fixed screen interval delay, generating continuous stimulus points, and the screen interval delay is the same as the reflex eye movement delay of a person.

[0054] Reflexive eye movements are based on a stimulus-response mechanism, where the human eye reflexively moves to focus on rapidly changing parts of the image. Therefore, when a user sees a point stimulus displayed on a smartphone, their eyes will perform a conditioned reflex movement. Stimulus-response verification is performed by matching the gaze trajectory and the temporal and spatial relationship of the stimuli to the stimuli, which appear randomly.

[0055] In a specific embodiment, the eye movement angle between two consecutive stimulation points is based on the Fick-gimbal visual angle model. min ,R max ] is uniformly distributed, R min is the minimum scanning angle that the camera can detect, R max It is the maximum viewing angle based on the constrained head movement of the viewing model.

[0056] Furthermore, the point stimulus generation process is specifically as follows: a series of continuous point stimuli are generated with a delay of 0.3s. When a significant change occurs in a person's visual field, the eyes will reflexively make saccadic movements. Given the difference in delay between saccadic movements and other conscious eye movements, the delay between two stimuli is set to 0.3s. Since a close distance between two adjacent point stimuli will lead to a large error in the estimation of the angular velocity of eye movements, and a distance that is too far will cause the user to unconsciously move their head, the visual angle between two adjacent point stimuli is set to be evenly distributed in [5°, 20°], and the pixel value distance before and after the stimulus point is estimated based on the average distance between the human eye and the screen.

[0057] In a specific embodiment, processing the captured user facial video in step 2 specifically includes:

[0058] Step 2.1: Use a convolutional neural network to detect faces and facial landmarks. The first layer of the convolutional neural network extracts facial features, and the last fully connected layer concatenates the face bounding box and facial landmarks, selects the face with the largest area, and obtains its facial landmarks.

[0059] Step 2.2: Crop the face and left and right eyes from the facial landmarks in step 2.1 from the original user face video and scale them to a fixed resolution.

[0060] Step 2.3: Use the cropped face and the left and right eyes from the facial landmarks in step 2.2 as input and feed them into the gaze detection network model. Then, use the user-customized calibration optimization model to obtain accurate eye movement tracks.

[0061] In one specific embodiment, a multi-task cascaded convolutional neural network is used for face detection and facial feature point alignment. The network is robust to faces from different perspectives. The face with the largest area (i.e., closest) is then selected from the identified faces to obtain its facial feature points. The selected face and its left and right eye images are cropped and scaled to a resolution of 224*224. The front layers of the convolutional neural network extract facial features, and the final fully connected layer concatenates the face bounding box with the facial feature points, sharing the previous facial features.

[0062] In a specific embodiment, step 2.3 specifically includes: sharing the weights of the convolutional layers of the left and right eyes of the gaze detection network model, connecting the convolutional layers of the left and right eye images and the convolutional layer of the face image using a fully connected layer, inputting this fully connected layer and the feature points into the last layer, and the last layer outputting the two-dimensional screen coordinates of the gaze point; the eye movement trajectory is composed of the position of the two-dimensional screen coordinates of the gaze point output by the last layer of the gaze detection network model, which is the position coordinate of the Cartesian coordinate system in the xy plane;

[0063] Among them, the eye movement trajectory is the two-dimensional coordinate output of the above-mentioned gaze detection network, that is, the position of the two eyes' sight moving on the screen when observing the stimulus point extracted from n frames of images; the position is the position coordinate of the Cartesian coordinate system in the xy plane of the smartphone (with the upper left corner of the screen as the origin), composed of a complex number (real coordinate is x, imaginary coordinate is y).

[0064] During the training phase, the gaze calibration model sets the weights of training samples according to the Euclidean distance between the stimulus point and the center of the screen; the eye movement trajectory is calibrated through the gaze calibration model to obtain accurate eye movement trajectory.

[0065] In one specific embodiment, in step 2, the previously processed facial and eye image sequences are fed into an existing open-source image-based gaze tracking model (e.g., the iTracker model) for gaze detection. The gaze trajectory is then calibrated using a support vector regression model to obtain accurate eye movement tracks. During the initialization phase, when the user registers their face, the present invention sets up a gaze-calibrated support vector regression model.

[0066] Among them, the gaze calibration model adopts a machine learning regression model, and the Euclidean distance unit is centimeters.

[0067] In a specific embodiment, the behavior matching index in step 3 specifically includes two parts: a viewing angle index and a gaze detection index.

[0068] In a specific embodiment, the viewing angle indicator specifically includes:

[0069] (1) The average angular velocity, maximum angular velocity, and maximum angular acceleration of eye movements during the test;

[0070] (2) the maximum peak angular velocity and average angular velocity of eye movements during saccades and gazes;

[0071] (3) Average scanning time and total scanning time.

[0072] In a specific embodiment, Figure 2 As shown in Figure 2, the gaze detection indicators specifically include:

[0073] (1) Two-dimensional correlation, average distance, and root mean square error of fixation points;

[0074] (2) Successful fixation rate: The successful fixation rate is the ratio of successful fixations to the number of all stimulus points. A successful fixation means that within a fixed time after the stimulus point appears, the user's fixation point is detected inside a circle with a radius of r and a center of the stimulus point.

[0075] (3) The maximum value of the correlation coefficient in the cross-correlation between the gaze point trajectory and the stimulus point trajectory is calculated on the Cartesian coordinate system xy plane as follows:

[0076]

[0077] Among them, P is the user's gaze point trajectory, G is the stimulus point trajectory, k (k ≥ 0) is the lag time, is the original correlation, n is the maximum value of sequence X and sequence Y, Y i The complex conjugate of

[0078] The correlation estimation sequence is as follows:

[0079]

[0080] in, is the normalization coefficient, n T is the number of frames within the maximum delay threshold;

[0081] (4) The ratio of the number of detected user glances to the number of times the stimulus point changes.

[0082] The identification of the number of user scans and the number of times the stimulus point changes is specifically performed by using the Nystrom-Hol mqvist algorithm to perform gaze classification.

[0083] In a specific embodiment, during the viewing angle indicator detection process, the scanning and gazing process specifically includes:

[0084] Step 3.1: Convert the recorded user eye movement trajectory into a view sequence based on the Fick-gimbal model;

[0085] Step 3.2: Detect the converted user eye movement trajectory, including filtering, adaptive scanning and gaze detection.

[0086] In a specific embodiment, adaptive glance and gaze detection specifically includes:

[0087] like Figure 3 As shown in the figure, adaptive saccade and gaze detection consists of saccade threshold calculation and peak search. The saccade threshold is determined based on the average and variance of the eye angular velocity within a fixed time. The minimum number of peak samples considered as a saccade is determined based on the frame rate and eye movement speed of the smartphone camera. If a peak is detected within a fixed time frame exceeding the saccade threshold and the time frame exceeds the minimum number of peak samples, it is detected as a user saccade; if multiple peaks are detected, it is detected as invalid eye movement; the remaining time frames below the saccade threshold are detected as user gaze.

[0088] In one specific embodiment, the recorded user gaze trajectory is converted into a view sequence based on the Fick-Gimbal model. Gaze event detection is performed, including Savitzky-Golay filtering and denoising, adaptive saccade thresholding and peak detection, saccade start / offset detection, and gaze detection. The minimum number of peak samples considered a potential saccade is set to 1, and the start-peak velocity threshold is set to 50° / s.

[0089] In step 3, support vector machines are used to perform authentication based on user gaze behavior indicators.

[0090] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0091] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A face recognition and authentication method based on sight line features, characterized in that: include: Step 1: Randomly generate and display rapidly flashing stimulus points to induce reflexive eye movements in the user, and capture a video of the user's face; Step 2: Process the captured user facial video and feed it into the gaze calibration model to obtain accurate eye movement tracks. Step 3: Calculate the eye movement speed based on the eye movement trajectory, and detect whether the object corresponding to the user's facial video is alive based on the user's gaze trajectory matching and behavior matching indicators; The behavior matching index in step 3 specifically includes two parts: the viewing angle index and the gaze detection index; The viewing angle indicators specifically include: (1) The average angular velocity, maximum angular velocity, and maximum angular acceleration of eye movements during the test; (2) the maximum peak angular velocity and average angular velocity of eye movements during saccades and gazes; (3) average glance duration and total glance duration; The gaze detection indicators specifically include: (1) Two-dimensional correlation, average distance, and root mean square error of fixation points; (2) Successful fixation rate: The successful fixation rate is the ratio of successful fixations to the number of all stimulus points. A successful fixation means that within a fixed time after the stimulus point appears, the user's fixation point is detected inside a circle with a radius of r and a center of the stimulus point. (3) The maximum value of the correlation coefficient in the cross-correlation between the gaze point trajectory and the stimulus point trajectory is calculated on the Cartesian coordinate system xy plane as follows: Among them, P is the user's gaze point trajectory, G is the stimulus point trajectory, k (k ≥ 0) is the lag time, is the original correlation, n is the maximum value of sequence X and sequence Y, Y i The complex conjugate of The correlation estimation sequence is as follows: in, is the normalization coefficient; n T is the number of frames within the maximum delay threshold; (4) The ratio of the number of detected user glances to the number of times the stimulus point changes.

2. A face recognition and authentication method based on sight line features according to claim 1, characterized in that: The randomly generated and displayed rapidly flashing stimulus points in step 1 include a fixed screen interval delay to generate continuous stimulus points, and the screen interval delay is the same as the reflex eye movement delay of a person.

3. A face recognition and authentication method based on sight line features according to claim 2, characterized in that: Based on the Fick-gimbal visual angle model, the eye movement angle between two consecutive stimulation points is [R min ,R max ] is uniformly distributed, R min is the minimum scanning angle that the camera can detect, R max It is the maximum viewing angle based on the constrained head movement of the viewing model.

4. A face recognition and authentication method based on sight line features according to claim 1, characterized in that: The processing of the captured user facial video in step 2 specifically includes: Step 2.1: Use a convolutional neural network to detect faces and facial landmarks. The first layer of the convolutional neural network extracts facial features, and the last fully connected layer concatenates the face bounding box and facial landmarks, selects the face with the largest area, and obtains its facial landmarks. Step 2.2: Crop the face and the left and right eyes of the facial landmarks described in step 2.1 from the original user face video and scale them to a fixed resolution; Step 2.3: Use the cropped face and the left and right eyes from the facial landmarks in step 2.2 as input and feed them into the gaze detection network model. Then, use the user-customized calibration optimization model to obtain accurate eye movement tracks.

5. A face recognition and authentication method based on sight line features according to claim 4, characterized in that: The step 2.3 specifically includes: The gaze detection network model shares weights for the left and right eye convolution layers, uses a fully connected layer to connect the convolution layers of the left and right eye images and the convolution layer of the face image, and this fully connected layer and the feature points are input to the last layer, which outputs the two-dimensional screen coordinates of the gaze point; the eye movement trajectory is composed of the position of the two-dimensional screen coordinates of the gaze point output by the last layer of the gaze detection network model, which is the position coordinate of the Cartesian coordinate system in the xy plane; During the training phase, the gaze calibration model sets the weights of training samples according to the Euclidean distance between the stimulus point and the center of the screen; and calibrates the eye movement trajectory through the gaze calibration model to obtain an accurate eye movement trajectory.

6. A face recognition and authentication method based on sight line features according to claim 1, characterized in that: During the viewing angle indicator detection process, the scanning and gazing process specifically includes: Step 3.1: Convert the recorded user eye movement trajectory into a view sequence based on the Fick-gimbal model; Step 3.2: Detect the converted user eye movement trajectory, including filtering, adaptive scanning and gaze detection.

7. A face recognition and authentication method based on sight line features according to claim 6, characterized in that: The adaptive glance and gaze detection specifically includes: Adaptive saccade and gaze detection consists of saccade threshold calculation and peak search. The saccade threshold is determined based on the average and variance of the eye angular velocity within a fixed time. The minimum number of peak samples considered as a saccade is determined based on the frame rate and eye movement speed of the smartphone camera. If a peak is detected within a fixed time frame exceeding the saccade threshold and the time frame exceeds the minimum number of peak samples, it is detected as a user saccade; if multiple peaks are detected, it is detected as invalid eye movement; the remaining time frames below the saccade threshold are detected as user gaze.

Images