Face liveness detection method, device and system

Abstract

The application provides a face living body detection method, device and system, and relates to the field of face detection. The method comprises: extracting time sequence features of a video frame sequence by using a first feature extraction model to obtain time sequence visual features, extracting time sequence features of an IMU sensor data sequence by using a second feature extraction model to obtain time sequence sensor features, and fusing the time sequence visual features and the time sequence sensor features to obtain fused features; determining a first movement feature of a to-be-detected face in a first direction based on the video frame sequence, determining a second movement feature of the to-be-detected face in a second direction based on the IMU sensor data sequence, and determining a synchronization confidence of video and sensor data based on the first movement feature and the second movement feature; and inputting the fused features and the synchronization confidence into a joint classification model to obtain a near-far movement detection result and a fraud detection result. The application is used to solve the problems of low detection accuracy and the risk of fraud missing in the prior art.

Description

Technical Field

[0001] This invention relates to the field of face detection technology, specifically to a face liveness detection method, face liveness detection device, face liveness detection system, machine-readable storage medium, and computer program product. Background Technology

[0002] With the development of artificial intelligence and computer vision technologies, facial recognition, as a contactless and highly efficient method of identity verification, has been widely applied in fields such as financial payments, smart access control, attendance tracking, and public security. However, with the popularization of the technology, fake face attacks are frequent, and traditional liveness detection technologies face severe security challenges. In existing technologies, most facial liveness detection solutions rely on single-modal data (such as single visual modality data or single sensor data) for detection, resulting in low detection accuracy. Furthermore, existing facial liveness detection solutions also carry the risk of failing to detect fraud. Summary of the Invention

[0003] The purpose of this invention is to provide a face liveness detection method, apparatus, and system to solve the problems of low detection accuracy and risk of missed fraud detection in existing face liveness detection schemes.

[0004] To achieve the above objectives, embodiments of the present invention provide a face liveness detection method, comprising: Acquire the video frame sequence and IMU sensor data sequence of the face under test, which are synchronously captured. The video frame sequence is subjected to temporal feature extraction using a first feature extraction model to obtain temporal visual features. The IMU sensor data sequence is subjected to temporal feature extraction using a second feature extraction model to obtain temporal sensing features. The temporal visual features and the temporal sensing features are then fused to obtain fused features. Based on the video frame sequence, a first motion feature of the face under test in a first direction is determined; based on the IMU sensor data sequence, a second motion feature of the face under test in a second direction is determined; and based on the first motion feature and the second motion feature, the synchronization confidence of the video and sensor data is determined. The fused features and the synchronization confidence are input into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model.

[0005] Optionally, the step of fusing the temporal visual features and the temporal sensing features to obtain fused features includes: The temporal visual features and the temporal sensing features are globally stitched together and fused to obtain the first sub-fusion feature; The temporal visual features and the temporal sensing features are fused by attention on a video frame-by-frame basis to obtain the second sub-fusion feature; The temporal visual features and the temporal sensing features are divided into feature blocks and attention fused in video frames to obtain the third sub-fusion feature; The first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature are fused together to obtain the fused feature.

[0006] Optionally, the step of fusing the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature to obtain the fused feature includes: The fusion feature is obtained by weighted summation of the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature.

[0007] Optionally, determining a first motion feature of the face under test in a first direction based on the video frame sequence, determining a second motion feature of the face under test in a second direction based on the IMU sensor data sequence, and determining the synchronization confidence of the video and sensor data based on the first motion feature and the second motion feature include: The acceleration sequence of face bounding box size change of the test face is determined based on the video frame sequence; the acceleration sequence of face bounding box size change includes the acceleration of face bounding box size change in each frame of the video frame sequence; the acceleration of face bounding box size change characterizes the movement feature of the test face in the near and far directions; the acceleration sequence of face bounding box size change serves as the first movement feature; The z-axis motion acceleration sequence of the face under test is determined based on the IMU sensor data sequence; the z-axis motion acceleration sequence includes the z-axis motion acceleration corresponding to each frame of the video frame sequence; the z-axis motion acceleration characterizes the movement feature of the face under test in the z-axis direction perpendicular to the near-far direction; the z-axis motion acceleration sequence serves as the second movement feature; Based on the correlation between the first motion feature and the second motion feature, the synchronization confidence level of the video and sensor data is determined.

[0008] Optionally, the joint classification model includes a near-far movement detection model and a fraud detection model; The step of inputting the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model includes: The fused features are input into the near-far motion detection model to obtain the near-far motion detection results of the test face output by the near-far motion detection model; the near-far motion detection results include the confidence scores for moving away, moving closer, and non-near-far motion actions. The synchronized confidence level is fed into the fraud detection model to obtain the fraud confidence level of the test face output by the fraud detection model; the fraud confidence level is used as the fraud detection result.

[0009] Optionally, the step of transmitting the synchronized confidence level to the fraud detection model to obtain the fraud confidence level of the test face output by the fraud detection model includes: Based on the synchronization confidence level, the fraud confidence level of the face under test is determined; the fraud confidence level is negatively correlated with the synchronization confidence level.

[0010] Optionally, after inputting the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model, the method further includes: Data integrity analysis is performed on the video frame sequence and the IMU sensor data sequence to obtain the data integrity analysis results of the video frame sequence and the IMU sensor data sequence; If the data integrity analysis results indicate that both the video frame sequence and the IMU sensor data sequence are complete, the confidence scores for the pulling-away action, the approach action, the non-far-near movement action, and the spoofing confidence score are output. If the data integrity analysis results indicate that either the video frame sequence or the IMU sensor data sequence is incomplete, the system outputs the confidence level of the pulling-away action, the confidence level of the approaching action, and a data missing information; the data missing information includes a video frame missing information or a sensor data missing information. If the data integrity analysis results indicate that both the video frame sequence and the IMU sensor data sequence are incomplete, a system anomaly warning will be output.

[0011] Optionally, the near-far movement detection model is trained through the following steps: Acquire the sample video frame sequence and sample IMU sensor data sequence simultaneously acquired from the sample face; Repeat the following steps until the set stopping condition is met: The sample video frame sequence is subjected to temporal feature extraction using a first feature extraction model to obtain sample temporal visual features. The sample IMU sensor data sequence is subjected to temporal feature extraction using a second feature extraction model to obtain sample temporal sensing features. The sample temporal visual features and the sample temporal sensing features are then fused to obtain sample fused features. The sample fusion features are input into the near-far motion detection model to obtain the near-far motion prediction results output by the near-far motion detection model. Based on the near and far motion prediction results and the near and far motion ground truth labels of the sample video frame sequences, the mean squared error loss and cross-entropy loss are determined. The comprehensive loss function is determined based on the mean squared error loss and the cross-entropy loss; The model parameters of the near and far motion detection model are adjusted based on the comprehensive loss function.

[0012] On the other hand, embodiments of the present invention also provide a face liveness detection device, comprising: The acquisition module is used to acquire the video frame sequence and IMU sensor data sequence of the face under test being synchronously captured; The first extraction module is used to extract temporal features from the video frame sequence using a first feature extraction model to obtain temporal visual features, extract temporal features from the IMU sensor data sequence using a second feature extraction model to obtain temporal sensing features, and fuse the temporal visual features and the temporal sensing features to obtain fused features. The second extraction module is used to determine the first motion feature of the face under test in a first direction based on the video frame sequence, to determine the second motion feature of the face under test in a second direction based on the IMU sensor data sequence, and to determine the synchronization confidence of the video and sensor data based on the first motion feature and the second motion feature. The prediction module is used to input the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model.

[0013] On the other hand, embodiments of the present invention also provide a face liveness detection system, comprising: The image acquisition module is used to acquire video frame sequences of the face to be tested; The sensor acquisition module is used to acquire IMU sensor data sequences of the face to be tested; The data processing module is used to acquire a video frame sequence and an IMU sensor data sequence synchronously captured from the face under test; to extract temporal features from the video frame sequence using a first feature extraction model to obtain temporal visual features; to extract temporal features from the IMU sensor data sequence using a second feature extraction model to obtain temporal sensing features; and to fuse the temporal visual features and the temporal sensing features to obtain fused features; to determine a first movement feature of the face under test in a first direction based on the video frame sequence; to determine a second movement feature of the face under test in a second direction based on the IMU sensor data sequence; and to determine the synchronization confidence of the video and sensor data based on the first movement feature and the second movement feature; and to input the fused features and the synchronization confidence into a joint classification model to obtain the near and far movement detection results and fraud detection results of the face under test output by the joint classification model.

[0014] Through the above technical solution, this embodiment of the invention extracts features from the video frame sequence and IMU sensor data sequence simultaneously acquired from the face under test to obtain fused features and synchronization confidence; then, based on the fused features and synchronization confidence, it performs face liveness detection to obtain the near / far movement detection result and fraud detection result of the face under test. Thus, this embodiment of the invention integrates video frame sequences and IMU sensor data sequences for face liveness detection, overcoming the shortcomings of existing solutions that rely on single-modal data for detection and improving detection accuracy. Furthermore, the fraud detection result obtained by this embodiment of the invention can achieve fraud detection of face liveness, solving the problem of missed fraud detection risks in existing solutions.

[0015] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is one of the flowcharts illustrating the face liveness detection method provided by the present invention; Figure 2 This is the second flowchart of the face liveness detection method provided by the present invention; Figure 3 This is a schematic diagram of the multimodal feature fusion process provided by the present invention; Figure 4 This is a flowchart illustrating the dual-task joint output provided by the present invention; Figure 5 This is a flowchart illustrating the hierarchical anomaly handling mechanism provided by the present invention; Figure 6 This is a flowchart illustrating the dynamic threshold adjustment mechanism provided by the present invention; Figure 7 This is a schematic diagram of the face liveness detection system provided by the present invention; Figure 8 This is a schematic diagram of the face liveness detection device provided by the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0018] Method Implementation Examples Please refer to Figure 1 This invention provides a face liveness detection method, comprising: Step 100: Obtain the video frame sequence and IMU sensor data sequence of the face to be tested, which are synchronously captured.

[0019] The face liveness detection method of this invention can be applied to a face liveness detection system, which includes a data processing module that executes the face liveness detection method of this invention. The data processing module acquires a video frame sequence and an IMU sensor data sequence synchronously acquired from the face to be tested. In one embodiment, the core input of the data processing module is a 50-frame video face stream (i.e., a video frame sequence, with an input size of [insert size here]) synchronously acquired from the face to be tested. W) and frame-level IMU sensor data (i.e., IMU sensor data sequence, input size is ...). The mathematical definition of the input video frame sequence (including the 3-axis angular velocity of the gyroscope and the 3-axis acceleration of the accelerometer) is as follows: ; in, Represents a video frame sequence. Where D is the length of the video frame sequence (fixed at 5), and D is the number of RGB channels (3). for This refers to the resolution of a single-frame face image. ; in, This represents the frame-level IMU sensor data input (IMU sensor data sequence), with the 6-dimensional features corresponding to the 3 axes of the gyroscope ( Angular velocity and accelerometer 3-axis ( Acceleration corresponds synchronously to the video frame sequence.

[0020] This invention also includes data standardization and synchronization verification preprocessing. The data processing module performs face cropping, size scaling, and pixel normalization on the video face stream (video frame sequence), outputting standardized data X of size A×3×512×512 (A=50). video The IMU sensor data is numerically normalized and dimensionally calibrated to output standardized data X of size A×6 (A=50). imu The synchronization of two data streams is verified based on frame timestamps. If the synchronization error is ≤1 frame, frame alignment is performed directly; if the synchronization error is >1 frame, a minor anomaly alert is triggered and invalid frames are removed, ensuring strict synchronization of the remaining data frames. This embodiment of the invention corrects data synchronization errors by converting the original data into standardized data that meets the model input requirements, ensuring the accuracy and compatibility of subsequent model processing.

[0021] Step 200: Use the first feature extraction model to extract temporal features from the video frame sequence to obtain temporal visual features; use the second feature extraction model to extract temporal features from the IMU sensor data sequence to obtain temporal sensing features; and fuse the temporal visual features and the temporal sensing features to obtain fused features.

[0022] Please refer to Figure 2 This invention designs four collaborative links, forming a complete multimodal face liveness detection system by clearly defining the location, composition, and accuracy of each link. The first link is a deep link for modal decoupling, association, and aggregation of heterogeneous data, while the latter three are auxiliary and synchronous links. The four links have clear divisions of labor and work collaboratively. The first link is a deep link for modal decoupling, association, and aggregation of heterogeneous data. This first link consists of a video understanding link and a temporal sensor link, which adopt a jointly trained decoupling-association-aggregation method to achieve deep fusion of temporal visual features and temporal sensor features, jointly completing the core tasks of face liveness detection and near / far distance regression.

[0023] The data processing module can use various temporal feature extraction models (3D ResNet, TCN model) to extract temporal features from video frame sequences. For example, in one embodiment, the data processing module can use a lightweight PrunedCompact 3D ResNet (PC-3D-ResNet) network (first feature extraction model) to extract temporal features from the input video face stream, outputting the temporal visual features of the video frame sequence. The feature extraction formula is as follows: ; in, For temporal visual features, 3D-ResNet-Mini() is the first feature extraction model. The number of feature channels indicates that this lightweight Pruned Compact 3D ResNet (PC-3D-ResNet) network can effectively capture visual features such as spatial size changes, facial proportion adjustments, and depth blur of the subject face during near and far movement. This lightweight Pruned Compact 3D ResNet (PC-3D-ResNet) network can be referred to as the video understanding link.

[0024] The data processing module also extracts temporal sensing features from the IMU sensor data sequence using a Temporal Convolutional Network (TCN) to capture the device's motion state. The feature extraction formula is as follows:

[0025] in, For time-series sensing features, denoted as the number of sensor feature channels. TCN() represents the second feature extraction model. Temporal convolutional networks can be referred to as temporal sensor chains.

[0026] The data processing module fuses temporal visual features and temporal sensing features to obtain fused features. In some embodiments, the data processing module can perform data fusion based on at least two of the following data fusion methods: block fusion, frame-level attention fusion, and frame-level block attention fusion. For example, in one embodiment, the data processing module can perform data fusion based on both block fusion and frame-level attention fusion methods, and then fuse the block fusion result and the frame-level attention fusion result through an optimized fusion function (weighted summation) to obtain fused features. Please refer to... Figure 3 In another embodiment, the data processing module can perform data fusion based on three data fusion methods: whole-block fusion, frame-level attention fusion, and frame-level block attention fusion. Then, the whole-block fusion result, frame-level attention fusion result, and frame-level block attention fusion result are combined using an optimized fusion function (weighted summation) to obtain fused features. Whole-block fusion refers to directly performing global stitching and fusion of temporal visual features and temporal sensor features. Frame-level attention fusion refers to introducing an attention mechanism to perform weighted fusion of temporal visual features and temporal sensor features for each frame. Frame-level block attention fusion refers to performing attention fusion after feature segmentation of the temporal visual features and temporal sensor features for each frame.

[0027] This invention achieves feature complementarity by decoupling, associating, and aggregating deep multimodal links in heterogeneous data formed by jointly training temporal visual features and temporal sensing features, thus providing a high-quality data foundation for subsequent face liveness detection.

[0028] Step 300: Determine the first motion feature of the face under test in the first direction based on the video frame sequence, determine the second motion feature of the face under test in the second direction based on the IMU sensor data sequence, and determine the synchronization confidence of the video and sensor data based on the first motion feature and the second motion feature.

[0029] For example, the data processing module can determine the first motion feature of the face under test in the near-far direction (first direction) based on the video frame sequence, and then determine the second motion feature of the face under test in the direction perpendicular to the near-far direction (second direction) based on the IMU sensor data sequence. Finally, based on the correlation between the first motion feature and the second motion feature, the synchronization confidence of the video and sensor data is determined.

[0030] For example, in one embodiment, the second link among the four collaborative links is a video-specific physical sensing path. This second link is independent of the first heterogeneous data modal decoupling-association-aggregation deep link, requiring no complex network model. It can directly calculate the acceleration of the face frame size change (first motion feature) in each frame at the physical logic layer based on the size change of the face frame in the video frame sequence. The acceleration of the face frame size change in each frame is calculated from the difference in the area of ​​the face frames in adjacent frames, used to intuitively reflect the rate change of face movement at near and far distances. The third link among the four collaborative links is an IMU sensor-specific physical sensing path. This third link is independent of the first multimodal link, requiring no complex network model. It directly extracts the X-axis acceleration or Z-axis acceleration (second motion feature) from the IMU sensor data at the physical logic layer. The X-axis acceleration or Z-axis acceleration is directly extracted from the frame-level IMU sensor data, used to intuitively reflect the X-axis or Z-axis motion state of the device as the face moves near and far. In this embodiment of the invention, it is assumed that the face movement is in the Y-axis direction. The fourth link in the four collaborative working links described above is a synchronization link. Data input uses the first motion feature from the second link (physical layer) and the second motion feature from the third link (physical layer) as common inputs, independent of the feature fusion process of the first link. This embodiment of the invention calculates the correlation between the first and second motion features to determine the synchronicity between the video face bounding box change and the device's X-axis or Z-axis movement, thereby determining the synchronization confidence level between the video and sensor data. The correlation between the first and second motion features can be calculated using cosine similarity or Pearson correlation coefficient. This embodiment of the invention verifies the temporal synchronization between visual motion and device motion by calculating the correlation between the first and second motion features. This link can effectively identify attacks where attackers forge single-modal motion data. This embodiment of the invention allows setting a synchronization confidence threshold of 0.2, which can be dynamically adjusted according to business needs. If the synchronization confidence level is less than or equal to 0.2, it indicates poor temporal synchronization between visual motion and device motion, posing a risk of face spoofing; conversely, if the synchronization confidence level is less than or equal to 0.2, it indicates good temporal synchronization between visual motion and device motion, posing no risk of face spoofing.

[0031] It should be noted that in some embodiments, steps 200 and 300 can be executed in parallel, thereby further improving the efficiency of face liveness detection.

[0032] Step 400: Input the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model.

[0033] In some embodiments, the joint classification model includes a near-far motion detection model and a fraud detection model; the step of inputting the fused features and the synchronization confidence into the joint classification model to obtain the near-far motion detection result and fraud detection result of the test face output by the joint classification model includes: inputting the fused features into the near-far motion detection model to obtain the near-far motion detection result of the test face output by the near-far motion detection model; the near-far motion detection result includes a moving-away action confidence score and a moving-close action confidence score; inputting the synchronization confidence score into the fraud detection model to obtain the fraud confidence score of the test face output by the fraud detection model; the fraud confidence score is used as the fraud detection result.

[0034] The near-far movement detection model is trained based on sample video frame sequences and ground truth labels for near-far movement obtained simultaneously from sample faces. The model can be trained using supervised training methods. The confidence score for a zooming-out action represents the probability of the user's face moving away, corresponding to the "zooming-out" category in the face distance regression task. The confidence score for a moving-in action represents the probability of the user's face moving closer, corresponding to the "moving-in" category in the same task. In other embodiments, the near-far movement detection results also include the confidence score for non-near-far movement actions. This confidence score represents the probability of a user performing a non-near-far movement action, corresponding to the "non-near-far movement" category. In this embodiment, thresholds can be set for the confidence scores for zooming-out, moving-in, and non-near-far movement actions to determine the category. Taking the confidence score for zooming-out actions as an example, the confidence threshold for zooming-out actions is 0.8. When the confidence score for zooming-out actions is 0.9, it means that the face is zooming out, as the confidence score of 0.9 is greater than or equal to the confidence threshold of 0.8. The classification of confidence scores for moving closer to the face and for non-near-far movement actions follows the same logic and will not be elaborated here.

[0035] A fraud detection model can be constructed based on the negative correlation between synchronization confidence and fraud confidence. The synchronization confidence is then applied to the fraud detection model to obtain the fraud confidence of the test face output by the fraud detection model, including: determining the fraud confidence of the test face based on the synchronization confidence; the fraud confidence is negatively correlated with the synchronization confidence. For example, in one embodiment, fraud confidence can be set to 1 - synchronization confidence. Fraud confidence represents the probability that the current input is a fake face attack, corresponding to the "fraud" category in the face liveness binary classification task, i.e. ( When the confidence threshold for fraud is reached, the current input is determined to be a fake face attack.

[0036] In one embodiment, please refer to Figure 4This invention simultaneously achieves the dual tasks of face liveness binary classification and face distance regression by link merging and integration. It performs joint inference by combining the fusion features of the first link with the synchronous confidence of the fourth link (generated jointly by the second and third links), and finally outputs a four-dimensional confidence vector [a,b,c,d]. The inference formula is as follows: ; in, ( ) represents the joint classifier, which outputs the probabilities of each confidence level through the softmax function, satisfying the following constraints: . Indicates fusion features, This represents the synchronization confidence level. The four-dimensional confidence vector [a, b, c, d] has the following meanings for each dimension: a is the confidence level for the zooming action, representing the probability of the user's face zooming away, corresponding to the "zooming away action" category in the face distance regression task. b is the confidence level for the moving action, representing the probability of the user's face moving closer, corresponding to the "moving closer action" category in the face distance regression task; c is the confidence level for other actions (non-distance movement action confidence level), representing the probability of the user's non-distance movement action, corresponding to the "non-distance movement action" category. Fraud confidence represents the probability that the current input is a fake face attack, corresponding to the "fraud" category in the face liveness binary classification task.

[0037] This invention extracts features from a video frame sequence and an IMU sensor data sequence simultaneously acquired from the face under test, obtaining fused features and synchronization confidence. Then, based on the fused features and synchronization confidence, face liveness detection is performed to obtain the near / far movement detection result and fraud detection result of the face under test. Therefore, this invention integrates video frame sequences and IMU sensor data sequences for face liveness detection, overcoming the shortcomings of existing solutions that rely on single-modal data for detection and improving detection accuracy. Furthermore, the fraud detection results obtained by this invention can achieve fraud detection of face liveness, solving the problem of missed fraud detection risks in existing solutions.

[0038] In other aspects of the embodiments of the present invention, step 200, which involves fusing the temporal visual features and the temporal sensing features to obtain fused features, includes: globally stitching and fusing the temporal visual features and the temporal sensing features to obtain a first sub-fused feature; performing attention fusion on a video frame basis on the temporal visual features and the temporal sensing features to obtain a second sub-fused feature; performing feature segmentation and attention fusion on a video frame basis on the temporal visual features and the temporal sensing features to obtain a third sub-fused feature; and fusing the first sub-fused feature, the second sub-fused feature, and the third sub-fused feature to obtain the fused feature.

[0039] This invention utilizes three fusion strategies: block fusion, frame-level attention fusion, and frame-level block attention fusion. The weight allocation of these three fusion methods is dynamically adjusted by optimizing the fusion function. Block fusion directly performs global stitching and fusion of temporal visual features and temporal sensor features, as shown in the following formula: ; in, This is the first sub-fusion feature after the entire block is merged. () represents the feature concatenation function, which can be implemented using the concat function. For temporal visual features, It is a time-series sensing feature. The number of feature channels, This represents the number of sensor feature channels. This method achieves an accuracy of [missing information]. The requirements for simultaneous changes in dual modes are relatively low, which may lead to fraud and missed detection.

[0040] Frame-level attention fusion introduces an attention mechanism to perform weighted fusion of temporal visual features and temporal sensor features for each frame, as shown in the following formula: ; ; in, This is the second sub-fusion feature after frame-level attention fusion. The attention weights are used to dynamically allocate the fusion weights of temporal visual features and temporal sensor features. `Attention()` is the attention calculation function, and `Softmax()` is the normalization function. This method achieves an accuracy of [insert accuracy here]. It has high requirements for synchronization and a high fraud recall rate.

[0041] Frame-level block attention fusion fuses the temporal visual features and temporal sensor features of each frame after feature segmentation, as shown in the following formula:

[0042] in, This is the third sub-fusion feature after frame-level block attention fusion. The number of feature blocks, For the first Attention weights for block features These are the first two time-series visual features and the second time-series sensing features, respectively. Block features, this method achieves an accuracy rate of 100%. It adapts to specific scenario integration needs.

[0043] In some embodiments, the step of fusing the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature to obtain a fused feature includes: performing a weighted summation based on the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature to obtain the fused feature.

[0044] To address the performance trade-offs of existing fusion methods, this invention optimizes the cross-modal fusion strategy of this link by comparing the advantages and disadvantages of the three schemes mentioned above. This achieves deep interactive alignment of features between the two core sub-links, solving the problems of traditional fusion methods either missing fraud detection or having insufficient accuracy. The feature formula after fusion is as follows: ; in, This represents the number of feature channels after fusion. (Model Calibration & Refinement) optimizes the fusion function, for example, in one embodiment, Used to dynamically adjust the weight allocation of the three fusion methods, where a1, a2, and a3 are weight coefficients. To optimize the fused features, in other embodiments, the first sub-fused feature, the second sub-fused feature, and the third sub-fused feature can be fused using SoftAttention to obtain the fused feature.

[0045] This invention implements three fusion strategies—block fusion, frame-level attention fusion, and frame-level block attention fusion—based on temporal visual and temporal sensor features. By optimizing the fusion function and dynamically adjusting the weight distribution of these three fusion methods, it achieves deep fusion of visual and sensor features, resolving the performance trade-off problem of low accuracy or fraudulent missed detection in traditional fusion methods. The resulting fusion features possess high discriminative power and robustness. Addressing the performance trade-off problem of existing fusion methods, this invention optimizes the cross-modal fusion strategy by comparing the advantages and disadvantages of block fusion, frame-level attention fusion, and frame-level block attention fusion, achieving deep interactive alignment between visual and sensor features.

[0046] In other aspects of the embodiments of the present invention, the step of determining a first motion feature of the face under test in a first direction based on the video frame sequence, determining a second motion feature of the face under test in a second direction based on the IMU sensor data sequence, and determining the synchronization confidence of the video and sensor data based on the first motion feature and the second motion feature includes: determining a face bounding box size change acceleration sequence of the face under test based on the video frame sequence; the face bounding box size change acceleration sequence includes the face bounding box size change acceleration of each frame in the video frame sequence; the face bounding box size change acceleration characterizes the face under test... The method measures the movement features of a face in the near and far directions; the acceleration sequence of the face bounding box size change is used as the first movement feature; the z-axis motion acceleration sequence of the face under test is determined based on the IMU sensor data sequence; the z-axis motion acceleration sequence includes the z-axis motion acceleration corresponding to each frame of the video frame sequence; the z-axis motion acceleration characterizes the movement features of the face under test in the z-axis direction perpendicular to the near and far directions; the z-axis motion acceleration sequence is used as the second movement feature; the synchronization confidence of the video and sensor data is determined based on the correlation between the first movement feature and the second movement feature.

[0047] Specifically, in this embodiment of the invention, the second link directly calculates the acceleration of face bounding box size change at the physical logic layer based on the size change of the face bounding box in the video frame sequence. If the input is a 32-frame video sequence, the output is a list of face bounding box size change accelerations containing 32 elements (i.e., a face bounding box size change acceleration sequence, the first motion feature), used to characterize the rate of change of face bounding box size with the frame sequence. The mathematical definition is as follows: ; in, For the first The acceleration of the face bounding box size change corresponding to a frame is calculated from the difference in the area of ​​the face bounding boxes in adjacent frames. This is used to visually reflect the rate of change of the measured face's movement in the near-far direction (i.e., the first direction). The independent detection accuracy of this link can reach [percentage missing]. .

[0048] The third link directly extracts the Z-axis (i.e., the second direction) acceleration from the IMU sensor data at the physical logic layer. If the input is 32 frames of synchronized IMU sensor data, the output is a 32-element array. The z-axis acceleration list (i.e., the z-axis motion acceleration sequence, the second movement feature) is used to characterize the device's motion. The acceleration in the axial direction is mathematically defined as follows: ; in, For the first The Z-axis acceleration of the IMU sensor data corresponding to the frame is directly extracted from the frame-level IMU sensor data to intuitively reflect the Z-axis motion state of the device as the face moves near or far. The independent detection accuracy of this link can reach 85.2%.

[0049] In this embodiment of the invention, the fourth link (synchronization link) is jointly generated by the second link (physical layer) and the third link (physical layer), and is independent of the feature fusion process of the first multimodal link (i.e., the first link). It directly inputs the 32-frame acceleration list output by the above two physical layer links. and By calculating the correlation between the two lists, the synchronization between the changes in the video face bounding box and the device's Z-axis motion is determined, and the core confidence index is output: The confidence level of synchronization can be calculated using the cosine similarity formula as follows:

[0050] in, The synchronization confidence level of video and sensor data is used to quantify the correlation between the acceleration lists of two physical layer links, thereby verifying the temporal synchronization between visual motion and device motion. This link can effectively identify attacks where attackers forge single-modal motion data. The synchronization confidence threshold is tentatively set at 0.2, but can be dynamically adjusted according to business needs. When the synchronization confidence level is less than or equal to 0.2, it indicates poor temporal synchronization between visual motion and device motion, and there is a high probability that the tested face is a forged face. In other embodiments, It can also be characterized by calculating the Pearson correlation coefficient between the first and second movement features.

[0051] Existing single-visual modal solutions suffer from insufficient robustness. Traditional pure-visual solutions rely solely on the appearance and texture features of static images or video streams, making it difficult to effectively distinguish between real and fake faces. Malicious attackers can bypass detection mechanisms by simulating the visual features of real faces through low-cost methods such as photo re-capture, screen playback, and 3D mask forgery. Especially in scenarios involving facial movement (such as access control personnel approaching or face adjustments in payment scenarios), attackers can simulate changes in the size of real faces by moving a forged face carrier (such as a photo or screen). Single-visual modal solutions cannot distinguish the authenticity of this movement, leading to a significant increase in the success rate of fake attacks. Furthermore, pure-visual solutions are also affected by environmental factors such as changes in lighting and shooting angles, resulting in insufficient detection stability. Existing single-sensor modal solutions suffer from the pseudo-proposition of insufficient accuracy. Existing solutions rely solely on sensor data such as gyroscopes and accelerometers for detection, failing to incorporate facial visual features. They are susceptible to device jitter and environmental interference, unable to accurately distinguish between genuine user movement and random device jitter, and their detection accuracy is insufficient to meet actual business needs.

[0052] The present invention, through the above technical solution, solves the problems of insufficient detection accuracy, weak anti-attack capability, and insufficient robustness of existing technologies in liveness detection under near and far face movement scenarios, and achieves the following significant technical effects: 1. Detection accuracy has been greatly improved. By employing multimodal feature fusion of the core link (the first link), auxiliary judgment from two additional physical layer links (the second and third links), and a synchronous verification mechanism based on the fourth synchronous link (an additional link acceleration list), the overall detection accuracy of this invention reaches [a certain level]. Far higher than a single video link With a single sensor link The improvement in detection accuracy can be quantified using the following formula:

[0053] in, To improve the overall accuracy of the four-link modal decoupling-association-aggregation deep link (video understanding link + temporal sensor link), For the independent accuracy of the second link (physical layer), For the independent accuracy of the third link (physical layer), These represent the accuracy improvement percentages of the joint multimodal link compared to the second and third links, respectively. Furthermore, the fraud trigger rate in this embodiment of the invention is as low as 0.036%, effectively reducing the false positive and false negative rates of face liveness detection and meeting the accuracy requirements of high-security scenarios.

[0054] 2. Significantly enhanced resistance to attacks By introducing a second link (physical layer, accuracy 95.4%) and a third link (physical layer, accuracy 85.2%), and having these two links jointly generate a fourth synchronization link, this invention can accurately verify the synchronization between changes in the video face frame and the device's Z-axis movement. It effectively distinguishes between the near and far movement of a real face and the fake movement of a fake face, and can resist various common fake face attacks such as photo re-capture, screen playback, 3D mask forgery, and moving fake face carriers. It solves the security vulnerability that single-modal solutions are easily bypassed, and significantly improves the security of identity verification.

[0055] This invention reduces the risk of fraudulent detection by verifying the time synchronization between two modal data: the acceleration sequence of face frame size change and the acceleration sequence of z-axis motion, while meeting the requirements of high accuracy and high resistance to attacks.

[0056] In other aspects of the embodiments of the present invention, the near-far movement detection model is trained through the following steps: Acquire the sample video frame sequence and sample IMU sensor data sequence simultaneously acquired from the sample face; Repeat the following steps until the set stopping condition is met: Temporal features are extracted from the sample video frame sequence using a first feature extraction model to obtain sample temporal visual features. Temporal features are extracted from the sample IMU sensor data sequence using a second feature extraction model to obtain sample temporal sensing features. The sample temporal visual features and sample temporal sensing features are then fused to obtain sample fused features. The sample fused features are input into a near-far motion detection model to obtain the near-far motion prediction result output by the near-far motion detection model. Based on the near-far motion prediction result and the near-far motion ground truth labels of the sample video frame sequence, mean squared error loss and cross-entropy loss are determined. A comprehensive loss function is determined based on the mean squared error loss and the cross-entropy loss. The model parameters of the near-far motion detection model are adjusted based on the comprehensive loss function.

[0057] This invention embodiment can use a video understanding link (first feature extraction model), a temporal sensor link (second feature extraction model), and a joint classification model as an aggregated deep link model, trained using an end-to-end joint training method. The sample video frame sequences and sample IMU sensor data sequences can be real-world scene-collected bimodal datasets of near and far-moving faces (including video face streams and IMU sensor data sequences), incorporating attack samples such as photo re-enhancing, screen playback, and 3D mask forgery. This invention embodiment can set the training set:test set ratio to 8:2, with a dataset sample size ≥ 100,000, covering face samples of different ages, genders, skin colors, shooting angles, and lighting conditions. The optimizer for model training can be the AdamW optimizer, with a learning rate of 1e. -4 Weight decay = 1e -5Training rounds: 100 rounds, early stopping strategy (training stops if the validation set accuracy shows no improvement for 5 consecutive rounds). In each iteration, the data processing module uses a first feature extraction model to extract temporal features from the sample video frame sequence to obtain sample temporal visual features, uses a second feature extraction model to extract temporal features from the sample IMU sensor data sequence to obtain sample temporal sensing features, and fuses the sample temporal visual features and sample temporal sensing features to obtain sample fused features. The specific feature fusion process can be found in the above-mentioned feature fusion using three fusion strategies: block fusion, frame-level attention fusion, and frame-level block attention fusion, and the method of dynamically adjusting the weight allocation of the three fusion methods by optimizing the fusion function, which will not be elaborated here. The data processing module then inputs the sample fused features into the near-far motion detection model to obtain the near-far motion prediction results output by the near-far motion detection model; based on the near-far motion prediction results and the near-far motion ground truth labels of the sample video frame sequence, the mean squared error loss and cross-entropy loss are determined, and a comprehensive loss function is determined based on the mean squared error loss and the cross-entropy loss. In one embodiment, the comprehensive loss function is calculated as 0.7 * cross-entropy loss + 0.3 * mean squared error loss. This embodiment of the invention constructs a comprehensive loss function using cross-entropy loss and mean squared error loss for model training. This improves gradient diversity, facilitates a more comprehensive shaping of the feature space, enhances model generalization ability, and increases detection accuracy. The data processing module then uses gradient descent to adjust the model parameters of the near-far motion detection model (which may include a first feature extraction model and a second feature extraction model) based on the comprehensive loss function. Ultimately, the training effect of the near-far motion detection model in this embodiment is as follows: after joint training, the overall accuracy of the near-far motion detection model on the test set reaches 99.9%, meeting the accuracy requirements of high-security scenarios.

[0058] In other aspects of the embodiments of the present invention, after inputting the fused features and the synchronization confidence into the joint classification model to obtain the near-far movement detection result and fraud detection result of the test face output by the joint classification model, the method further includes: performing data integrity analysis on the video frame sequence and the IMU sensor data sequence to obtain the data integrity analysis result of the video frame sequence and the IMU sensor data sequence; if the data integrity analysis result indicates that both the video frame sequence and the IMU sensor data sequence are complete, outputting the confidence of the zooming-out action, the confidence of the approaching action, the confidence of the non-near-far movement action, and the fraud confidence; if the data integrity analysis result indicates that one of the data in the video frame sequence and the IMU sensor data sequence is incomplete, outputting the confidence of the zooming-out action, the confidence of the approaching action, and the data missing prompt; the data missing prompt includes a video frame missing prompt or a sensor data missing prompt; if the data integrity analysis result indicates that both the video frame sequence and the IMU sensor data sequence are incomplete, outputting a system anomaly prompt.

[0059] Please refer to Figure 5 To improve project usability, this embodiment of the invention also designs a tiered anomaly handling mechanism, providing differentiated outputs for different data loss scenarios. The data processing module first performs data integrity analysis on the video frame sequence and the IMU sensor data sequence to obtain the data integrity analysis results. The data integrity analysis is combined with a data integrity judgment function. This indicates that the data is complete. (Indicates missing data). For IMU sensor data sequences, the data integrity check function can use the CRC16 checksum method. For video frame sequences, the data integrity check function can use the RTP sequence number detection method.

[0060] The mathematical expression of the exception handling logic in this embodiment of the invention is as follows:

[0061] The specific tiered processing is as follows: 1. When both the video frame sequence and the IMU sensor data sequence are complete, a four-dimensional confidence vector is output normally. .

[0062] 2. When only video frame sequence is missing video data, output the confidence scores a for the distance movement and b for the proximity movement based on IMU sensor data, and mark the abnormal message "video missing".

[0063] 3. When only the IMU sensor data sequence is missing sensor data, output the confidence scores a for the distance movement and b for the proximity movement based on the video frame sequence, and mark the abnormal message "sensor missing".

[0064] 4. When both the video frame sequence and the IMU sensor data sequence are incomplete (missing), a system exception will be thrown.

[0065] Please refer to Figure 6 Furthermore, in this embodiment of the invention, key parameters such as synchronization confidence threshold and fraud confidence threshold can be set as adjustable environment variables. After deployment, the thresholds can be directly modified through environment variable configuration without redeploying the model, thus adapting to the security requirements of different business scenarios. The mathematical constraints for threshold adjustment are as follows: ; in, This is the synchronization confidence threshold (currently 0.2 by default). The fraud confidence threshold, modified via the environment variable export command, is applied to the inference process in real time without requiring a system restart or model retraining, significantly improving the flexibility of engineering deployment.

[0066] Existing solutions lack flexibility and fault tolerance in engineering deployment. Specifically, when multimodal data is missing, existing solutions often directly throw system anomalies without providing degraded handling results, exhibiting poor fault tolerance and difficulty in coping with complex real-world application environments. Furthermore, detection thresholds are mostly fixed parameters built into the model, which cannot be flexibly adjusted according to the security requirements of different business scenarios such as finance, security, and civilian applications after deployment, resulting in insufficient flexibility in engineering deployment.

[0067] The hierarchical anomaly handling mechanism of this invention can still provide degraded detection results when some modal data is missing, avoiding direct system failure and improving system fault tolerance in complex environments. Simultaneously, the dynamically adjustable threshold mechanism allows business users to flexibly adjust detection strategies according to the security requirements of different scenarios without retraining or deploying the model, significantly improving the flexibility of engineering deployment and enabling rapid adaptation to various application scenarios such as financial payments, smart access control, public security, and smart homes. This invention demonstrates strong scenario robustness and engineering availability. Furthermore, this invention is lightweight and easy to deploy, meeting real-time requirements on the edge. This invention adopts a lightweight network structure, with a small model size and fast inference speed, and can be deployed to mobile and edge devices (such as mobile phones, smart cameras, and facial recognition terminals) to achieve real-time inference, meeting the real-time requirements of edge devices and possessing excellent engineering deployment capabilities.

[0068] The technical solution of this invention has been verified through actual deployment and can run stably on mobile devices (phones), edge devices (smart cameras), and server devices, and has the following implementation effects: High detection accuracy: The overall face liveness detection accuracy reaches 99.9%, which is far higher than that of a single video link (95.4%) and a single sensor link (85.2%). The fraud trigger rate is as low as 0.036%, and the false positive rate and false negative rate are both <0.1%.

[0069] Strong anti-attack capability: It can effectively resist various common fake face attacks such as photo copying, screen playback, 3D mask forgery, and mobile fake face carriers, with an attack recognition success rate of ≥99.8%.

[0070] Good real-time performance: single frame data processing time ≤20ms, overall detection process response time ≤100ms, meeting the real-time inference requirements of the end side and edge side.

[0071] High fault tolerance and flexibility: It supports degradation processing when some data is missing, the threshold parameters can be flexibly adjusted, there is no need to retrain the model, and it can be quickly adapted to different business scenarios such as finance, security, and civilian use.

[0072] Lightweight and easy to deploy: It adopts a lightweight network structure with a model size of <200MB, which can be deployed on various mobile and edge devices without the need for high-performance hardware support, and has strong engineering implementation capabilities.

[0073] System Implementation Examples Please refer to Figure 7 On the other hand, embodiments of the present invention also provide a face liveness detection system, including: Image acquisition module 701 is used to acquire video frame sequences of the face to be tested; The sensor acquisition module 702 is used to acquire the IMU sensor data sequence of the face to be tested; The data processing module 703 is used to acquire a video frame sequence and an IMU sensor data sequence synchronously acquired from the face under test; to extract temporal features from the video frame sequence using a first feature extraction model to obtain temporal visual features; to extract temporal features from the IMU sensor data sequence using a second feature extraction model to obtain temporal sensing features; and to fuse the temporal visual features and the temporal sensing features to obtain fused features; to determine a first movement feature of the face under test in a first direction based on the video frame sequence; to determine a second movement feature of the face under test in a second direction based on the IMU sensor data sequence; and to determine the synchronization confidence of the video and sensor data based on the first movement feature and the second movement feature; and to input the fused features and the synchronization confidence into a joint classification model to obtain the near and far movement detection results and fraud detection results of the face under test output by the joint classification model.

[0074] Specifically, the face liveness detection system of this invention is used to execute the face liveness detection method of the above-described method embodiments. The face liveness detection system of this invention can be deployed on mobile devices (phones, tablets), edge devices (smart cameras, face recognition terminals), and servers. The core hardware consists of an image acquisition module 701, a sensor acquisition module 702, a data processing module 703, a storage module 704, and an output module 705. All hardware components are electrically connected, allowing for unidirectional / bidirectional data flow. The overall architecture is modular and can be flexibly adapted to different deployment scenarios. The specific structure, connection relationships, and functions are as follows: The structure of the image acquisition module 701 consists of a high-definition camera (resolution ≥1080P, frame rate ≥30fps), a face detection lens, and an image preprocessing chip, supporting continuous video frame acquisition and real-time face bounding box localization. The connection relationship of the image acquisition module 701 is as follows: it is electrically connected to the data processing module 703, transmitting the acquired video face stream to the data processing module 703 in real time, and simultaneously receiving the face bounding box localization command from the data processing module 703. The image acquisition module 701 is responsible for: synchronously acquiring 50 consecutive frames of video face stream, performing preprocessing such as denoising, white balance, and face cropping on the original video frames, and outputting standardized video face stream data with a size of A×3×512×512 (A=50), providing basic input for subsequent video understanding links.

[0075] The sensor acquisition module 702 consists of a six-axis IMU sensor (including a three-axis gyroscope and a three-axis accelerometer) and a sensor data sampling chip. The sampling frequency is synchronized with the frame rate of the image acquisition module 701 (30fps). The sensor acquisition module 702 is connected synchronously with the image acquisition module 701 and electrically with the data processing module 703, transmitting frame IMU sensor data to the data processing module 703 in real time. The sensor acquisition module 702 is responsible for synchronously acquiring 50 frames of frame-level IMU sensor data with the image acquisition module 701, capturing the changes in angular velocity and acceleration of the device in space, and outputting standardized sensor data with a size of A×6 (A=50). The 6-dimensional features correspond to the angular velocity of the gyroscope along the x / y / z axis and the acceleration of the accelerometer along the x / y / z axis, respectively, providing basic input for subsequent time-series sensor links.

[0076] The structure of the data processing module 703: It consists of an embedded processor (mobile / edge) or server CPU / GPU (server), and a neural network inference chip. It incorporates the four-link processing model, cross-modal fusion module, and joint inference module designed in this invention. The connection relationship of the data processing module 703: It is the core hardware of the system and is electrically connected to the image acquisition module 701, sensor acquisition module 702, storage module 704, and output module 705 respectively. It receives input data from the first two, transmits processing data to the storage module 704, and transmits the final detection result to the output module 705. The data processing module 703 is responsible for executing all the face liveness detection method logic of this invention, including four-link feature extraction, cross-modal feature fusion, synchronization verification, joint inference, anomaly handling, etc. It is the core unit for realizing liveness detection and distance regression, and supports real-time inference (single frame processing time ≤20ms).

[0077] The structure of storage module 704: It consists of flash memory (mobile / edge) or cloud storage (server), and is divided into temporary storage area and permanent storage area; The connection relationship of storage module 704: It is bidirectionally electrically connected to data processing module 703, receives temporary processing data and final result data from data processing module 703, and can feed back historical configuration parameters (such as threshold parameters) to data processing module 703. The storage module 704 serves two purposes: a temporary storage area for caching raw data acquired by images / sensors and intermediate feature data from each link, which is automatically cleared after processing; and a permanent storage area for storing system configuration parameters (such as synchronization confidence threshold and fraud confidence threshold) and detection result logs, supporting flexible reading and modification of parameters.

[0078] The structure of the output module 705 consists of a display screen (mobile / edge terminal) and a data interface (such as API interface, serial port, server / edge terminal); the connection relationship of the output module 705 is that it is electrically connected to the data processing module 703 and receives the final detection results and abnormal prompt information from the data processing module 703. The function of output module 705 is to output four-dimensional confidence vector, face distance regression results, and abnormal prompts (such as missing video or sensor) to users / business systems. It supports visualization and data interface docking, and realizes real-time feedback of detection results and linkage with business systems.

[0079] Device Examples Please refer to Figure 8 On the other hand, embodiments of the present invention also provide a face liveness detection device, comprising: The acquisition module 801 is used to acquire the video frame sequence and IMU sensor data sequence of the face to be tested being synchronously captured; The first extraction module 802 is used to extract temporal features from the video frame sequence using a first feature extraction model to obtain temporal visual features, extract temporal features from the IMU sensor data sequence using a second feature extraction model to obtain temporal sensing features, and fuse the temporal visual features and the temporal sensing features to obtain fused features. The second extraction module 803 is used to determine the first movement feature of the face under test in a first direction based on the video frame sequence, to determine the second movement feature of the face under test in a second direction based on the IMU sensor data sequence, and to determine the synchronization confidence of the video and sensor data based on the first movement feature and the second movement feature. The prediction module 804 is used to input the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model.

[0080] Optionally, the step of fusing the temporal visual features and the temporal sensing features to obtain fused features includes: The temporal visual features and the temporal sensing features are globally stitched together and fused to obtain the first sub-fusion feature; The temporal visual features and the temporal sensing features are fused by attention on a video frame-by-frame basis to obtain the second sub-fusion feature; The temporal visual features and the temporal sensing features are divided into feature blocks and attention fused in video frames to obtain the third sub-fusion feature; The first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature are fused together to obtain the fused feature.

[0081] Optionally, the step of fusing the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature to obtain the fused feature includes: The fusion feature is obtained by weighted summation of the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature.

[0082] Optionally, determining a first motion feature of the face under test in a first direction based on the video frame sequence, determining a second motion feature of the face under test in a second direction based on the IMU sensor data sequence, and determining the synchronization confidence of the video and sensor data based on the first motion feature and the second motion feature include: The acceleration sequence of face bounding box size change of the test face is determined based on the video frame sequence; the acceleration sequence of face bounding box size change includes the acceleration of face bounding box size change in each frame of the video frame sequence; the acceleration of face bounding box size change characterizes the movement feature of the test face in the near and far directions; the acceleration sequence of face bounding box size change serves as the first movement feature; The z-axis motion acceleration sequence of the face under test is determined based on the IMU sensor data sequence; the z-axis motion acceleration sequence includes the z-axis motion acceleration corresponding to each frame of the video frame sequence; the z-axis motion acceleration characterizes the movement feature of the face under test in the z-axis direction perpendicular to the near-far direction; the z-axis motion acceleration sequence serves as the second movement feature; Based on the correlation between the first motion feature and the second motion feature, the synchronization confidence level of the video and sensor data is determined.

[0083] Optionally, the joint classification model includes a near-far movement detection model and a fraud detection model; The step of inputting the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model includes: The fused features are input into the near-far motion detection model to obtain the near-far motion detection results of the test face output by the near-far motion detection model; the near-far motion detection results include the confidence scores for moving away, moving closer, and non-near-far motion actions. The synchronized confidence level is fed into the fraud detection model to obtain the fraud confidence level of the test face output by the fraud detection model; the fraud confidence level is used as the fraud detection result.

[0084] Optionally, the step of transmitting the synchronized confidence level to the fraud detection model to obtain the fraud confidence level of the test face output by the fraud detection model includes: Based on the synchronization confidence level, the fraud confidence level of the face under test is determined; the fraud confidence level is negatively correlated with the synchronization confidence level.

[0085] Optionally, following the device, the following further includes: The output module is used to perform data integrity analysis on the video frame sequence and the IMU sensor data sequence to obtain the data integrity analysis results. If the data integrity analysis results indicate that both the video frame sequence and the IMU sensor data sequence are complete, the module outputs the confidence scores for the distance movement, the proximity movement, the non-far-near movement, and the spoofing. If the data integrity analysis results indicate that one of the data sequences in the video frame sequence and the IMU sensor data sequence is incomplete, the module outputs the confidence scores for the distance movement, the proximity movement, and a data missing information. The data missing information includes either a video frame missing information or a sensor data missing information. If the data integrity analysis results indicate that both the video frame sequence and the IMU sensor data sequence are incomplete, the module outputs a system anomaly message.

[0086] Optionally, the near-far movement detection model is trained through the following steps: Acquire the sample video frame sequence and sample IMU sensor data sequence simultaneously acquired from the sample face; Repeat the following steps until the set stopping condition is met: The sample video frame sequence is subjected to temporal feature extraction using a first feature extraction model to obtain sample temporal visual features. The sample IMU sensor data sequence is subjected to temporal feature extraction using a second feature extraction model to obtain sample temporal sensing features. The sample temporal visual features and the sample temporal sensing features are then fused to obtain sample fused features. The sample fusion features are input into the near-far motion detection model to obtain the near-far motion prediction results output by the near-far motion detection model. Based on the near and far motion prediction results and the near and far motion ground truth labels of the sample video frame sequences, the mean squared error loss and cross-entropy loss are determined. The comprehensive loss function is determined based on the mean squared error loss and the cross-entropy loss; The model parameters of the near and far motion detection model are adjusted based on the comprehensive loss function.

[0087] The face liveness detection device includes a processor and a memory. The aforementioned acquisition module 801, first extraction module 802, second extraction module 803, and prediction module 804 are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. The processor contains a kernel, which retrieves the corresponding program units from the memory. One or more kernels may be provided. The memory may include non-permanent memory in computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.

[0088] In another embodiment, the face liveness detection device is developed based on the deep learning framework (PyTorch) and adopts a modular design, corresponding one-to-one with the hardware architecture. The core consists of a data input module, a four-link processing module, a cross-modal fusion module, a joint inference module, an anomaly handling module, a threshold configuration module, and a result output module. Each software module is connected to the others, following a logical flow of "input → processing → fusion → inference → output," with no redundant modules. The specific composition, connection relationships, and functions are as follows: Data input module Connection relationship: It connects to the image acquisition module and sensor acquisition module of the hardware, provides basic input for all subsequent software modules, and is the starting module of the system; Function: Receives raw video face streams and IMU sensor data sequences acquired by hardware, performs data standardization, synchronization verification, and dimension matching, and outputs standardized data (X) that meets the model input requirements. video ∈R A×3×512×512 X imu ∈R A×6 It also checks the frame synchronization of the two data streams. If the synchronization error is greater than 1 frame, it triggers a minor anomaly alert and performs frame alignment correction.

[0089] Four-link processing module The core processing module consists of four sub-modules: a deep link for heterogeneous data decoupling, association, and aggregation (the first link), a video-specific physical sensing path (the second link), a sensor-specific physical sensing path (the third link), and a synchronization link (the fourth link). Each sub-module processes data independently and interacts with some data. All sub-modules are connected to the data input module. The first three are processed in parallel, while the synchronization link is connected to the latter two.

[0090] Cross-modal fusion module Connection relationships: It is connected to the video understanding link (core), the temporal sensor link (core), and the joint inference module, and is a key module for core feature fusion; Function: It incorporates three fusion strategies: whole-block fusion, frame-level attention fusion, and frame-level block attention fusion. By optimizing the fusion function, it dynamically adjusts the weight distribution of the three strategies to achieve deep interactive alignment between visual features and sensor features, solving the problem of "accuracy versus anti-attack capability trade-off" in traditional fusion methods. Finally, it outputs the optimized fused feature F. fusion ∈R A×Cfusion This provides a core feature foundation for joint reasoning.

[0091] Joint reasoning module Connection relationships: It connects with the cross-modal fusion module, synchronization link, exception handling module, and result output module, and is the core of the system's reasoning and decision-making. Function: Built-in joint classifier, receives fused features F fusion and synchronization confidence S sync It performs joint inference of two tasks: face liveness binary classification and face distance regression. It outputs a four-dimensional confidence vector [a,b,c,d] that meets the constraints (a+b+c+d=1, 0≤a,b,c,d≤1) through the Softmax function, where a is the confidence of the action of moving away, b is the confidence of the action of moving closer, c is the confidence of other actions, and d is the confidence of fraud. The fraud trigger rate is as low as 0.036%.

[0092] Exception handling module Connection relationships: The data connection with the data input module, joint inference module, and result output module is a key module for improving the system's fault tolerance; Function: The built-in data integrity judgment function I(X) performs integrity checks on the raw data of the data input module, executes hierarchical anomaly handling logic based on the detection results, provides degradation detection results when some modal data is missing, avoids direct system failure, and only throws system anomalies when all video and sensor data are missing.

[0093] Threshold configuration module Connection relationships: It connects with the synchronization link, joint inference module, and storage module, and is a core module for improving the flexibility of system engineering; Function: To set the synchronization confidence threshold θ sync Fraud confidence threshold θ fraud Set as an adjustable environment variable, allowing users to modify the threshold parameter via configuration interface / command line. Modifications take effect immediately and are stored in the storage module, eliminating the need to retrain the model or redeploy the system. It is adaptable to security requirements in various business scenarios such as finance, security, and civilian applications. The threshold constraint is 0 < θ. sync <1、0<θ fraud <1.

[0094] Result Output Module Connection relationships: It connects with the joint inference module, exception handling module, and hardware output module, and is the final feedback module of the system; Function: Receives the four-dimensional confidence vector from the joint inference module and the anomaly prompts from the anomaly handling module, performs result standardization processing, and transmits the visualization results (such as mobile screen display) and standardized data interface results (such as server-side API output) to the hardware output module, thereby achieving real-time, multi-form feedback of detection results.

[0095] The following describes the execution steps of the face near / far liveness detection method based on multimodal fusion, focusing on the data input module, four-link processing module, cross-modal fusion module, joint inference module, anomaly handling module, threshold configuration module, and result output module. The face liveness detection method of this invention is implemented based on at least the above software architecture. The overall execution flow is as follows: data acquisition → data preprocessing → four-link feature extraction and synchronous verification → cross-modal feature fusion → joint inference → anomaly handling → result output, totaling seven core steps. Each step is interconnected, and key steps have been optimized to achieve high-precision, high-attack resistance, and high-fault-tolerance liveness detection results. The specific steps, their functions, and key implementation details are as follows: Step 1: Synchronously acquire video face stream and IMU sensor data Execution components: hardware image acquisition module and sensor acquisition module; software data input module; Function: To acquire basic bimodal data for face liveness detection, ensuring the time synchronization and validity of the data, and providing raw data for subsequent processing.

[0096] Key implementation details: Through a hardware synchronization triggering mechanism, the sampling frequency of the image acquisition module and the sensor acquisition module is kept consistent (30fps). 50 frames of continuous video face stream and frame-level IMU sensor data are acquired simultaneously. The camera focuses on the face area to ensure that the face accounts for ≥30% of the video frame. The IMU sensor is in close contact with the main body of the device to avoid sensor data distortion caused by sensor offset.

[0097] Step 2: Data Standardization and Synchronization Verification Preprocessing Execution entity: The software's data input module; Function: To convert raw data into standardized data that meets the requirements of model input, correct data synchronization errors, and ensure the accuracy and compatibility of subsequent model processing; Key implementation details: The video face stream is processed by face cropping, resizing, and pixel normalization, outputting standardized data X of size A×3×512×512 (A=50). video ; The IMU sensor data is numerically normalized and dimensionally calibrated to output standardized data X of size A×6 (A=50). imu ; The synchronization of the two data streams is verified based on the frame timestamp. If the synchronization error is ≤ 1 frame, frame alignment is performed directly. If the synchronization error is > 1 frame, a minor anomaly warning is triggered and invalid frames are removed to ensure that the remaining data frames are strictly synchronized.

[0098] Step 3: Parallel Feature Extraction and Synchronization Verification of Four Links Execution entity: The four-link processing module of the software, which is one of the core steps of the method; Function: Simultaneously extract core deep learning features and simple physical layer features to complete the synchronization verification of video and sensor data, and realize feature complementarity and preliminary identification of attack behavior.

[0099] Step 4: Cross-modal feature optimization and fusion The execution entity is the cross-modal fusion module of the software, which is the core and key step of the method. Function: To achieve deep fusion of visual features and sensor features, solve the performance trade-off problem of "low accuracy or fraudulent missed detection" in traditional fusion methods, and output fused features with high recognition and high robustness. Key implementation details: Three fusion strategies are implemented respectively: block fusion is achieved by splicing F blocks together. video and F imu Get F fusion1 Frame-level attention fusion obtains F through dynamic weighting via an attention mechanism. fusion2 Frame-level block attention fusion is performed by weighted concatenation after block segmentation to obtain F. fusion3 ; By optimizing the fusion function and assigning weights to the three fusion features, high weights are given to features that contribute significantly to accuracy and features that contribute significantly to fraud detection. The final output is the optimized fusion feature F. fusion It balances detection accuracy with anti-attack capabilities.

[0100] Step 5: Dual-task joint reasoning Execution entity: The software's joint reasoning module, which is the core step in the method's reasoning and decision-making process; Function: Based on fused features and synchronous confidence, it simultaneously completes the dual tasks of liveness binary classification and face distance regression, outputting accurate confidence results, and realizing the recognition of real face distance movement and the detection of fake face attacks. Key implementation details: The joint classifier receives F fusion and S sync Feature fusion and inference are performed using deep learning networks; The Softmax function outputs a four-dimensional confidence vector [a,b,c,d] that satisfies the constraint a+b+c+d=1, where d is the fraud confidence level. If d≥θ fraud The attack was identified as a fake face attack, and the remaining dimensions were used to perform near and far distance regression of the face.

[0101] Step 6: Graded Anomaly Handling Execution Entity: The software's exception handling module, which is a key step in designing a highly fault-tolerant method; Function: To provide differentiated processing results for scenarios with missing data, improve the fault tolerance and availability of the system in complex real-world application environments, and prevent the system from failing directly due to missing data.

[0102] Step 7: Standardize the output results Execution entities: software output module, hardware output module; Function: To convert the detection results into a form that can be recognized by users / business systems, enabling real-time feedback and business linkage, and completing the entire liveness detection process; Key implementation details: The joint inference results and anomaly alerts are standardized. The mobile / edge terminals display the confidence results, near and far distance regression results, and anomaly alerts on the screen. The server / edge terminals output standardized data through API interfaces, serial ports, and other data interfaces, supporting seamless integration with financial payment, smart access control, and other business systems. The response time for detection results is ≤100ms.

[0103] The core parameters are configured as follows: The core parameters of this invention are all configurable. The default configuration is suitable for most high-security scenarios. The specific default parameters are as follows: video frame sequence length A=50, physical layer link output acceleration list length=32; synchronization confidence threshold θ sync =0.2, fraud confidence threshold θ fraud =0.5; Normalized resolution of video face stream = 512×512; IMU sensor data dimension = 6; Number of feature channels in 3D-ResNet-Mini network C video =128, Number of feature channels C in TCN network imu =64, Number of feature channels C after fusion fusion =256.

[0104] The details of joint model training are as follows: The core heterogeneous data modal decoupling-association-aggregation deep link model of this invention (video understanding link, temporal sensor link, joint classifier) ​​adopts an end-to-end joint training method. The training dataset is a real-scene collected face near and far movement dual-modal dataset (including video face stream and IMU sensor data), and attack samples such as photo flipping, screen playback, and 3D mask forgery are added. The training details are as follows: Training set: test set = 8:2, dataset sample size ≥ 100,000, covering face samples of different ages, genders, skin colors, shooting angles, and lighting conditions; Optimizer: AdamW optimizer, learning rate = 1e -4 Weight decay = 1e -5 ; Loss function: The joint loss function is adopted, which is cross-entropy loss + mean squared error loss (near and far distance regression), with a weight ratio of 7:3; Training rounds: 100 rounds, early stop strategy (training stops if the validation set accuracy does not improve for 5 consecutive rounds); Training results: After joint training, the model achieved an overall accuracy of 99.9% on the test set, meeting the accuracy requirements for high-security scenarios.

[0105] The above content has detailed the system hardware architecture, software module composition, connection relationships and functions of each module, and method execution steps of the present invention, and explained the design principles and implementation effects of key steps, enabling those skilled in the art to completely reproduce the technical solution of the present invention. The present invention is a face liveness detection method, device, and system based on cross-modal fusion and synchronization of asynchronous video face streams and frame-level IMU sensor data. It belongs to the field of computer vision and sensor fusion technology and is applicable to high-security scenarios requiring face liveness detection, such as financial payments, smart access control, and public security.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting human face liveness, characterized in that, include: Acquire the video frame sequence and IMU sensor data sequence of the face under test, which are synchronously captured. The video frame sequence is subjected to temporal feature extraction using a first feature extraction model to obtain temporal visual features. The IMU sensor data sequence is subjected to temporal feature extraction using a second feature extraction model to obtain temporal sensing features. The temporal visual features and the temporal sensing features are then fused to obtain fused features. Based on the video frame sequence, a first motion feature of the face under test in a first direction is determined; based on the IMU sensor data sequence, a second motion feature of the face under test in a second direction is determined; and based on the first motion feature and the second motion feature, the synchronization confidence of the video and sensor data is determined. The fused features and the synchronization confidence are input into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model.

2. The face liveness detection method according to claim 1, characterized in that, The step of fusing the temporal visual features and the temporal sensing features to obtain fused features includes: The temporal visual features and the temporal sensing features are globally stitched together and fused to obtain the first sub-fusion feature; The temporal visual features and the temporal sensing features are fused by attention on a video frame-by-frame basis to obtain the second sub-fusion feature; The temporal visual features and the temporal sensing features are divided into feature blocks and attention fused in video frames to obtain the third sub-fusion feature; The first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature are fused together to obtain the fused feature.

3. The face liveness detection method according to claim 2, characterized in that, The step of fusing the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature to obtain the fused feature includes: The fusion feature is obtained by weighted summation of the first sub-fusion feature, the second sub-fusion feature, and the third sub-fusion feature.

4. The face liveness detection method according to claim 1, characterized in that, The process of determining a first motion feature of the face under test in a first direction based on the video frame sequence, determining a second motion feature of the face under test in a second direction based on the IMU sensor data sequence, and determining the synchronization confidence of the video and sensor data based on the first motion feature and the second motion feature includes: The acceleration sequence of face bounding box size change of the test face is determined based on the video frame sequence; the acceleration sequence of face bounding box size change includes the acceleration of face bounding box size change in each frame of the video frame sequence; the acceleration of face bounding box size change characterizes the movement feature of the test face in the near and far directions; the acceleration sequence of face bounding box size change serves as the first movement feature; The z-axis motion acceleration sequence of the face under test is determined based on the IMU sensor data sequence; the z-axis motion acceleration sequence includes the z-axis motion acceleration corresponding to each frame of the video frame sequence; the z-axis motion acceleration characterizes the movement feature of the face under test in the z-axis direction perpendicular to the near-far direction; the z-axis motion acceleration sequence serves as the second movement feature; Based on the correlation between the first motion feature and the second motion feature, the synchronization confidence level of the video and sensor data is determined.

5. The face liveness detection method according to claim 1, characterized in that, The joint classification model includes a near-far movement detection model and a fraud detection model; The step of inputting the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model includes: The fused features are input into the near-far motion detection model to obtain the near-far motion detection results of the test face output by the near-far motion detection model; the near-far motion detection results include the confidence scores for moving away, moving closer, and non-near-far motion actions. The synchronized confidence level is fed into the fraud detection model to obtain the fraud confidence level of the test face output by the fraud detection model; the fraud confidence level is used as the fraud detection result.

6. The face liveness detection method according to claim 5, characterized in that, The step of applying the synchronized confidence level to the fraud detection model to obtain the fraud confidence level of the test face output by the fraud detection model includes: Based on the synchronization confidence level, the fraud confidence level of the face under test is determined; the fraud confidence level is negatively correlated with the synchronization confidence level.

7. The face liveness detection method according to claim 5, characterized in that, After inputting the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model, the method further includes: Data integrity analysis is performed on the video frame sequence and the IMU sensor data sequence to obtain the data integrity analysis results of the video frame sequence and the IMU sensor data sequence; If the data integrity analysis results indicate that both the video frame sequence and the IMU sensor data sequence are complete, the confidence scores for the pulling-away action, the approach action, the non-far-near movement action, and the spoofing confidence score are output. If the data integrity analysis results indicate that either the video frame sequence or the IMU sensor data sequence is incomplete, the system outputs the confidence level of the pulling-away action, the confidence level of the approaching action, and a data missing information; the data missing information includes a video frame missing information or a sensor data missing information. If the data integrity analysis results indicate that both the video frame sequence and the IMU sensor data sequence are incomplete, a system anomaly warning will be output.

8. The face liveness detection method according to claim 5, characterized in that, The near-far movement detection model is trained through the following steps: Acquire the sample video frame sequence and sample IMU sensor data sequence simultaneously acquired from the sample face; Repeat the following steps until the set stopping condition is met: The sample video frame sequence is subjected to temporal feature extraction using a first feature extraction model to obtain sample temporal visual features. The sample IMU sensor data sequence is subjected to temporal feature extraction using a second feature extraction model to obtain sample temporal sensing features. The sample temporal visual features and the sample temporal sensing features are then fused to obtain sample fused features. The sample fusion features are input into the near-far motion detection model to obtain the near-far motion prediction results output by the near-far motion detection model. Based on the near and far motion prediction results and the near and far motion ground truth labels of the sample video frame sequences, the mean squared error loss and cross-entropy loss are determined. The comprehensive loss function is determined based on the mean squared error loss and the cross-entropy loss; The model parameters of the near and far motion detection model are adjusted based on the comprehensive loss function.

9. A face liveness detection device, characterized in that, include: The acquisition module is used to acquire the video frame sequence and IMU sensor data sequence of the face under test being synchronously captured; The first extraction module is used to extract temporal features from the video frame sequence using a first feature extraction model to obtain temporal visual features, extract temporal features from the IMU sensor data sequence using a second feature extraction model to obtain temporal sensing features, and fuse the temporal visual features and the temporal sensing features to obtain fused features. The second extraction module is used to determine the first motion feature of the face under test in a first direction based on the video frame sequence, to determine the second motion feature of the face under test in a second direction based on the IMU sensor data sequence, and to determine the synchronization confidence of the video and sensor data based on the first motion feature and the second motion feature. The prediction module is used to input the fused features and the synchronization confidence into the joint classification model to obtain the near and far movement detection results and fraud detection results of the test face output by the joint classification model.

10. A face liveness detection system, characterized in that, include: The image acquisition module is used to acquire video frame sequences of the face to be tested; The sensor acquisition module is used to acquire IMU sensor data sequences of the face to be tested; The data processing module is used to acquire the video frame sequence and IMU sensor data sequence of the face to be tested being synchronously captured; to extract temporal features from the video frame sequence using a first feature extraction model to obtain temporal visual features; to extract temporal features from the IMU sensor data sequence using a second feature extraction model to obtain temporal sensing features; and to fuse the temporal visual features and the temporal sensing features to obtain fused features. Based on the video frame sequence, a first motion feature of the face under test in a first direction is determined, and a second motion feature of the face under test in a second direction is determined based on the IMU sensor data sequence. The synchronization confidence of the video and sensor data is determined based on the first motion feature and the second motion feature. The fused features and the synchronization confidence are input into a joint classification model to obtain the near and far motion detection results and fraud detection results of the face under test output by the joint classification model.

Images