A face recognition and living body detection fusion method and device based on a binocular camera

Abstract

The application provides a face recognition and living body detection fusion method and device based on a binocular camera, and the method comprises the following steps: after collecting original left and right face images, performing binocular image preprocessing to obtain a standard binocular image pair; performing multi-scale stereo feature extraction on the standard binocular image pair to obtain a depth-texture joint feature representation and a disparity map; combining the disparity map, performing adaptive 3D face reconstruction and time sequence dynamic feature modeling, and through multi-level living body detection fusion decision and face feature extraction and matching, completing end-to-end fusion verification of face living body detection and face recognition. Through deep fusion of binocular stereo vision and deep learning, multi-dimensional living body discrimination features are constructed, and various known and unknown attacks can be effectively resisted. An adaptive stereo matching and 3D face reconstruction mechanism is designed, the unstable recognition problem caused by illumination change and posture change is overcome, and the system robustness is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and biometric recognition technology, and in particular to a method and apparatus for fusing face recognition and liveness detection based on a binocular camera. Background Technology

[0002] Currently, mainstream facial recognition systems face severe security challenges. Traditional monocular camera solutions are vulnerable to attacks using photos, videos, and 3D masks. Existing liveness detection technologies, such as blink detection and motion commands, offer a poor user experience and are easily bypassed. While binocular cameras can provide depth information, current methods fail to fully exploit the complementary information between the two perspectives and are ill-suited to changes in lighting, pose, and advanced attack methods.

[0003] In recent years, deep learning-based liveness detection methods have emerged, but most still rely on monocular images, ignoring the unique advantages offered by stereo vision. The current monocular approach suffers from the inability to acquire accurate 3D structural information, insufficient defense against high-fidelity attacks, and a high false rejection rate, which are pressing technical problems that need to be solved in this field. Summary of the Invention

[0004] This invention aims to solve these problems by providing a method and apparatus for fusing face recognition and liveness detection based on binocular cameras. This addresses the issues of monocular solutions being unable to acquire accurate 3D structural information, having insufficient defense against high-fidelity attacks, and having a high false rejection rate.

[0005] This invention provides a method for fusing face recognition and liveness detection based on a binocular camera, comprising:

[0006] After acquiring the original left and right face images, binocular image preprocessing is performed to obtain standard binocular image pairs;

[0007] Multi-scale stereo feature extraction is performed on the standard binocular image pairs to obtain a joint depth-texture feature representation and disparity map;

[0008] By combining the disparity map, adaptive 3D face reconstruction and temporal dynamic feature modeling are performed. Through multi-level liveness detection fusion decision-making and face feature extraction and matching, end-to-end fusion verification of face liveness detection and face recognition is completed.

[0009] The binocular image preprocessing takes the original left and right view face images synchronously acquired by the binocular cameras as input, and outputs a standard binocular image pair after epipolar alignment, spatiotemporal synchronization and illumination equalization processing, which serves as the input for multi-scale stereo feature extraction.

[0010] The binocular image preprocessing includes two sub-steps: stereo calibration and adaptive illumination equalization.

[0011] Adaptive illumination equalization is based on Retinex theory to construct a dual-path processing network, which dynamically adjusts the processing strategy according to the ambient light intensity to achieve image illumination equalization in strong light, weak light, and backlight scenes.

[0012] The multi-scale stereo feature extraction takes preprocessed standard binocular image pairs as input and outputs depth-texture joint feature representation and high-precision disparity map as input for the adaptive 3D face reconstruction step.

[0013] The multi-scale stereo feature extraction employs a weight-sharing Siamese two-stream network structure, using an improved MobileNetV3 as the backbone network to extract multi-scale features from left and right viewpoint images. Adaptive fusion of the dual-viewpoint features is achieved through a stereo attention module. The attention weight calculation formula for the stereo attention module is as follows:

[0014]

[0015] in, σ is the attention weight matrix output by the stereo attention module; σ is the Sigmoid activation function; Fl is the multi-scale feature map extracted from the left-view image, and Fr is the multi-scale feature map extracted from the right-view image; is the cross-correlation operation of the feature maps; C is the number of channels in the feature maps.

[0016] The temporal dynamic feature modeling involves extracting facial micro-motion features between consecutive frames using a lightweight optical flow estimation network, modeling the temporal evolution of these micro-motions using a temporal convolutional network, and establishing motion consistency constraints for left and right viewpoints. The consistency of the dynamic features from both viewpoints is constrained by a viewpoint transition invariant loss function, the formula for which is:

[0017]

[0018] in, is the viewpoint transformation invariance loss value; N is the total dimension of the feature vectors; n is the dimension index of the feature vectors; The nth dynamic feature value extracted from the left perspective at time t; This is the feature transformation matrix between the left and right viewpoints, calculated from the extrinsic calibration results of the binocular camera. The nth dynamic feature value extracted from the right perspective at time t; It is the square of the L2 norm, used to calculate the Euclidean distance deviation of the eigenvectors.

[0019] The multi-level liveness detection fusion decision employs a four-level hierarchical verification mechanism, consisting of rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection. An adaptive uncertainty weight is assigned to each verification level, and the final face liveness probability is calculated using an uncertainty gating mechanism. The formula for calculating the liveness probability is as follows:

[0020]

[0021] in, is the final face liveness probability; i is the level index of the four-level hierarchical verification; wi is the adaptive weight corresponding to the i-th level verification result; finally, the fusion verification conclusion is output by combining the liveness detection results.

[0022] According to another aspect of this patent, this patent also provides a face recognition and liveness detection fusion device based on a binocular camera, comprising:

[0023] The acquisition and processing unit is configured to acquire the original left and right face images, perform binocular image preprocessing, and obtain a standard binocular image pair.

[0024] The extraction unit is configured to perform multi-scale stereo feature extraction on the standard binocular image pair to obtain a joint depth-texture feature representation and a disparity map.

[0025] The fusion verification unit is configured to perform adaptive 3D face reconstruction and temporal dynamic feature modeling in conjunction with the disparity map, and complete end-to-end fusion verification of face liveness detection and face recognition through multi-level liveness detection fusion decision and face feature extraction and matching.

[0026] The acquisition and processing unit further includes

[0027] The binocular image preprocessing takes the original left and right view face images synchronously acquired by the binocular cameras as input, and outputs a standard binocular image pair after epipolar alignment, spatiotemporal synchronization and illumination equalization processing, which serves as the input for multi-scale stereo feature extraction.

[0028] The binocular image preprocessing includes two sub-steps: stereo calibration and adaptive illumination equalization.

[0029] Adaptive illumination equalization is based on Retinex theory to construct a dual-path processing network, which dynamically adjusts the processing strategy according to the ambient light intensity to achieve image illumination equalization in strong light, weak light, and backlight scenes.

[0030] The extraction unit further includes,

[0031] The multi-scale stereo feature extraction takes preprocessed standard binocular image pairs as input and outputs depth-texture joint feature representation and high-precision disparity map as input for the adaptive 3D face reconstruction step.

[0032] The multi-scale stereo feature extraction employs a weight-sharing Siamese two-stream network structure, using an improved MobileNetV3 as the backbone network to extract multi-scale features from left and right viewpoint images. Adaptive fusion of the dual-viewpoint features is achieved through a stereo attention module. The attention weight calculation formula for the stereo attention module is as follows:

[0033]

[0034] in, σ is the attention weight matrix output by the stereo attention module; σ is the Sigmoid activation function; Fl is the multi-scale feature map extracted from the left-view image, and Fr is the multi-scale feature map extracted from the right-view image; is the cross-correlation operation of the feature maps; C is the number of channels in the feature maps.

[0035] The fusion verification unit further includes

[0036] The temporal dynamic feature modeling extracts facial micro-motion features between consecutive frames using a lightweight optical flow estimation network, models the temporal evolution of micro-motions using a temporal convolutional network, and establishes motion consistency constraints for left and right viewpoints. The consistency of the dynamic features from both viewpoints is constrained by a viewpoint transition invariance loss function, the formula for which is:

[0037]

[0038] in, is the viewpoint transformation invariance loss value; N is the total dimension of the feature vectors; n is the dimension index of the feature vectors; The nth dynamic feature value extracted from the left perspective at time t; This is the feature transformation matrix between the left and right viewpoints, calculated from the extrinsic calibration results of the binocular camera. The nth dynamic feature value extracted from the right perspective at time t; It is the square of the L2 norm, used to calculate the Euclidean distance deviation of the eigenvectors.

[0039] The fusion verification unit further includes

[0040] The multi-level liveness detection fusion decision adopts a four-level hierarchical verification mechanism, namely, rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection. An adaptive uncertainty weight is assigned to the verification result at each level, and the final face liveness probability is calculated through an uncertainty gating mechanism. The formula for calculating the liveness probability is as follows:

[0041]

[0042] in, is the final face liveness probability; i is the level index of the four-level hierarchical verification; wi is the adaptive weight corresponding to the i-th level verification result; finally, the fusion verification conclusion is output by combining the liveness detection results.

[0043] This invention provides a method for fusing face recognition and liveness detection based on a binocular camera, comprising: acquiring original left and right face images, performing binocular image preprocessing to obtain a standard binocular image pair; performing multi-scale stereo feature extraction on the standard binocular image pair to obtain a depth-texture joint feature representation and a disparity map; combining the disparity map to perform adaptive 3D face reconstruction and temporal dynamic feature modeling; and completing end-to-end fusion verification of face liveness detection and face recognition through multi-level liveness detection fusion decision-making and face feature extraction and matching.

[0044] This invention overcomes the information limitations of monocular vision by deeply integrating binocular stereo vision with deep learning to construct multi-dimensional liveness detection features, effectively resisting various known and unknown attacks. It also designs an adaptive stereo matching and 3D face reconstruction mechanism to overcome recognition instability caused by changes in lighting and pose, thereby improving the robustness of the system.

[0045] The above description is merely an overview of this solution. In order to better understand the technical means of this invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this invention more apparent and understandable, specific embodiments of this invention are described below. Attached Figure Description

[0046] To more clearly illustrate this technical solution, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this technology. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This invention illustratively demonstrates a method for fusing face recognition and liveness detection based on a binocular camera;

[0048] Figure 2This is a schematic block diagram illustrating the structure of a face recognition and liveness detection fusion device based on a binocular camera, as shown in the present invention. Detailed Implementation

[0049] The embodiments of the present invention will be described in detail below, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0050] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0051] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0052] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0053] Currently, the main technical solutions in the field of face liveness detection and recognition can be divided into the following categories:

[0054] Traditional monocular vision-based methods include Eyeblink detection, texture analysis (LBP, HOG features, etc.), and spectral analysis. These methods determine whether an image is real by analyzing texture anomalies and physiological signals. However, they have limited defense capabilities against high-quality attacks (such as high-resolution photos and video replays).

[0055] Monocular methods based on deep learning, such as FaceNet and DeepPixBiS, utilize CNN networks to learn distinguishing features between live and non-live subjects. These methods perform well on standard datasets, but lack generalization ability, are poorly robust to unseen attack types, and cannot utilize depth information.

[0056] Basic binocular vision methods: such as the face liveness detection method based on binocular vision proposed in patent CN110782654A, which determines the face depth by calculating the disparity map of the left and right images, but only uses static depth features and does not consider the temporal dynamic characteristics, thus having limited defense capabilities against high-precision 3D mask attacks.

[0057] Multimodal fusion methods, such as the face verification system that combines RGB, infrared and depth information as described in US20210097345A1, rely on special hardware (infrared camera), are costly, and are not optimized for the characteristics of binocular cameras.

[0058] Existing monocular solutions cannot acquire accurate 3D structural information, have insufficient defense against high-fidelity attacks, and have a high false rejection rate.

[0059] like Figure 1 As shown in the embodiment of this patent, a method for fusing face recognition and liveness detection based on a binocular camera is provided, including:

[0060] After acquiring the original left and right face images, binocular image preprocessing is performed to obtain standard binocular image pairs;

[0061] Multi-scale stereo feature extraction is performed on the standard binocular image pairs to obtain a joint depth-texture feature representation and disparity map;

[0062] By combining the disparity map, adaptive 3D face reconstruction and temporal dynamic feature modeling are performed. Through multi-level liveness detection fusion decision-making and face feature extraction and matching, end-to-end fusion verification of face liveness detection and face recognition is completed.

[0063] This patent combines binocular stereo vision and deep learning to achieve end-to-end fusion verification of face liveness detection and face recognition; it breaks through the information limitations of monocular vision, and constructs multi-dimensional liveness discrimination features through the deep fusion of binocular stereo vision and deep learning, effectively resisting various known and unknown attacks; it designs an adaptive stereo matching and 3D face reconstruction mechanism to overcome the recognition instability caused by changes in lighting and pose, and improve the robustness of the system.

[0064] The binocular image preprocessing takes the original left and right view face images synchronously acquired by the binocular cameras as input, and outputs a standard binocular image pair after epipolar alignment, spatiotemporal synchronization and illumination equalization processing, which serves as the input for multi-scale stereo feature extraction.

[0065] The binocular image preprocessing includes two sub-steps: stereo calibration and adaptive illumination equalization.

[0066] Adaptive illumination equalization is based on Retinex theory to construct a dual-path processing network, which dynamically adjusts the processing strategy according to the ambient light intensity to achieve image illumination equalization in strong light, weak light, and backlight scenes.

[0067] Specifically, the binocular image preprocessing takes the original left and right view face images synchronously acquired by the binocular cameras as input, and outputs a standard binocular image pair after epipolar alignment, spatiotemporal synchronization, and illumination equalization processing, which serves as the core input for the multi-scale stereo feature extraction step. The binocular image preprocessing includes two sub-steps: stereo calibration and adaptive illumination equalization. The stereo calibration uses the Zhang Zhengyou calibration method to calibrate the intrinsic and extrinsic parameters of the binocular cameras, eliminates the frame rate deviation between the two cameras through a frame synchronization algorithm, and performs epipolar alignment processing on the left and right view images. The adaptive illumination equalization constructs a dual-path processing network based on Retinex theory, dynamically adjusts the processing strategy according to the ambient light intensity, and completes the image illumination equalization in strong light, weak light, and backlight scenes.

[0068] The multi-scale stereo feature extraction takes preprocessed standard binocular image pairs as input and outputs depth-texture joint feature representation and high-precision disparity map as input for the adaptive 3D face reconstruction step.

[0069] The multi-scale stereo feature extraction employs a weight-sharing Siamese two-stream network structure, using an improved MobileNetV3 as the backbone network to extract multi-scale features from left and right viewpoint images. Adaptive fusion of the dual-viewpoint features is achieved through a stereo attention module. The attention weight calculation formula for the stereo attention module is as follows:

[0070]

[0071] in, σ is the attention weight matrix output by the stereo attention module; σ is the Sigmoid activation function; Fl is the multi-scale feature map extracted from the left-view image, and Fr is the multi-scale feature map extracted from the right-view image; is the cross-correlation operation of the feature maps; C is the number of channels in the feature maps.

[0072] Specifically, multi-scale stereo feature extraction takes preprocessed standard binocular image pairs as input and outputs a depth-texture joint feature representation and a high-precision disparity map, which serve as the core input for the adaptive 3D face reconstruction step. The multi-scale stereo feature extraction employs a weight-sharing Siamese dual-stream network structure, using an improved MobileNetV3 as the backbone network to extract multi-scale features from the left and right viewpoint images. Adaptive fusion of the dual-viewpoint features is achieved through a stereo attention module (SAM). The attention weight calculation formula for the stereo attention module is as follows:

[0073]

[0074] in, The attention weight matrix output by the stereo attention module, with dimensions and left and right viewpoint features. Figure 1 σ is used to characterize the matching correlation of features from both perspectives; σ is the Sigmoid activation function, used to normalize the weight values ​​to the 0~1 range; Fl is the multi-scale feature map extracted from the left perspective image, and Fr is the multi-scale feature map extracted from the right perspective image; ⊗ is the cross-correlation operation of the feature maps, used to calculate the similarity of features from both perspectives; C is the number of channels of the feature map, used to scale and normalize the cross-correlation results to avoid numerical overflow; finally, multi-level cost volume aggregation is completed based on attention weights to generate a high-precision disparity map, and the depth features corresponding to the disparity map are fused with RGB texture features to obtain a depth-texture joint feature representation.

[0075] The temporal dynamic feature modeling extracts facial micro-motion features between consecutive frames using a lightweight optical flow estimation network, models the temporal evolution of micro-motions using a temporal convolutional network, and establishes motion consistency constraints for left and right viewpoints. The consistency of the dynamic features from both viewpoints is constrained by a viewpoint transition invariance loss function, the formula for which is:

[0076]

[0077] in, is the viewpoint transformation invariance loss value; N is the total dimension of the feature vectors; n is the dimension index of the feature vectors; The nth dynamic feature value extracted from the left perspective at time t; This is the feature transformation matrix between the left and right viewpoints, calculated from the extrinsic calibration results of the binocular camera. The nth dynamic feature value extracted from the right perspective at time t; It is the square of the L2 norm, used to calculate the Euclidean distance deviation of the eigenvectors.

[0078] Specifically, the temporal dynamic feature modeling takes a personalized face point cloud model output from adaptive 3D face reconstruction and a continuous frame binocular image sequence as input, and outputs temporal dynamic liveness features verified by consistency, which serve as input for the multi-level liveness detection fusion decision step. The temporal dynamic feature modeling extracts facial micro-motion features between consecutive frames through a lightweight optical flow estimation network, models the temporal evolution of micro-motions through a temporal convolutional network (TCN), and simultaneously establishes motion consistency constraints for left and right perspectives. The consistency of the dual-view dynamic features is constrained through a perspective transition invariance loss function, the formula for which is:

[0079]

[0080] in, The value represents the viewpoint transformation invariance loss; the smaller the value, the stronger the consistency of dynamic features between the left and right views. N is the total dimension of the feature vector; n is the dimension index of the feature vector. The nth dynamic feature value extracted from the left perspective at time t; This is the feature transformation matrix between the left and right views, calculated from the binocular camera extrinsic calibration results, used to achieve spatial alignment of features from both views. The nth dynamic feature value extracted from the right perspective at time t; It is the square of the L2 norm, used to calculate the Euclidean distance deviation of the eigenvectors.

[0081] The multi-level liveness detection fusion decision adopts a four-level hierarchical verification mechanism, namely, rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection. An adaptive uncertainty weight is assigned to the verification result at each level, and the final face liveness probability is calculated through an uncertainty gating mechanism. The formula for calculating the liveness probability is as follows:

[0082]

[0083] in, is the final face liveness probability; i is the level index of the four-level hierarchical verification; wi is the adaptive weight corresponding to the i-th level verification result; finally, the fusion verification conclusion is output by combining the liveness detection results.

[0084] Specifically, the multi-level liveness detection fusion decision and face feature extraction and matching takes depth-texture joint feature representation, 3D face material features, and temporal dynamic liveness features as inputs, and outputs the final liveness detection result and face recognition matching result. The multi-level liveness detection fusion decision adopts a four-level hierarchical verification mechanism, namely, rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection. Adaptive uncertainty weights are assigned to the verification results at each level, and the final face liveness probability is calculated through an uncertainty gating mechanism. The formula for calculating the liveness probability is:

[0085]

[0086] in, The final face liveness probability ranges from 0 to 1, with values ​​closer to 1 indicating higher confidence in a live face. `i` represents the level index of the four-level hierarchical verification, with values ​​of 1, 2, 3, and 4, corresponding to rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection, respectively. `wi` is the adaptive weight corresponding to the i-th level verification result, dynamically allocated by the uncertainty quantification value of that level's verification result; the lower the uncertainty, the larger the value of `wi`. `si` is the normalized score of the i-th level verification, ranging from 0 to 1; a higher score indicates higher confidence in a live face at that level. The face feature extraction and matching dynamically adjusts the matching threshold based on the liveness probability `Plive`. A lenient threshold is used to improve recognition efficiency under high liveness confidence, while a strict threshold is used to enhance security under low liveness confidence. Finally, a fusion verification conclusion is output based on the liveness detection results.

[0087] The following section will use real-time face verification for consumer-grade mobile smart terminals as an example to illustrate this patent in detail.

[0088] This embodiment is applied to an Android smartphone equipped with a dual-lens RGB camera with a baseline distance of 35mm, for device unlocking and small-amount mobile payment scenarios. The goal is to achieve real-time inference of ≥30FPS on the Snapdragon 778G mobile platform, balancing a seamless user experience with payment-grade basic security requirements. The implementation process is as follows.

[0089] In the binocular image preprocessing stage, this embodiment uses a 2-megapixel global shutter binocular module with a native frame rate of 30 FPS for both left and right cameras and an original acquisition resolution of 1920×1080. The binocular camera calibration is performed in advance using the Zhang Zhengyou calibration method, obtaining the intrinsic parameter matrix of the left camera (fx=1056.3, fy=1054.7, cx=962.5, cy=541.2), the intrinsic parameter matrix of the right camera (fx=1055.8, fy=1054.2, cx=958.7, cy=539.8), distortion coefficients, and extrinsic parameters (rotation matrix R, translation vector T). After acquiring the original left and right face images, a spatiotemporal synchronization algorithm is used to match the left and right frames at the same time, eliminating frame rate deviations of ±1 frame. Then, stereo correction is used to align the epipolar lines, ensuring that corresponding points in the left and right images are on the same horizontal line. The corrected image pairs are then downsampled to a resolution of 1280×720 and output.

[0090] The ambient lighting is a typical indoor scene with 500 lux. After obtaining the light intensity through the light sensor, an adaptive weight α=0.6 is used to construct a dual-path processing network based on Retinex theory: one path preserves the original image texture details, and the other path decomposes the image into the illumination component L and the reflection component R to complete the illumination equalization process. Finally, a standard binocular image pair is output as the input for the multi-scale stereo feature extraction step.

[0091] In the multi-scale stereo feature extraction stage, the preprocessed standard binocular image pairs are used as input. A weight-shared Siamese two-stream network is adopted, and the backbone network is an improved MobileNetV3-Large. The left and right images are respectively input into two branches with completely shared weights. Multi-scale feature maps are extracted at the 2nd, 4th and 8th layers of the network, with corresponding resolutions of 320×180, 160×90 and 80×45.

[0092] The attention weights for dual-view features are calculated using a stereo attention module (SAM), and the formula is as follows:

[0093] Book

[0094] In this example, the number of feature channels C=128, and Fl and Fr are the multi-scale feature maps extracted from the left and right viewpoints, respectively. For feature cross-correlation operation, σ is the Sigmoid activation function, and the normalized attention weight matrix is ​​calculated. After weighted fusion of dual-view features, a high-precision disparity map with a disparity range of 0-64 pixels is generated by aggregation through a 3-level cost volume. Finally, depth features and RGB texture features are fused to output a depth-texture joint feature representation and disparity map, which serve as inputs for the adaptive 3D face reconstruction step.

[0095] In the adaptive 3D face reconstruction and temporal dynamic feature modeling process, the disparity map is used as the core input. Based on confidence weighting, low-confidence regions such as hair and background are filtered out, while the disparity points of the face ROI region are retained. A sparse face point cloud with 1280 vertices is generated through a fast triangulation algorithm. Then, the sparse point cloud is fitted to the BSM baseline 3D face model through a non-rigid ICP algorithm to generate a personalized 3D face reconstruction model. The surface normal vector field is extracted from the 3D model, and the reflection characteristics under different viewpoints are calculated to obtain the surface roughness and reflectivity. The 3D face material features are constructed, and the personalized face point cloud model is output as the input for temporal dynamic feature modeling.

[0096] Using a sequence of three consecutive binocular images and a personalized face point cloud model as input, a lightweight LiteFlowNet optical flow estimation network is used to calculate the motion field between consecutive frames, extracting micro-expressions and micro-motion features such as eyelid micro-movements. A two-layer temporal convolutional network (TCN) is used to model the temporal evolution. Simultaneously, a left-right viewpoint motion consistency constraint is established, and feature consistency is verified using a viewpoint transformation invariance loss function. The formula is as follows:

[0097]

[0098] In this embodiment, the feature dimension N=256. To obtain the left and right view feature transformation matrices from the calibration, the real human sample was calculated. =0.023, which is less than the preset threshold of 0.1, indicating that the consistency of dynamic features meets the requirements. The temporal dynamic liveness feature is then output as the input for the multi-level liveness detection fusion decision step.

[0099] In the multi-level liveness detection fusion decision and face feature matching process, depth-texture joint features, 3D face material features, and temporal dynamic liveness features are used as inputs to perform four-level hierarchical verification, obtaining normalized scores for each level: rapid depth screening s1=0.98, texture-depth consistency verification s2=0.96, temporal biometric verification s3=0.95, and adversarial example detection s4=0.99. Adaptive weights are assigned according to the uncertainty of each level of verification results: w1=0.2, w2=0.3, w3=0.3, w4=0.2, and the final liveness probability is calculated using the formula:

[0100]

[0101] Calculated =0.965, which is greater than the preset liveness threshold of 0.9, and is therefore determined to be a live human. Based on this liveness probability, a relaxed matching threshold of 0.65 is used to extract adversarial robust facial features and perform similarity matching with the features registered in the device. The similarity is 0.89, which is greater than the threshold, and the person is determined to be the real person, and the verification result is output. In this embodiment, the verification time per frame is 28ms, the inference speed reaches 35FPS, which meets the real-time requirements of mobile devices, and the false rejection rate (FRR) is ≤1.2%.

[0102] like Figure 2 As shown, according to another aspect of this patent, this patent also provides a face recognition and liveness detection fusion device based on a binocular camera, comprising:

[0103] The acquisition and processing unit is configured to acquire the original left and right face images, perform binocular image preprocessing, and obtain a standard binocular image pair.

[0104] The extraction unit is configured to perform multi-scale stereo feature extraction on the standard binocular image pair to obtain a joint depth-texture feature representation and a disparity map.

[0105] The fusion verification unit is configured to perform adaptive 3D face reconstruction and temporal dynamic feature modeling in conjunction with the disparity map, and complete end-to-end fusion verification of face liveness detection and face recognition through multi-level liveness detection fusion decision and face feature extraction and matching.

[0106] The acquisition and processing unit also includes

[0107] The binocular image preprocessing takes the original left and right view face images synchronously acquired by the binocular cameras as input, and outputs a standard binocular image pair after epipolar alignment, spatiotemporal synchronization and illumination equalization processing, which serves as the input for multi-scale stereo feature extraction.

[0108] The binocular image preprocessing includes two sub-steps: stereo calibration and adaptive illumination equalization.

[0109] Adaptive illumination equalization is based on Retinex theory to construct a dual-path processing network, which dynamically adjusts the processing strategy according to the ambient light intensity to achieve image illumination equalization in strong light, weak light, and backlight scenes.

[0110] The extraction unit also includes,

[0111] The multi-scale stereo feature extraction takes preprocessed standard binocular image pairs as input and outputs depth-texture joint feature representation and high-precision disparity map as input for the adaptive 3D face reconstruction step.

[0112] The multi-scale stereo feature extraction employs a weight-sharing Siamese two-stream network structure, using an improved MobileNetV3 as the backbone network to extract multi-scale features from left and right viewpoint images. Adaptive fusion of the dual-viewpoint features is achieved through a stereo attention module. The attention weight calculation formula for the stereo attention module is as follows:

[0113]

[0114] in, σ is the attention weight matrix output by the stereo attention module; σ is the Sigmoid activation function; Fl is the multi-scale feature map extracted from the left-view image, and Fr is the multi-scale feature map extracted from the right-view image; is the cross-correlation operation of the feature maps; C is the number of channels in the feature maps.

[0115] The fusion verification unit also includes

[0116] The temporal dynamic feature modeling extracts facial micro-motion features between consecutive frames using a lightweight optical flow estimation network, models the temporal evolution of micro-motions using a temporal convolutional network, and establishes motion consistency constraints for left and right viewpoints. The consistency of the dynamic features from both viewpoints is constrained by a viewpoint transition invariance loss function, the formula for which is:

[0117]

[0118] in, is the viewpoint transformation invariance loss value; N is the total dimension of the feature vectors; n is the dimension index of the feature vectors; The nth dynamic feature value extracted from the left perspective at time t; This is the feature transformation matrix between the left and right viewpoints, calculated from the extrinsic calibration results of the binocular camera. The nth dynamic feature value extracted from the right perspective at time t; It is the square of the L2 norm, used to calculate the Euclidean distance deviation of the eigenvectors.

[0119] The fusion verification unit also includes

[0120] The multi-level liveness detection fusion decision adopts a four-level hierarchical verification mechanism, namely, rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection. An adaptive uncertainty weight is assigned to the verification result at each level, and the final face liveness probability is calculated through an uncertainty gating mechanism. The formula for calculating the liveness probability is as follows:

[0121]

[0122] in, is the final face liveness probability; i is the level index of the four-level hierarchical verification; wi is the adaptive weight corresponding to the i-th level verification result; finally, the fusion verification conclusion is output by combining the liveness detection results.

[0123] For details of the embodiments of the apparatus claims, please refer to the method embodiments section. To avoid repetition, they will not be repeated here.

[0124] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fusing face recognition and liveness detection based on a binocular camera, characterized in that, include: After acquiring the original left and right face images, binocular image preprocessing is performed to obtain standard binocular image pairs; Multi-scale stereo feature extraction is performed on the standard binocular image pairs to obtain a joint depth-texture feature representation and disparity map; By combining the disparity map, adaptive 3D face reconstruction and temporal dynamic feature modeling are performed. Through multi-level liveness detection fusion decision-making and face feature extraction and matching, end-to-end fusion verification of face liveness detection and face recognition is completed.

2. The face recognition and liveness detection fusion method based on a binocular camera as described in claim 1, characterized in that, The binocular image preprocessing takes the original left and right view face images synchronously acquired by the binocular cameras as input, and outputs a standard binocular image pair after epipolar alignment, spatiotemporal synchronization and illumination equalization processing, which serves as the input for multi-scale stereo feature extraction. The binocular image preprocessing includes two sub-steps: stereo calibration and adaptive illumination equalization. Adaptive illumination equalization is based on Retinex theory to construct a dual-path processing network, which dynamically adjusts the processing strategy according to the ambient light intensity to achieve image illumination equalization in strong light, weak light, and backlight scenes.

3. The face recognition and liveness detection fusion method based on a binocular camera as described in claim 2, characterized in that, The multi-scale stereo feature extraction takes preprocessed standard binocular image pairs as input and outputs depth-texture joint feature representation and high-precision disparity map as input for the adaptive 3D face reconstruction step. The multi-scale stereo feature extraction employs a weight-sharing Siamese two-stream network structure, using an improved MobileNetV3 as the backbone network to extract multi-scale features from left and right viewpoint images. Adaptive fusion of the dual-viewpoint features is achieved through a stereo attention module. The attention weight calculation formula for the stereo attention module is as follows: in, σ is the attention weight matrix output by the stereo attention module; σ is the Sigmoid activation function; Fl is the multi-scale feature map extracted from the left-view image, and Fr is the multi-scale feature map extracted from the right-view image; is the cross-correlation operation of the feature maps; C is the number of channels in the feature maps.

4. The face recognition and liveness detection fusion method based on a binocular camera as described in claim 3, characterized in that, The temporal dynamic feature modeling extracts facial micro-motion features between consecutive frames using a lightweight optical flow estimation network, models the temporal evolution of micro-motions using a temporal convolutional network, and establishes motion consistency constraints for left and right viewpoints. The consistency of the dynamic features from both viewpoints is constrained by a viewpoint transition invariance loss function, the formula for which is: in, is the viewpoint transformation invariance loss value; N is the total dimension of the feature vectors; n is the dimension index of the feature vectors; The nth dynamic feature value extracted from the left perspective at time t; This is the feature transformation matrix between the left and right viewpoints, calculated from the binocular camera extrinsic calibration results. The nth dynamic feature value extracted from the right perspective at time t; The square of the L2 norm is used to calculate the Euclidean distance deviation of the eigenvectors.

5. The face recognition and liveness detection fusion method based on a binocular camera as described in claim 4, characterized in that, The multi-level liveness detection fusion decision adopts a four-level hierarchical verification mechanism, namely, rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection. An adaptive uncertainty weight is assigned to the verification result at each level, and the final face liveness probability is calculated through an uncertainty gating mechanism. The formula for calculating the liveness probability is as follows: in, is the final face liveness probability; i is the level index of the four-level hierarchical verification; wi is the adaptive weight corresponding to the i-th level verification result; finally, the fusion verification conclusion is output by combining the liveness detection results.

6. A face recognition and liveness detection fusion device based on a binocular camera, characterized in that, include: The acquisition and processing unit is configured to acquire the original left and right face images, perform binocular image preprocessing, and obtain a standard binocular image pair. The extraction unit is configured to perform multi-scale stereo feature extraction on the standard binocular image pair to obtain a joint depth-texture feature representation and a disparity map. The fusion verification unit is configured to perform adaptive 3D face reconstruction and temporal dynamic feature modeling in conjunction with the disparity map, and complete end-to-end fusion verification of face liveness detection and face recognition through multi-level liveness detection fusion decision and face feature extraction and matching.

7. The face recognition and liveness detection fusion device based on a binocular camera as described in claim 6, characterized in that, The acquisition and processing unit also includes The binocular image preprocessing takes the original left and right view face images synchronously acquired by the binocular cameras as input, and outputs a standard binocular image pair after epipolar alignment, spatiotemporal synchronization and illumination equalization processing, which serves as the input for multi-scale stereo feature extraction. The binocular image preprocessing includes two sub-steps: stereo calibration and adaptive illumination equalization. Adaptive illumination equalization is based on Retinex theory to construct a dual-path processing network, which dynamically adjusts the processing strategy according to the ambient light intensity to achieve image illumination equalization in strong light, weak light, and backlight scenes.

8. The face recognition and liveness detection fusion device based on a binocular camera as described in claim 7, characterized in that, The extraction unit also includes, The multi-scale stereo feature extraction takes preprocessed standard binocular image pairs as input and outputs depth-texture joint feature representation and high-precision disparity map as input for the adaptive 3D face reconstruction step. The multi-scale stereo feature extraction employs a weight-sharing Siamese two-stream network structure, using an improved MobileNetV3 as the backbone network to extract multi-scale features from left and right viewpoint images. Adaptive fusion of the dual-viewpoint features is achieved through a stereo attention module. The attention weight calculation formula for the stereo attention module is as follows: in, σ is the attention weight matrix output by the stereo attention module; σ is the Sigmoid activation function; Fl is the multi-scale feature map extracted from the left-view image, and Fr is the multi-scale feature map extracted from the right-view image; is the cross-correlation operation of the feature maps; C is the number of channels in the feature maps.

9. The face recognition and liveness detection fusion device based on a binocular camera as described in claim 8, characterized in that, The fusion verification unit also includes The temporal dynamic feature modeling extracts facial micro-motion features between consecutive frames using a lightweight optical flow estimation network, models the temporal evolution of micro-motions using a temporal convolutional network, and establishes motion consistency constraints for left and right viewpoints. The consistency of the dynamic features from both viewpoints is constrained by a viewpoint transition invariance loss function, the formula for which is: in, is the viewpoint transformation invariance loss value; N is the total dimension of the feature vectors; n is the dimension index of the feature vectors; The nth dynamic feature value extracted from the left perspective at time t; This is the feature transformation matrix between the left and right viewpoints, calculated from the binocular camera extrinsic calibration results. The nth dynamic feature value extracted from the right perspective at time t; The square of the L2 norm is used to calculate the Euclidean distance deviation of the eigenvectors.

10. The face recognition and liveness detection fusion device based on a binocular camera as described in claim 9, characterized in that, The fusion verification unit also includes The multi-level liveness detection fusion decision adopts a four-level hierarchical verification mechanism, namely, rapid depth screening, texture-depth consistency verification, temporal biometric verification, and adversarial example detection. An adaptive uncertainty weight is assigned to the verification result at each level, and the final face liveness probability is calculated through an uncertainty gating mechanism. The formula for calculating the liveness probability is as follows: in, is the final face liveness probability; i is the level index of the four-level hierarchical verification; wi is the adaptive weight corresponding to the i-th level verification result; finally, the fusion verification conclusion is output by combining the liveness detection results.

Images