A dynamic palmprint and palm vein dual-mode decision fusion living body detection method and system

Abstract

This invention relates to the field of biometric recognition and liveness detection technology, and particularly to a dynamic palmprint and palm vein dual-modal decision fusion method and system for liveness detection. The method dynamically acquires key points of the palm in a video stream in real time. Using the backbone network of an improved AG-FAS model, it extracts liveness features of the palm from the input multi-condition fused palmprint and palm vein image dataset through transfer learning, outputting liveness feature data. Two independent detection models, a palmprint liveness model and a palm vein liveness model, are used to obtain corresponding realism confidence scores. A fusion judgment is then performed based on the confidence scores of the two independent models to obtain the determination result. Applying this invention can significantly improve the accuracy, robustness, and generalization of live hand detection.

Description

Technical Field

[0001] This invention relates to the field of biometric recognition and liveness detection technology, and in particular to a dynamic palmprint and palm vein dual-modal decision fusion method and system for liveness detection. Background Technology

[0002] Palmprint and palm vein recognition, as important biometric identification technologies, have been widely used in the field of identity authentication in recent years due to their high uniqueness and difficulty in forgery. They are often used in identity verification scenarios such as security gates and payments.

[0003] Current biometric technologies have many shortcomings. For example, some systems rely solely on single palm print or palm vein features for identification, making them vulnerable to deception by using fake hands or photographs. Some systems collect images only from specific areas; if the hand is tilted, near, or far away, invalid information is collected, affecting not only the judgment but also user experience. For instance, most methods employ static ROI extraction strategies, which are poorly adaptable to changes in hand position, distance, and angle, leading to biased feature extraction and consequently impacting liveness detection accuracy. Some recognition models are only suitable for young adults and are prone to errors when detecting children or middle-aged individuals, making it difficult to maintain consistent recognition performance across different age groups and potentially resulting in high false rejection rates. Furthermore, some systems initially combine two types of features for analysis; if one feature is compromised, the overall result will be incorrect.

[0004] Existing palmprint and palm vein biometric technologies suffer from problems such as weak resistance to attacks, poor adaptability, insufficient compatibility with the entire population, and low security of fusion, making it difficult to meet the needs of real-world applications. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic palmprint and palm vein dual-modal decision fusion liveness detection method and system, which solves the problems of weak anti-attack capability and poor adaptability of existing palmprint and palm vein biometric technologies, realizes a fusion recognition scheme with high security, and improves the accuracy, robustness and generalization of live hand detection.

[0006] To achieve the above objectives, the present invention provides the following technical solution: According to one aspect of the present invention, a dynamic palmprint and palm vein dual-modal decision fusion liveness detection method is provided, comprising the following steps: S1: Construct a multi-condition fusion dataset of palm print and palm vein dual-modality; S2: Real-time acquisition and detection of key points of the palm in the video stream, continuous acquisition of multiple frames of images within a set time window, and output of aligned multi-frame palm print and palm vein images; S3: Using the backbone network of the improved AG-FAS model, extract palm liveness features from the input multi-condition fusion dataset through transfer learning, and output palm liveness feature data; S4: Input the multi-frame palm print and palm vein images and the palm live feature data into two independent detection models, the palm print live model and the palm vein live model, respectively, to obtain the corresponding first and second authenticity confidence scores; S5: Calculate the average of the realism confidence scores of the two independent detection models and compare them with a preset threshold: only when the average confidence scores of both bidirectional modes exceed the corresponding thresholds is the output "live" result determined; otherwise, it is determined as an "attack".

[0007] According to an embodiment of the present invention, in step S1, the palm print and palm vein bimodal dataset includes a self-built dataset and the publicly available CASIA hand dataset; the self-built dataset includes IR and RGB bimodal images acquired synchronously covering different ages, fake materials and lighting conditions, and the images in the self-built dataset are preprocessed by cropping and scaling according to the standard of the CASIA hand dataset.

[0008] According to an embodiment of the present invention, in step S2, the real-time acquisition supports synchronous acquisition of IR and RGB under three lighting conditions: bright light and dark light.

[0009] According to one embodiment of the present invention, in step S2, key points of the palm are detected in real time using MediaPipe's Hands solution, and the palm bounding box is dynamically adjusted. In step S2, continuously acquiring multiple frames of images within a set time window means acquiring multiple frames of images at a rate of more than 10 frames per second within the time window, with palm print and palm vein images each accounting for half; in step S4, the authenticity confidence is calculated using the multiple valid frames of images uniformly sampled within the time window.

[0010] According to an embodiment of the present invention, in step S3, the backbone network of the improved AG-FAS model, based on the AG-FAS idea, generates a "real palm" image by training the Stable Diffusion Unet generative model through the CASIA dataset, and compares it with the input image to obtain a residual image. Then, it combines the self-built dataset, the CASIA dataset and the corresponding residual image to fine-tune the non-realistic attention network OA-Net, thereby realizing the transfer and adaptation of the face domain model to the palm domain.

[0011] According to one embodiment of the present invention, the OA-Net network processes the input image through the VIT model, processes the residual image through the ResNet model, and then fuses the processing results of the two models through a cross-attention mechanism to focus on the key feature regions in the residual image.

[0012] According to one embodiment of the present invention, the self-built dataset covers people aged 12 to 40 and above, and the fake materials include silicone prosthetic hands, plastic prosthetic hands, and photographs.

[0013] According to one embodiment of the present invention, the living features of the palm include the optical reflectivity of the skin texture and the blood flow information of the veins.

[0014] On the other hand, the present invention also provides a dynamic palmprint and palm vein dual-modal decision fusion liveness detection system, the system comprising: The data acquisition and dynamic ROI extraction module is used to detect key points of the palm in the video stream in real time through MediaPipe's Hands solution, dynamically adjust the palm bounding box, continuously acquire multiple frames of palm print and palm vein dual-modal images within a set time window, and pause acquisition when the palm is not recognized. Data preprocessing module: used to standardize the acquired bimodal images, locate the core area of ​​the palm, and crop and scale it to 128×128 pixels; Dual-modal feature extraction module: Based on the improved AG-FAS backbone network, it generates a "real hand" image through the Stable DiffusionUnet generative model, compares it with the input image to obtain a residual image, and then extracts hand-specific liveness features through the OA-Net network adapted by transfer learning. The independent discrimination module is used to input the preprocessed valid image into two independent models: the palm print liveness model and the palm vein liveness model. Each model outputs a corresponding realism confidence score. The decision fusion module is used to calculate the average confidence score of the outputs of two independent models, compare it with the preset threshold, and output the final judgment result. In the attack scenario, an alarm is triggered. The judgment result includes "live" and "attack". Only when the average confidence scores of the palm print and palm vein dual modal exceed the corresponding thresholds will the "live" judgment result be output.

[0015] The present invention provides a dynamic palmprint and palm vein dual-modal decision fusion liveness detection method and system. Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The bimodal decision-level fusion method described above significantly enhances the system's resistance to various deception attacks (photos, screen copying, silicone or plastic gloves). By combining the bimodal decision-level fusion mechanism (independent discrimination and dual verification) with residual image feature recognition, the recognition rate for various attack types such as silicone simulated hands, plastic simulated hands, photos, and screen copying reaches almost 100%, solving the problem that single-modal or feature-level fusion is easily deceived.

[0016] 2. The dynamic ROI extraction using MediaPipe effectively overcomes the impact of changes in hand position and angle, improving user experience and system stability. Based on MediaPipe, the dynamic ROI extraction technology can adapt to changes in hand position, angle, and distance in real time, eliminating the need for users to manually align the capture frame and significantly improving ease of operation. It can accurately extract features even in scenarios with a 30° posture shift or a 5cm distance change, reducing the false rejection rate to 0%.

[0017] 3. Using a handprint training dataset encompassing multiple age groups ensures accurate detection for different populations, including teenagers, young adults, and middle-aged individuals, thus avoiding age discrimination. The self-built dataset covers individuals aged 12 to 40 and above. During model training, features of non-young users are enhanced, achieving near 100% detection accuracy across different age groups, effectively avoiding the age discrimination issues of existing technologies.

[0018] 4. It possesses strong generalization ability. By designing datasets with various lighting conditions and simulation materials, and supplementing them with the CASIA public dataset, the model achieved 100% accuracy in tests across different time periods (daytime and nighttime with varying lighting) and validation on different datasets (CASIA dataset), demonstrating its adaptability to complex real-world application scenarios. For the first time, the AG-FAS face liveness detection framework was adapted to the hand domain through transfer learning. Combined with the Stable Diffusion Unet generative model, the accuracy of hand liveness feature extraction was improved, and training efficiency was increased by 30% compared to training the model from scratch.

[0019] 5. The model design balances accuracy and efficiency, with a detection time of less than 10 seconds to meet the real-time requirements of practical application scenarios: the RGB image discriminator has an accuracy of 99.742%, and the infrared image discriminator has an accuracy of 100%. It completes the entire process of acquisition, extraction, discrimination, and fusion within 10 seconds, meeting the real-time requirements of scenarios such as access control and payment. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a dynamic palmprint and palm vein dual-modal decision fusion liveness detection method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a dynamic palmprint and palm vein dual-modal decision fusion liveness detection system according to an embodiment of the present invention; Figure 3 This is a diagram of the improved AG-FAS hand-based liveness network architecture according to an embodiment of the present invention; Figure 4This is a configuration diagram of the core feature extraction parameters in the improved AG-FAS palm liveness network according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hierarchical relationship and parameters of residual networks 1-4 in an embodiment of the present invention; Figure 6 This is a schematic diagram of IR and RGB images of a forged hand in a self-built dataset according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the IR and RGB images of a real hand in a self-built dataset according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a CASIA dataset image example according to an embodiment of the present invention; Figure 9 This is a flowchart of the entire preprocessing process from data collection to standardized implementation of dynamic ROI extraction according to an embodiment of the present invention; Figure 10 This is a flowchart of the training process for the feature extraction model and the Stable Diffusion Unet generative model according to an embodiment of the present invention. Figure 11 This is a flowchart of the training process of the OA-Net network and FAS classifier according to an embodiment of the present invention; Figure 12 This is a flowchart of the independent discrimination and confidence score calculation of the palm vein live model according to an embodiment of the present invention; Figure 13 This is a flowchart of the independent discrimination and confidence score calculation of the palm print liveness model according to an embodiment of the present invention. Detailed Implementation

[0021] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0022] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0023] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0024] like Figure 1 As shown, a flowchart of a dynamic palmprint and palm vein dual-modal decision fusion liveness detection method is presented. The method includes the following steps: S1: Construct a multi-condition fusion dataset of palm print and palm vein dual-modality; S2: Real-time acquisition and detection of key points of the palm in the video stream, continuous acquisition of multiple frames of images within a set time window, and output of aligned multi-frame palm print and palm vein images; S3: Using a fine-tuned Stable Diffusion Unet generative model, a corresponding "simulated real palm image" is generated for each palm print and palm vein image to be detected, and the input image is compared with the simulated real image to generate a feature residual image; S4: Using the backbone network of the improved OA-Net model, the multi-frame palm print and palm vein images and the corresponding residual images are input into two independent detection models, the palm print liveness model and the palm vein liveness model, respectively, to extract features and obtain the realism confidence of the images. S5: Calculate the average of the realism confidence scores of the two independent detection models and compare them with a preset threshold: only when the average confidence scores of both bidirectional modes exceed the corresponding thresholds will the "live" determination result be output; otherwise, it will be determined as an "attack".

[0025] like Figure 2 The diagram shows a schematic of a dynamic palmprint and palm vein dual-modal decision fusion liveness detection system, which is used to implement the above method, including... The data acquisition and dynamic ROI extraction module is used to detect key points of the palm in the video stream in real time through MediaPipe's Hands solution, dynamically adjust the palm bounding box, continuously acquire multiple frames of palm print and palm vein dual-modal images within a set time window, and pause acquisition when the palm is not recognized. Data preprocessing module: used to standardize the acquired bimodal images, locate the core area of ​​the palm, and crop and scale it to 128×128 pixels; The residual image generation module generates a simulated real hand image based on existing features using a fine-tuned Stable Diffusion Unet generative model. This image is then compared with the input image to obtain a residual image, resulting in a special contrast feature extraction map. The independent feature extraction and discrimination module inputs the palm print and palm vein images into the corresponding OA-Net networks respectively. The input images and the corresponding residual images are combined through transfer learning to adapt the OA-Net networks of their respective modalities to extract palm-specific live features. The live features are then input into the discrimination model to output the corresponding realism confidence score. The decision fusion module is used to calculate the average confidence score of the outputs of two independent models, compare it with the preset threshold, and output the final judgment result. In the attack scenario, an alarm is triggered. The judgment result includes "live" and "attack". Only when the average confidence scores of the palm print and palm vein dual modal exceed the corresponding thresholds will the "live" judgment result be output.

[0026] In the dual-modal biometric detection technology of palmprint and palm vein, the Region of Interest (ROI) is a core area specifically selected from an image or video stream that is useful for a particular target task, while other irrelevant areas such as background, finger edges, and device frames are excluded, retaining only the important information for subsequent processing. By selecting the ROI, the goal of focusing on key information can be achieved, allowing the model to focus only on the core features of palmprints and palm veins, thereby improving detection efficiency and accuracy. Traditional technologies use static ROIs with fixed areas, such as a fixed 256×256 pixel bounding box. The ROI of this invention is dynamically adjusted. Through MediaPipe, 21 key points of the palm are tracked in real time, and the bounding box of the ROI is adjusted synchronously according to the movement, tilt, and distance changes of the palm, so as to always accurately surround the core area of ​​the palm, solving the problem of not being able to obtain effective features when the hand is tilted or close to or far away.

[0027] like Figure 3 The diagram shows the OA-Net and classifier structures used for feature extraction and high-level feature interaction in the palm liveness network of the AG-FAS model. The improved OA-Net module is used to adapt the AG-FAS model from face recognition to palm recognition and to extract palm liveness features. The key layers and functions of the model architecture are as follows: For the contrast residual map, the input is a contrast difference map based on the preprocessed original image and the simulated real image output by Stable Diffusion Unet, which is then fed into the convolutional activation layer; The convolutional activation layer is used to perform convolution and ReLU activation operations on the input residual map. The convolutional kernel size is 3×3. Its core function is to perform secondary filtering and enhancement on the features output by the residual network, highlighting the key features of the palm comparison and suppressing the influence of noise; the output of the initially extracted feature map is sent to the max pooling layer. The max pooling layer is used to adjust the size of the output feature map to a uniform feature map processing dimension before outputting it to the residual network layer. The residual network layer consists of four residual network layers (residual network 1, 2, 3, and 4). By continuously extracting features from the input residual map layer by layer from shallow to deep, the extracted residual map features are gradually enriched to include palm contrast features and then further input into the next layer. Adaptive average pooling and reshaping automatically adjust the size of the output feature map, compressing the feature map to a fixed target size and changing the data dimension to conform to the format of cross self-attention, before outputting it to Cross Attention.

[0028] For the preprocessed original image, the input is a preprocessed standard format input image, which is input to the Patch Embedding layer; The Patch Embedding layer is used to uniformly segment the input palm image. Each patch corresponds to a local region of the palm image, such as a small patch of veins or skin texture. Each patch is converted into a one-dimensional feature vector, and the final output is a sequence of feature vectors. This solves the problem that the Transformer encoder cannot directly process two-dimensional feature maps, laying the foundation for the attention mechanism to focus on key local features. The Transformer encoder layer, consisting of eight layers including multi-head self-attention and a feedforward neural network (FFN), is used to calculate the correlation between different patch vectors through the multi-head self-attention mechanism. For example, it strengthens the feature correlation between the main vein and its branches, and between skin texture and the vein crossing area, allowing the network to more accurately capture the unique feature combination patterns of the living hand, such as the correlation between the optical reflection differences corresponding to venous blood flow and texture. The feedforward neural network performs nonlinear transformations on the feature vectors after attention interaction to improve the feature representation ability. Cross Attention and CLS token layer: Cross Attention introduces feature extraction information from the residual map and performs cross-attention calculation with the patch vector output by the Transformer encoder, guiding the network to prioritize features of the core area of ​​the palm (such as the center of the palm print and dense vein areas). The CLS token adds a classification token (CLS token) at the beginning of the patch vector sequence. This token aggregates the feature information of all patch vectors through an attention mechanism, and finally outputs a 768-dimensional global feature vector. The linear mapping layer performs a linear mapping on the 768-dimensional global feature vector output by the CLS token, compressing the dimension to a dimension that is compatible with subsequent linear layers. The output is a specific frame image discrimination score, which supports the final discrimination of hand liveness features.

[0029] like Figure 4 The diagram shows the composition of key modules for core feature extraction in the AG-FAS hand liveness network, as detailed below: The initial convolutional activation layer, labeled "convolutional activation" in the figure, performs preliminary feature extraction on the residual image through the convolutional layer, performs batch normalization, and performs ReLU operation, laying the foundation for subsequent residual feature extraction; The Patch Embedding layer is used to uniformly segment the input palm image into 16×16 pixel patches, each patch corresponding to a local region of the palm image. Each patch is converted into a one-dimensional feature vector, and preliminary feature extraction is performed through convolutional layers, resulting in preliminary image processing. This addresses the issue that multi-head self-attention Transformer encoders cannot directly process two-dimensional feature maps, achieving format conversion from two-dimensional spatial features to one-dimensional sequence features. The residual network layer, connected to the max pooling layer, is labeled "residual network" in the diagram. It extracts features through convolutional batch normalization activation and performs convolutional batch normalization to complete feature transformation and adapt residual connections. It is then merged with the original input and fed into the next residual layer.

[0030] like Figure 5 The diagram shows the specific parameters of the residual network module in the AG-FAS improved backbone network, and details the internal connection logic and parameter configuration of the OA-Net network. The modules involved are as follows: The convolutional activation layer consists of batch normalization of convolutional layers combined with ReLU, with a kernel size of 7×7, stride of 2, padding of 3, and 64 output channels. The convolutional layer performs preliminary feature extraction on the residual image, laying the foundation for subsequent residual feature extraction. The max pooling layer has a kernel size of 3×3, a stride of 2, padding of 1, and 64 output channels. It reduces the feature map resolution by taking the local maximum value, thereby reducing the computational load and enhancing the network's anti-interference ability. The residual network module is the core component for realizing deep feature extraction and gradient stabilization. The structural parameters of each residual network are as follows: Residual Network 1 consists of batch-normalized activation and batch-normalized activation of convolutional layers forming basic residual blocks, with 1×1 convolutional layers used as residual connections; all convolutional kernels are 3×3 in size, with a stride of 1, padding of 1, and 64 output channels; used to connect... Figure 4 The output features of the max pooling layer are used for preliminary deep feature extraction to enhance the underlying texture features of the palm in the residual image.

[0031] The Residual Network 2 has the same "batch normalization activation of convolutional layers → batch normalization of convolutional layers" structure as the Residual Network 1, only adjusting the stride of some layers; the stride of some convolutional layers is adjusted to 2, which is used to halve the feature map size and increase the number of output channels to 128; the Residual Network 2 is used to improve the feature dimension and focus on the mid-layer features of the palm of the residual image, such as the branching structure of veins.

[0032] The structure of Residual Network 3 maintains the core logic of the residual block, and the residual connection adapts to the change in the number of channels; the stride of the key convolutional layer is 2, and the number of output channels is increased to 256; it is used to further deepen feature extraction and capture high-level features of the palm in the residual image, such as optical reflection difference features related to venous blood flow.

[0033] The structure of Residual Network 4 continues the core logic of the residual block, and the residual connection ensures gradient propagation; the number of output channels is increased to 512, which is used to output the final deep palm features of the network, supporting residual feature analysis and subsequent classifier classification.

[0034] also, Figure 5 It also annotates Patch Embedding parameters, with a Patch size of 16×16 and an output dimension of 768. It connects the residual network with subsequent Transformer modules, realizing the conversion from feature maps to feature vectors, providing feature data in a suitable format for attention mechanism analysis.

[0035] like Figure 6 As shown, images of fake hand samples are presented, including bimodal images: the left side is the IR image of the fake hand, and the right side is the RGB image of the fake hand. The fake hand samples are divided into three categories based on material: (1) Silicone prosthetic hand sample. No visible veins are visible in the IR image. The simulated color is more uniform than that of a real palm. There is no difference in brightness caused by blood flow. The skin texture is stiff in the RGB image. Simulated veins are visible. It does not have the delicate texture of real skin. (2) Plastic prosthetic hand sample. There are almost no vein features in the IR image, which means that the plastic material cannot simulate the optical properties of blood vessels. In the RGB image, the edge of the palm outline is sharp and does not have the natural curvature of a real palm; (3) Photo sample. The palm cannot be displayed in the IR image, the paper in the photo cannot reflect infrared light to simulate blood vessels, the paper texture is present in the RGB image, and the details of the palm (such as between the fingers) are blurred due to the limitations of the printing resolution.

[0036] All the fake hand samples were photographed under "indoor light" conditions, with diverse backgrounds and lighting conditions covering both bright and dark scenes.

[0037] like Figure 7 As shown, real hand sample images are presented, including bimodal images: the left side is the IR image of the real hand, and the right side is the RGB image of the real hand. The sample set covers both bright and dark lighting conditions, reflecting the dataset's coverage of lighting variations. The real hand samples are divided into three categories based on age: (1) Juvenile samples (less than 12 years old): Due to incomplete hand development, the veins in the IR image are thinner and sparser, while the skin texture in the RGB image is delicate and without obvious signs of aging. (2) Young people (13-25 years old): The hands are fully developed, the veins are clear and the branches are complete in the IR image, the skin texture is uniform in the RGB image, and the palm print features are obvious; (3) Middle-aged samples (26-40 years old): Skin thickness increased, the clarity of veins in IR images decreased slightly, and fine aging lines appeared in the skin texture in RGB images, with the edges of palm prints slightly blurred.

[0038] like Figure 8 As shown, standardized images from the CASIA public dataset are presented. All hand samples are real hand samples, without age annotations but meeting the standardized collection requirements of the CASIA dataset. All images have a pure black background, with the hand centered, in a correct posture, without tilt or offset. Veins are clearly marked in the IR images, and the palm print area is unobstructed in the RGB images, meeting the standardized feature extraction benchmarks. The CASIA public dataset, as a supplement to the self-built dataset of this invention, provides industry-standardized samples for basic feature learning in the early stages of model training and for cross-dataset generalization validation in later stages.

[0039] like Figure 9 The diagram illustrates the ROI extraction preprocessing workflow for a multi-condition fusion dataset. The core objective of image preprocessing is to transform the raw palm print and palm vein images into standardized data that meets the model training requirements, while ensuring data diversity and feature integrity.

[0040] S91: Data acquisition phase.

[0041] Two types of raw data were input: one was a self-built dataset (covering people aged 12-40, using fake materials such as silicone / plastic prosthetic hands / photographs, and in bright / dark lighting scenarios, simultaneously collecting 5100 IR images and 6100 RGB images); the other was the publicly available CASIA hand dataset (standardized real hand images). Both types of data fully covered the central area of ​​the hand, ensuring coverage of multispectral (IR, RGB) and multi-scene lighting conditions.

[0042] S92: Palm key point detection.

[0043] By calling the Hands model through the MediaPipe framework, 21 key points of the palm, including the fingertips, the base of the fingers and the center of the palm, are detected in real time in all raw images to locate the core area of ​​the palm and eliminate interference from irrelevant areas such as the background and the edges of the fingers.

[0044] S93: Bounding box positioning and clipping.

[0045] Based on the detected key points, the minimum bounding box of the palm is dynamically calculated, and the region containing only the core features of the palm is cropped out (avoiding finger afterimages and device borders), ensuring that each image retains only the key areas of palm print texture and palm vein distribution.

[0046] S94: Size standardization.

[0047] The cropped hand images are uniformly scaled to 128×128 pixels. This size can preserve the subtle features of the hand while reducing the computational complexity of subsequent models, thus adapting to the input requirements of convolutional neural networks and generative models.

[0048] S95: Pixel value standardization.

[0049] The scaled image is Z-Score normalized (the mean pixel value is adjusted to 0 and the variance is adjusted to 1) to eliminate the interference of pixel value differences under different lighting conditions, and to ensure that the model training focuses on the features themselves rather than brightness differences.

[0050] S96: Data cleaning and filtering.

[0051] Blurry or occluded invalid images are removed, and the valid images are retained for subsequent segmentation to ensure the quality of training data.

[0052] S97: Divide the training set and validation set.

[0053] The self-built dataset and the CASIA dataset are divided in an 8:2 ratio into a training set (for model parameter optimization) and a validation set (for real-time evaluation of training performance and to avoid overfitting). The standardized dataset is then output as follows: Figure 10 and Figure 11 The model training provides qualified standardized input samples.

[0054] like Figure 10 The diagram shows the training flowchart for the feature extraction model and the Stable Diffusion Unet generative model. The core objective of cross-domain model adaptation training is to optimize the AG-FAS framework in the face domain to a model adapted to palm features through transfer learning, while simultaneously training an improved model to enhance the feature differences between real and fake palms. The training process is specifically divided into the following five key steps: S101: Input preprocessed image.

[0055] take over Figure 9 The output standardized training set images (128×128 pixels, IR and RGB dual modal) are real hand images from the CASIA dataset (high degree of standardization, suitable for initial model tuning).

[0056] S102: Feature extractor fine-tuning.

[0057] The initial model uses the pre-trained feature extractor of the original adapted face in the AG-FAS framework; The fine-tuning operation uses real hand images from the CASIA dataset for targeted fine-tuning, with a learning rate of 1e-3, a batch size of 32, and 500 iterations. The optimizer is AdamW, and the loss function is cross-entropy loss. The core objective is to guide the high-level network to focus on palm-specific features, such as vein patterns and skin optical reflection texture, while retaining the general features of the AG-FAS framework model at the bottom level (such as edge detection and texture recognition). This enables an initial transition from face adaptation to palm adaptation and outputs a finely tuned palm feature extractor.

[0058] S103: Model Integration.

[0059] The finely tuned feature extractor is embedded as a conditional guiding unit into the Stable Diffusion Unet denoising network, enabling U-Net to generate images guided by palm features rather than the default face features.

[0060] S104: Stable Diffusion Unet supervised fine-tuning.

[0061] The fine-tuning data consisted of 3000 real hand images from the CASIA dataset. The weights of the cross-attention mechanism in the U-Net denoising network were adjusted to increase the attention to details of the hand structure (such as the palm region and dense vein areas), thereby enhancing the reproduction of hand features in the generated images. The fine-tuning parameters included a learning rate of 1e-5, a batch size of 16, and 300 iterations. The goal was to enable the model to learn to generate simulated real hand images with a structural similarity of ≥98% to the real hand, and to output the optimized StableDiffusion Unet generation model.

[0062] S105: Output optimization model.

[0063] The final output consists of two types of models: a feature extraction model adapted to the palm and a Stable Diffusion Unet generative model that can generate high-quality "realistic palm" images. Figure 11 The core tool for classification training.

[0064] like Figure 11 The diagram shows the training flowchart for the OA-Net network and classifier. The liveness detection model training process shown is used to train the OA-Net network and classifier to distinguish between "real palms / forged attacks," forming two classification detection models: an independent palmprint detection model and an independent palm vein liveness detection model.

[0065] The core process of training the liveness detection model shown is divided into the following 6 key steps: S1101: Input three types of images.

[0066] The received data includes three categories: 6100 real hand images from a pre-processed self-built dataset, 5100 fake hand images (including silicone / plastic prosthetic hands and photographs), and... Figure 10 The model outputs 11,200 "simulated real hand images"; all three types of images are standardized images of 128×128 pixels.

[0067] S1102: Residual image generation.

[0068] The "real hand image / fake hand image" is compared with the "simulated real hand image" at the pixel level to generate a "residual image". The residual between the real hand image and the simulated image is small (indicating high feature similarity). The residual between the fake hand image and the simulated image is significant because it lacks live features such as venous blood flow texture and skin reflection characteristics, thus amplifying the feature differences between the real and fake hands.

[0069] S1103: Training the OA-Net network.

[0070] The network input consists of the input image and the corresponding residual image obtained in step S1102. Key distinguishing regions in the input image and residual image are fused through a cross-attention mechanism, such as areas with missing veins and areas with abnormal skin texture, to extract core features for distinguishing between "liveness" and "attack". The training parameters include a learning rate of 1e-3, a batch size of 2, and 500 iterations. The loss function is binary cross-entropy loss. Through training, OA-Net can accurately capture "liveness feature missing signals" in the image and output a distinguishing feature vector with uniform dimensions.

[0071] S1104: Classifier construction.

[0072] The classifier consists of a fully connected Softmax layer. The input is the discriminative feature vector output by OA-Net, and the output is a ground truth confidence score in the range of 0-1. A higher score indicates a greater probability that the input image represents a real live subject. The bimodal classifier is trained using palmprint images and palm vein images to train two independent classifiers: a palmprint liveness classifier (adapted to RGB image features) and a palm vein liveness classifier (adapted to IR image features). Each classifier is trained for 500 epochs until the validation set accuracy is ≥98.5%.

[0073] S1105: Output liveness detection model.

[0074] The final output includes two independent liveness detection models: a palmprint liveness model and a palm vein liveness model. Figure 12 The independent discrimination provides core model support.

[0075] like Figure 12 and Figure 13 The flowchart shown illustrates the process of bimodal independent discrimination and confidence calculation. The core objective of employing "bimodal independent model discrimination" is to calculate the authenticity confidence scores using two independent models—palmprint and palm vein—separately, providing foundational data for subsequent decision fusion. The process includes the following five key steps: S1201: Input multi-frame dual-modal images.

[0076] take over Figure 9 The multi-frame images output by the dynamic ROI extraction module are 100 frames acquired at 10 frames / second within 10 seconds, including 50 palm print images (RGB mode) and 50 palm vein images (infrared mode). All images are standardized images of 128×128 pixels.

[0077] S1202: Uniform sampling to select valid images.

[0078] To avoid noise interference from single-frame images, 50 frames of palm print images and 50 frames of palm vein images were uniformly sampled: one frame was sampled every 2 seconds, and 5 valid images were retained for each (covering different time periods of the 10-second acquisition window) to ensure the representativeness of the sampled images.

[0079] S1203: Dual-modal independent model input.

[0080] Input the valid palm print images (5 frames) Figure 11 A trained, independent palmprint liveness model; input the effective palm vein images (5 frames). Figure 11 A trained independent palm vein live model; two independent models are processed in parallel without interfering with each other, avoiding single-modal interference from affecting the discrimination of the other modality.

[0081] S1204: Model discrimination and confidence calculation.

[0082] Each independent model performs the following operations on a single frame of input image: (1) Call Figure 10 The generative model generates a "simulated real hand image" corresponding to the frame image; (2) Compare the input image with the “simulated real image” and input the comparison image into the ResNet model in the OA-Net module of the model to extract residual features; (3) Input the input image into the VIT model in the OA-Net module of the model to extract the basic features of the image; (4) The feature images output by the ResNet model and the VIT model are fused through the cross attention mechanism in the OA-Net module, and then the feature images are input into the classifier; (5) The classifier calculates and outputs the truth confidence score (0-1 interval) of the frame image based on the feature image.

[0083] S1205: Confidence score output.

[0084] Figure 12 The palm vein live model outputs 5 palm vein confidence scores. Figure 13 The palmprint liveness model outputs 5 palmprint confidence scores, and the average of these scores is taken to obtain the average confidence score of the palmprint and the average confidence score of the palm vein.

[0085] Finally, the decision fusion and result judgment are performed. Preset dual-modal thresholds (palmprint 0.50, palm vein 0.50); If the average confidence level of both modes is greater than or equal to the corresponding threshold, the device is determined to be "live" and an "access" signal is output (such as access control unlocking or payment permission). If the average confidence level of any modality is less than the threshold, it is determined to be an "attack", and a "rejection" signal is output and an audible and visual warning is triggered (such as access control locking, background log recording).

[0086] Example 1: Constructing a dataset.

[0087] The construction of the dataset is fundamental to ensuring the model's generalization ability and robustness. Given the current lack of publicly available datasets that combine real and fake attack hands, this embodiment adopts a strategy of combining self-built datasets with publicly available datasets.

[0088] The self-built dataset aims to cover the complexities of multimodal biometrics, including different ages, forged materials (silicone prosthetic hands, plastic prosthetic hands, and photographs), and various lighting variations. The dataset was collected from 10 participants aged 12 to over 40 to encompass key stages of hand morphological changes in individuals of different ages. A total of 5100 IR images and 6100 RGB images were acquired, all simultaneously in IR and RGB formats to ensure complete multispectral information. Each image fully captures the central area of ​​the palm and was captured under various lighting conditions, including natural light, indoor light, and low light, to enhance data diversity and real-world representativeness.

[0089] Simultaneously, the publicly available CASIA hand image dataset is introduced. This CASI dataset has a standardized acquisition process and annotation, making it suitable for biometric recognition research. It can be used for training and validation of feature extraction models and Stable Diffusion Unet generative models, thereby improving the model's adaptability to unknown data.

[0090] The overall dataset construction takes into account both real-world scenario variations and industry benchmarks, utilizes self-built data to highlight practical challenges, and leverages CASIA data to provide standard references.

[0091] During data preprocessing, all images underwent standardization to improve training efficiency. The Mediapipe framework was used for hand keypoint detection; the palm region was then cropped and scaled to 128×128 pixels. This process reduced computational complexity while preserving key features, meeting the input requirements of convolutional neural networks.

[0092] Example 2: Training of the Feature Extraction Model and the Stable Diffusion Unet Generative Model The AG-FAS base model is used for face recognition. Its generative model and related feature extractors were originally trained on face data and cannot be directly used for hand liveness detection. To address the domain adaptation issue, the AG-FAS base model is improved based on transfer learning.

[0093] First, the publicly pre-trained feature extraction model from the CASIA dataset is fine-tuned using real hand images. Second, the fine-tuned feature extraction model is integrated as a conditional guidance module into the Stable DiffusionUnet generative model, and the overall model is again fine-tuned in a supervised manner using the CASIA dataset. In this process, the model learns to generate a "real hand" based on the features of the input hand image, and the denoising network is adjusted to focus more on the hand structure and details.

[0094] The aforementioned improvements significantly enhance the overall performance of the model. The fine-tuned feature extraction model, along with StableDiffusion Unet, can generate more discriminative palm features, effectively extracting features from the real palm and laying the foundation for building a robust AG-FAS system.

[0095] Example 3: Specific Training of the AG-FAS Network The AG-FAS model suffers from weak generalization ability when training data is insufficient, especially when it is difficult to obtain a large number of diverse cheating samples. This paper addresses this issue by embedding a newly trained, hand-adapted generative model and feature extractor into the AG-FAS framework.

[0096] First, a generative model is used to generate high-quality "real hand" images in batches. These images are then compared with corresponding real or fake hands to generate difference maps of the input images. Next, the feature extractor and classifier of AG-FAS are trained using a self-built dataset and its corresponding difference maps. RGB and IR images are trained separately to ensure correct extraction of key regions and accurate classification. This training significantly enhances the model's discriminative ability. The difference map guidance enables OA-Net to learn more robust real and fake features, thus more accurately distinguishing between genuine biometric features and forged attacks when faced with hands of various ages, materials, and lighting conditions within the self-built dataset.

[0097] Example 4: Classification and Recognition Process.

[0098] The acquisition unit continuously captures 10 frames of palmprint images and 10 frames of palm vein images within 1 second. The palmprint images are input into a pre-trained palmprint liveness detection model, and the palm vein images are input into a pre-trained palm vein liveness detection model. If the Mediapipe_ROI does not recognize the palm region, the recognition count is too low, and the recognition process is paused. Both the palmprint and palm vein liveness detection models uniformly sample the input images, and a generative model generates a corresponding real image for each sampled image, which is then stitched into the input image. This image is then input into the subject discrimination model, which outputs a continuous confidence score representing its probability of being real. The decision fusion module fuses the 10 confidence scores generated by the palmprint and palm vein liveness detection models within 10 seconds. The final liveness determination decision is obtained by calculating the average of the confidence scores of the two models. If the average confidence score of the two models exceeds a preset threshold, it is determined to be a real live hand; otherwise, it is determined to be a fake attack.

[0099] Example 5: Result Evaluation.

[0100] To comprehensively evaluate the performance of this method, experiments used single-image recognition accuracy, cross-time-period generalization test accuracy, and cross-dataset validation accuracy as core evaluation metrics. The model performed exceptionally well on the internal validation set, achieving a single-image recognition accuracy of 99.742% for the RGB image discriminator and 100% accuracy for the IR image discriminator. The strategy of extracting 100 images during classification further improved the accuracy of liveness detection. The robustness and practical value of the model were further validated in more rigorous cross-time-period external attack tests. Under different lighting conditions (morning, noon, and evening), liveness detection was performed on 10 newcomers outside the dataset, as well as various attack vectors such as silicone prosthetic hands, plastic prosthetic hands, and photographs. Test results show that the system successfully and correctly identified all unknown real hands and various deception attacks, achieving almost 100% cross-time-period generalization test accuracy. Finally, to verify the model's generalization ability on large-scale public benchmarks, validation was performed on all hand images in the CASIA dataset. The model correctly identified all real hand samples in the dataset, achieving 100% cross-dataset validation accuracy. Although direct cross-dataset comparisons are not possible due to the lack of comparable open-source code in this field, the series of quantitative indicators obtained from the above experiments, approaching or reaching 100%, form a complete chain of evidence. From internal validation and external anti-attack testing to standard benchmark testing, these indicators collectively and strongly demonstrate that the AG-FAS method proposed in this invention possesses extremely high accuracy, robustness, and generalization ability, marking significant progress in the field of live hand detection and proving the practical applicability of the results.

[0101] This invention achieves an innovative combination of dynamic ROI and cross-domain models. Dynamic data acquisition is achieved through MediaPipe, while the AG-FAS model (face transfer to the palm) is improved, solving the dual problems of limited pose during static acquisition and insufficient training data for the palm model. Furthermore, a collaborative approach using residual images, independent discrimination, and dual-threshold fusion is employed to combine residual image features with independent discrimination by two models. The residual images enhance the differences in liveness features, and the independent calculation of confidence scores and dual-threshold verification by the two models form a complete logical chain of feature enhancement, independent discrimination, and secure fusion, rather than a simple superposition of technologies. Existing technologies do not cover multi-condition datasets for age (12-40 years), fake materials (3 types), and lighting (3 types). This invention supplements these with a self-built dataset and CASIA, addressing the deficiencies of age discrimination and weak generalization ability in existing technologies. The data construction strategy is highly targeted.

[0102] This invention presents a dynamic palmprint and palm vein dual-modal decision fusion liveness detection method and system. It adapts a mature facial region model to the palm region, improving the accuracy of palm biometric feature extraction while maintaining detection efficiency and real-time performance. It solves the feature extraction bias problem caused by changes in palm posture (position shift, angle tilt, distance change), optimizing the user experience. It can achieve almost 100% recognition of various forgery attacks, including silicone prosthetic hands, plastic prosthetic hands, photos, and screen replicas, improving the system's anti-attack capability. The classification model used maintains consistent detection accuracy across different age groups, including teenagers, young adults, and middle-aged individuals. A safe and reliable dual-modal fusion mechanism is employed to avoid the influence of single-modal interference on the detection results.

[0103] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and other materials. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0104] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A dynamic palmprint and palm vein dual-modal decision fusion method for liveness detection, characterized in that, Includes the following steps: S1: Construct a multi-condition fusion dataset of palm print and palm vein dual-modality; S2: Real-time acquisition and detection of key points of the palm in the video stream, continuous acquisition of multiple frames of images within a set time window, and output of aligned multi-frame palm print and palm vein images; S3: Using the backbone network of the improved AG-FAS model, extract palm liveness features from the input multi-condition fusion dataset through transfer learning, and output palm liveness feature data; S4: Input the multi-frame palm print and palm vein images and the palm live feature data into two independent detection models, the palm print live model and the palm vein live model, respectively, to obtain the corresponding first and second authenticity confidence scores; S5: Calculate the average of the realism confidence scores of the two independent detection models and compare them with a preset threshold: only when the average confidence scores of both bidirectional modes exceed the corresponding thresholds is the output "live" result determined; otherwise, it is determined as an "attack". In step S3, the backbone network of the improved AG-FAS model, based on the AG-FAS idea, uses the Stable Diffusion Unet generative model trained on the CASIA dataset to generate a "real hand" image, and compares it with the input image to obtain a residual image. Then, it combines a self-built dataset, the CASIA dataset, and the corresponding residual image to fine-tune the non-realistic attention network OA-Net, realizing the transfer and adaptation of the face domain model to the hand domain. The OA-Net network processes the input image through the VIT model and the residual image through the ResNet model. Then, it fuses the processing results of the two models through a cross-attention mechanism, focusing on the key feature regions in the residual image. The OA-Net network involves the following modules: The convolutional activation layer consists of batch normalization of convolutional layers combined with ReLU; Max pooling layers reduce feature map resolution by "taking local maxima"; The residual network module consists of a basic residual block composed of batch normalized activation of convolutional layers and batch normalized activation of convolutional layers. The convolutional kernels are all 3×3 in size, with a stride of 1, padding of 1, and 64 output channels, which are used to receive the output features of the max pooling layer. Residual Network 2 shares the same "batch normalization activation of convolutional layers → batch normalization of convolutional layers" structure as Residual Network 1, only adjusting the stride of some layers, with the stride of some convolutional layers adjusted to 2. The structure of Residual Network 3 maintains the core logic of residual blocks, with residual connections adapting to changes in the number of channels; the stride of key convolutional layers is 2, and the number of output channels is increased to 256. The structure of Residual Network 4 continues the core logic of residual blocks, with residual connections ensuring gradient propagation; the number of output channels is increased to 512, used to output the final deep palm features of the network.

2. The method according to claim 1, characterized in that, In step S1, the palm print and palm vein bimodal dataset includes a self-built dataset and the publicly available CASIA hand dataset. The self-built dataset includes IR and RGB bimodal images acquired synchronously, covering different ages, fake materials, and lighting conditions. The images in the self-built dataset are preprocessed by cropping and scaling according to the standards of the CASIA hand dataset.

3. The method according to claim 2, characterized in that, In step S2, the real-time acquisition supports synchronous acquisition of IR and RGB under three lighting conditions: bright light and dark light.

4. The method according to claim 1, characterized in that, In step S2, key points of the hand are detected in real time using MediaPipe's Hands solution, and the hand bounding box is dynamically adjusted.

5. The method according to claim 1, characterized in that, In step S2, continuously acquiring multiple frames of images within a set time window means acquiring multiple frames of images within the time window at a speed of more than 10 frames per second, with palm print and palm vein images each accounting for half. In step S4, the authenticity confidence score is calculated using multiple frames of valid images sampled uniformly within the time window.

6. The method according to claim 2 or 3, characterized in that, The self-built dataset covers people aged 12 to 40 and above, and the fake materials include silicone prosthetic hands, plastic prosthetic hands, and photographs.

7. The method according to claim 1, characterized in that, The living features of the palm include the optical reflectivity of the skin texture and the blood flow information of the veins.

8. A dynamic palmprint and palm vein dual-modal decision fusion liveness detection system, used to implement the method described in any one of claims 1 to 7, characterized in that, The system includes: The data acquisition and dynamic ROI extraction module is used to detect key points of the palm in the video stream in real time through MediaPipe's Hands solution, dynamically adjust the palm bounding box, continuously acquire multiple frames of palm print and palm vein dual-modal images within a set time window, and pause acquisition when the palm is not recognized. Data preprocessing module: used to standardize the acquired bimodal images, locate the core area of ​​the palm, and crop and scale it to 128×128 pixels; Dual-modal feature extraction module: Based on the improved AG-FAS backbone network, it generates a "real hand" image through the Stable Diffusion Unet generative model, compares it with the input image to obtain a residual image, and then extracts hand-specific liveness features through the OA-Net network adapted by transfer learning. The independent discrimination module is used to input the preprocessed valid image into two independent models: the palm print liveness model and the palm vein liveness model. Each model outputs a corresponding realism confidence score. The decision fusion module is used to calculate the average confidence score of the two independent models, compare it with the preset threshold, and output the final judgment result. The judgment result includes "live" and "attack". In the attack scenario, an alarm is triggered. The "live" judgment result is output only when the average confidence scores of the palm print and palm vein dual modal exceed the corresponding threshold.

Images