A finger vein identification system based on transfer learning and GAN network

Abstract

The invention discloses a finger vein accurate recognition system (VARS) based on transfer learning and GAN network; it includes an image acquisition and preprocessing module, a sample quality assessment module, a deception detection module, a feature extraction module, an identity authentication module, and a data storage module; a ternary network deep learning model is trained as a feature extraction module benchmark model, and then based on a transfer learning architecture based on a CycleGAN network, a small amount of real finger vein data is used to perform transfer learning on the benchmark model to obtain an accurate feature extraction module for real finger veins; the finger vein identity recognition system constructed by the feature extraction module has a series of advantages such as good robustness, high accuracy, fast running speed, simple and lightweight, extremely short time required for adding new users, and low deployment price. The data storage module, deception detection module, and identity authentication module constructed around the module occupy very little system storage and computing resources, so they have low hardware dependence and reduce deployment costs.

Description

Technical Field

[0001] The present invention relates to the field of biometric identification technology, and specifically provides a finger vein recognition method and system based on transfer learning and a GAN network. Background Art

[0002] Finger vein recognition technology uses the vein pattern image obtained after near-infrared rays penetrate the finger to identify individuals. It is the world's most advanced biometric technology with high precision and high speed. Among various biometric technologies, it is a technology that uses the internal characteristics of organisms that cannot be seen from the outside for identification, so it has attracted much attention as a second-generation biometric technology with high anti-counterfeiting properties. Since it uses internal body features for identification and must be alive to be used, finger veins must be living blood with flowing blood, and hidden inside the finger, it is extremely difficult to copy. Compared with identification technologies such as fingerprints, it has taken a big step forward. Traditional methods, such as methods based on finger vein patterns, SIFT, LBP, WLD, HOG and other artificially designed feature extraction methods, lack robustness in adapting to diverse data. When problems such as scattering phenomena of optical instruments, low resolution, and unclear blood vessels due to excessively large fingers occur, the above traditional methods are difficult to solve perfectly, resulting in performance degradation. Deep learning, as a technology that can automatically learn feature extraction methods from a large number of image sets, has gradually attracted attention from the scientific research community and the industrial community. However, a major prerequisite for deep learning is the need for a large amount of image set data, and it is difficult to collect a large amount of data in the field of finger vein. When the amount of data is insufficient, deep learning will experience overfitting: the model works very well under the training set data, but the results are poor in the test set and actual application.

[0003] When the amount of data in a certain field is insufficient, we can train a high-quality model from another similar field where it is easy to obtain a large amount of data. Using some learning frameworks, we can gradually migrate the model parameters based on the trained model, and finally migrate the model to a high-quality model in the target field. This learning method is called transfer learning.

[0004] Using transfer learning methods to optimize deep network models not only brings out the feature extraction capabilities of deep learning, but also effectively avoids the problem of limited overfitting capabilities of deep learning under small data volumes. Summary of the invention

[0005] The VARS (Vein Accurate Recognition System) of the present invention adopts the transfer learning method for the first time, using the generated massive simulated finger vein data to train the ternary network deep learning model as the feature extraction module benchmark model, and then based on the transfer learning architecture based on the CycleGAN network, a small amount of real finger vein data is used to transfer the benchmark model to obtain an accurate feature extraction module for real finger veins. The finger vein identity recognition system constructed by the feature extraction module has a series of advantages such as good robustness, high accuracy, fast running speed, simple and lightweight, extremely short time required for adding new users and low deployment price.

[0006] The VARS system can provide highly reliable security management measures for important entrances and exits, and is widely used in high-security fields such as banks, computer rooms, armories, confidential rooms, prisons, troops, governments, schools, offices, and other identity recognition-related application fields, such as finger vein payment and finger vein signature.

[0007] The present invention has the following advantages:

[0008] This patent is based on the deep convolutional network obtained through transfer learning training, which inherits the advantages of convolutional network in feature extraction. Compared with traditional image processing, pattern recognition and other artificial feature extraction methods, it has more comprehensive feature extraction capabilities, and has good robustness and accuracy for different image qualities and environmental influences. It does not require too many operations in image preprocessing (only ROI cropping, size unification, and lumen normalization are required), which greatly shortens the overall recognition process time. In addition, the transfer learning framework based on CycleGAN adopted in this patent solves the problem that deep convolutional networks are prone to overfitting (good training effect, poor actual application effect) under small data volume.

[0009] Due to the training methods of transfer learning and ternary networks and the fine network structure of the feature extraction module itself, the feature extraction module of this patent only needs 128 dimensions to well characterize a finger vein sample. Moreover, due to the low dimensionality, the data storage module, deception detection module, and identity authentication module built around this module occupy very little system storage and computing resources, so it has low hardware dependence and reduces deployment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 VARS Finger Vein Identification System

[0011] Figure 2 “Add User” Data Flow Diagram

[0012] Figure 3 "Add User" Logical Flowchart

[0013] Figure 4 “Identity Authentication” Data Flow Diagram

[0014] Figure 5 “Identity Authentication” Logical Flowchart

[0015] Figure 6 Feature extraction module structure

[0016] Figure 7 Convolutional BN layer structure diagram

[0017] Figure 8 101 shallow feature extraction unit structure diagram

[0018] Fig. 9 102 Deep feature extraction unit structure diagram

[0019] Fig.10 A-type inception residual block structure diagram

[0020] Fig.11 B-type inception residual block structure diagram

[0021] Fig.12 C-type inception residual block structure diagram

[0022] Fig.13 A-type characteristic distillation block structure diagram

[0023] Fig.14 B-type characteristic distillation block structure diagram

[0024] Fig.15 Training scheme for the feature extraction module benchmark model

[0025] Fig.16 Simulated finger vein

[0026] Fig.17 The objective of the triplet loss function

[0027] Fig.18 True finger veins collected by the device

[0028] Fig.19 True finger vein after ROI cropping

[0029] Fig. 20 CycleGAN network structure

[0030] Fig.21 Transfer learning network structure

[0031] Fig. 22 Image acquisition preprocessing module

[0032] Fig.23Schematic diagram of image acquisition unit 201

[0033] Fig.24 Quality Assessment Module

[0034] Fig.25 Deception Detection Module

[0035] Fig.26 Identity authentication module Specific implementation plan

[0036] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] System structure:

[0038] The VARS (Vein Accurate Recognition System) described in the present invention refers to a vein identification system such as Figure 1 As shown in the figure, the modules can be divided into: image acquisition and preprocessing module, sample quality assessment module, deception detection module, feature extraction module, identity authentication module, and data storage module. The functions of each module are summarized as follows:

[0039] Image acquisition preprocessing module: responsible for the acquisition of finger veins and the preprocessing of acquired images. After the finger vein acquisition device images, a finger vein image with a size of 400*200 is obtained, and the ROI is cropped according to the finger contour, and then the size is unified into a 320*128 finger vein area image, and the image is normalized for lumen.

[0040] Sample quality assessment module: responsible for assessing the quality of the collected and preprocessed images. The input is the finger vein image (size 320*128) preprocessed by the image acquisition preprocessing module. After quality assessment, if the image quality meets the standard, the input finger vein image will be output; if it is determined that the image quality is not good enough and the vein information is insufficient, the current workflow is terminated here and no longer output to the next module.

[0041] Deception detection module: responsible for judging whether the input image is a fake finger vein or a real finger vein. The input is the finger vein image (size 320*128) evaluated by the sample quality assessment module. If the image is judged to be a real finger vein, the input finger vein image will be output; if it is judged to be a fake finger vein, the current workflow will be terminated here, the system will alarm, and the fake input record and timestamp will be output to the data storage module as a system identification record.

[0042] Feature extraction module: It is trained through transfer learning and is responsible for extracting finger vein features from images. Each image outputs a 128-dimensional finger vein feature vector. The input is the finger vein image (size 320*128) output by the sample quality assessment module. After passing through the feature extraction model, a 128-dimensional finger vein feature vector is output. The feature extraction process supports batch processing.

[0043] Identity authentication module: responsible for judging the identity of the input feature vector. This module has two inputs: a. 128-dimensional finger vein features (identity to be verified) output by the feature extraction module, b. Identity feature distribution information of all internal personnel output by the data storage module. Based on the coverage between a and b compared by Euclidean distance, the identity to be verified is determined. Output -1 indicates an unknown outsider, and the corresponding identity number indicates a known person. The determination result and timestamp will be transmitted to the data storage module as a system identification record.

[0044] Data storage module: responsible for storing and reading finger vein feature information, encrypted finger vein samples and system identification records. This module has 5 working contents: 1. Storing encrypted images: receiving finger vein images output from the sample quality assessment module, encrypting the images, and storing the encrypted image files. 2. Storing identity feature distribution information: receiving multiple feature vectors of newly added users from the feature extraction module, generating feature distribution information of the identity after calculation and processing, and packaging and storing it with the user input name. 3. Storing system identification records: recording the forged input records from the deception detection module and the access / rejection records from the identity authentication module, including timestamps. This module stores the original records line by line. 4. Reading identity feature distribution information: reading the feature distribution information of all internal personnel and outputting it to the identity authentication module. 5. Reading system identification records: reading existing system identification records and outputting them to the system user interface for system administrators to view.

[0045] System work execution process:

[0046] The VARS system includes two working modes: adding new users and identity authentication.

[0047] Add a new user:

[0048] Add a new user to the system. The VARS system requires the user to successfully record the finger veins of the same finger 5 times, and then use these samples to generate feature information representing the user (finger) and record it into the system. After becoming a user of the system, you will be considered an internal member of the system.

[0049] The data flow diagram of "Add User" is as follows Figure 2 As shown, the logic flow chart is as follows Figure 3 As shown, the work execution process is as follows:

[0050] Step 1: Data collection and preprocessing. The user's finger is placed steadily on the finger vein collection device. First, it passes through the data collection and preprocessing module, and after image collection, ROI cropping, size unification, and lumen normalization, a finger vein image of 320*128 size is output.

[0051] Step 2: Sample quality assessment. After the sample quality assessment module, according to the evaluation result of the assessment module: if it is judged to be of unqualified quality, it will be rejected and entered into the system database. The system will ask the user to re-enter until 5 qualified finger vein samples are successfully entered. The 5 finger vein samples are encrypted and sent to the data storage module for sample storage.

[0052] Step 3: Perform image enhancement on the five finger vein samples, perform small-range rotation, magnification, and position shift, and increase the data volume by 10 times to 50 samples. Input the 50 samples in batches into the feature extraction module to obtain a feature group consisting of 50 128-dimensional finger vein features. Input the feature group into the data storage module, and after calculation and processing, generate the feature distribution information of the identity, and package and store it with the identity name entered by the user (the user's real name or work number, determined by the actual usage scenario).

[0053] “Identity Authentication”:

[0054] Determine whether the current system user is an internal staff. If it is determined to be an internal staff, allow him to enter. Otherwise, deny him to enter. The data flow diagram of "identity authentication" is as follows Figure 4 As shown, the logic flow chart is as follows Figure 5 As shown, the work execution process is as follows:

[0055] Step 1: Data collection and preprocessing. The user's finger is placed steadily on the finger vein collection device. First, it passes through the data collection and preprocessing module, and after image collection, ROI cropping, size unification, and lumen normalization, a 320*128 finger vein image is output.

[0056] Step 2: Sample quality assessment. After the sample quality assessment module, according to the evaluation result of the assessment module: if the quality is judged to be unqualified, the system will ask the user to re-enter. If the quality is judged to be qualified, the input original image of the module is sent to the fraud detection module.

[0057] Step 3: Deception detection. After the deception detection module, check whether it is a real finger vein image. If it is judged to be a forged finger vein image, the user's input will be rejected, the system will alarm, and the forged sample record and timestamp will be output to the data storage module as a system identification record, and the "identity authentication" process will end early. If it is judged to be a real finger vein, the module's input original image will be sent to the feature extraction module.

[0058] Step 4: Feature extraction: After the feature extraction module, the preprocessed finger vein samples are subjected to feature extraction to obtain 128-dimensional finger vein features, which are then output to the feature extraction module.

[0059] Step 5: Identity authentication. After the identity authentication module, the identity feature distribution information of all internal personnel is read from the data storage module, and the 128-dimensional features provided by the feature extraction module are compared with the identity feature distribution information of all internal personnel. If the identity authentication module determines that it is a certain category, the corresponding identity number is output to indicate a known person, indicating that the person is allowed to enter; if it is determined to be an outsider, -1 is output to indicate that the person is denied entry. The judgment result and timestamp will be sent to the data storage module as a system identification record.

[0060] Each module structure and design scheme:

[0061] 1. Feature extraction module:

[0062] 1.1. Feature extraction module structure:

[0063] like Figure 6 The figure shows the feature extraction module of the VARS finger vein identification system, including 101-104: 101 shallow feature extraction unit, 102 deep feature extraction unit, 103 fully connected feature mapping unit, 104 L2 regularization unit.

[0064] Detailed introduction of network structure:

[0065] In the present invention, the 2D convolutional BN layer is used to replace the ordinary 2D convolutional layer in order to improve the training convergence speed. Figure 7 As shown in the figure, the 2D convolutional BN layer is composed of 2D convolution + BatchNormalization layer + RELU. In the subsequent documents, it will be referred to as "convolutional BN layer".

[0066] 101 shallow feature extraction units:

[0067] like Figure 8 As shown in the figure, it includes 6 convolutional BN layers and 1 3*3 maximum pooling layer, where the stride of the first and last convolutional BN layers is 2. The shallow feature extraction unit is designed to extract simple features of the finger vein, which are then handed over to the deep feature extraction unit for further extraction.

[0068] 102 deep feature extraction units:

[0069] like Fig. 9As shown in the figure, from input to output, it consists of 5 A-type inception residual blocks, A-type feature distillation blocks, 10 B-type inception residual blocks, B-type feature distillation blocks, 6 C-type inception residual blocks, average pooling layers and Dropout layers.

[0070] The composition of the A-type inception residual block is as follows Fig.10 As shown: the first branch B1 consists of a convolutional BN layer T1 with a 1*1 convolution kernel, the second branch B2 consists of a convolutional BN layer T2 with a 1*1 convolution kernel and a convolutional BN layer T3 with a 3*3 convolution kernel, and the third branch B3 consists of a 1*1 convolutional BN layer T4 and two 3*3 convolutional BN layers T5 and T6. The fusion layer T7 fuses the outputs of T1, T3 and T6, and the output of T7 passes through the 1*1 convolutional BN layer T8. The fusion layer T9 fuses the output of T8 with the original input X, and then passes through the Relu layer as the final output. The final output is consistent with the input size.

[0071] The composition of the B-type inception residual block is as follows Fig.11 As shown: the first branch B1 consists of a convolutional BN layer T1 with a 1*1 convolution kernel, and the second branch B2 consists of a 1*1 convolutional BN layer T2, a 1*7 convolutional BN layer T3 and a 7*1 convolutional BN layer T4. The fusion layer T5 fuses the outputs of T1 and T4, and the output of T5 passes through a 1*1 convolutional BN layer T6. The fusion layer T7 fuses the output of T6 with the original input X, and then passes through the Relu layer as the final output. The final output is consistent with the input size.

[0072] The composition of the C-type inception residual block is as follows Fig.12 As shown: the first branch B1 consists of a convolutional BN layer T1 with a 1*1 convolution kernel, and the second branch B2 consists of a 1*1 convolutional BN layer T2, a 1*3 convolutional BN layer T3 and a 3*1 convolutional BN layer T4. The fusion layer T5 fuses the outputs of T1 and T4, and the output of T5 passes through the 1*1 convolutional BN layer T6. The fusion layer T7 fuses the output of T6 with the original input X, and then passes through the Relu layer as the final output. The final output is consistent with the input size.

[0073] The composition of the A-type characteristic distillation block is as follows Fig.13 As shown: the first branch B1 consists of a convolutional BN layer T1 with a 3*3 convolution kernel and a stride of 2, the second branch B2 consists of a convolutional BN layer T2 with a 1*1 convolution kernel and two convolutional BN layers T3 and T4 with a 3*3 convolution kernel, where T4 has a stride of 2, and the third branch B3 consists of a maximum pooling layer T5 with a pooling size of 3*3 and a stride of 2. The fusion layer T6 fuses the outputs of T1, T4 and T5 as the final output.

[0074] The composition of the B-type characteristic distillation block is as follows Fig.14 As shown: the first branch B1 consists of a 1*1 convolution BN layer T1 and a 3*3 convolution kernel, a convolution BN layer T2 with a stride of 2, the second branch B1 consists of a 1*1 convolution BN layer T3 and a 3*3 convolution kernel, a convolution BN layer T4 with a stride of 2, the third branch B2 consists of a 1*1 convolution kernel convolution BN layer T5 and two 3*3 convolution kernel convolution BN layers T6 and T7, where T7 has a stride of 2, and the fourth branch B3 consists of a 3*3 pooling size, a maximum pooling layer T8 with a stride of 2. The fusion layer T9 fuses the outputs of T2, T4, T7 and T8 as the final output.

[0075] 103 fully connected feature mapping units:

[0076] It consists of a 128-dimensional fully connected layer with a linear activation function, receives the output of 102 deep feature extraction units and outputs a 128-dimensional feature vector.

[0077] 104L2 regular unit:

[0078] The output of the 103 fully connected feature mapping units is normalized by L2 norm to obtain the 128-dimensional L2 distance, i.e., Euclidean distance.

[0079] 1.2. Workflow of feature extraction module:

[0080] 101 shallow feature extraction unit: The preprocessed 3D finger vein tensor information (size is 320*128*1) is input into the 101 shallow feature extraction unit, and 159*64*32 data is obtained after processing by the T1 convolution layer, 157*62*32 data is obtained after processing by the T2 convolution layer, 157*62*64 data is obtained after processing by the T3 convolution layer, 78*30*64 data is obtained after processing by the T4 maximum pooling layer, 78*30*80 data is obtained after processing by the T5 convolution layer, 76*28*192 data is obtained after processing by the T6 convolution layer, and 37*13*256 shallow feature extraction data is obtained after processing by the T7 convolution layer.

[0081] 102 deep feature extraction unit: The 37*13*256 shallow feature extraction data obtained by the T7 convolution layer is input into the 102 deep feature extraction unit, and then it passes through 5 A-type inception residual blocks to obtain 37*13*256 feature data, and then passes through the A-type feature distillation block to obtain 18*6*896 distilled feature data, and then passes through 10 B-type inception residual blocks to obtain 18*6*896 feature data, and then passes through the B-type feature distillation block to obtain 8*2*1792 distilled feature data, and then passes through 6 C-type inception residual blocks to obtain 8*2*1792 data. After the average pooling layer, 1792*1-dimensional feature data is output, and after the Dropout layer, 20% of the parameters are discarded to prevent overfitting, and the output is still 1792-dimensional deep feature data.

[0082] 103 fully connected feature mapping unit: The 1792-dimensional deep feature data is input into the 128-dimensional fully connected feature mapping unit to obtain a 128-dimensional finger vein feature vector.

[0083] 104L2 regularization unit: perform L2 norm normalization on the 128-dimensional finger vein feature vector to obtain the 128-dimensional L2 distance finger vein feature, which is the final mapping result of the feature extraction module.

[0084] 1.3. Feature extraction module benchmark model training scheme:

[0085] like Fig.15 The following figure shows the training method of the benchmark feature extraction model required for transfer learning. The model adopts the training method of the ternary pair network, which aims to make the mapping vectors of the same category close to each other, while the mapping vectors of different categories are far away from each other. After the training of the benchmark feature extraction model is completed, we will use it as a benchmark to transfer learning from the model parameters to the real finger vein data, and finally obtain the feature extraction model for the real finger vein. Fig.16 Compared with real finger vein data, it has the advantages of more complexity and theoretically unlimited data volume.

[0086] Preprocessing: The input data comes from our synthetic simulated finger vein dataset. The input images are uniformly converted to 320*128 size. Then the image brightness is normalized. The following formula is used to normalize the image brightness:

[0087] x=(xu) / σ,

[0088] Where x* is the normalized value. x is the pixel value of the original pixel point, u is the average value of all pixel values in the image, and σ is the variance of all pixel values in the image. Triplet grouping: The input is the normalized sample image and the corresponding class label. According to the grouping rule, it is combined into an input method with three samples in a group. The combination rule of the three samples is: one finger vein sample, another sample of the same category as it, and one sample of a different category from it. These three samples are abbreviated as Anchor (A for short), Positive (P for short), and Negative (N for short). The detailed description of the grouping rule used by the module is as follows:

[0089] Randomly sample from the sample set and separate by category, which is called the Sample set. Use the mapping model to map all the sampled samples to obtain the mapping set Embedding.

[0090] Starting from each category, pair them up in pairs to form (A, P) pairs first.

[0091] Remove the samples in the mapping set that are of the same category as A and P. Calculate the distance for the remaining samples with respect to A, and find all the negative example sample sets that meet the condition of inter-class distance - intra-class distance < alpha, where alpha is a hyperparameter set by humans. Then randomly select a sample from the negative example sample set to form the final (a, p, n) triplet. Repeat continuously until the entire Sample set is traversed to obtain a series of triplet sets. The image size is still 320*128, named Triplets.

[0092] Feature extraction: Input the flattened Triplets triplet set into the feature extraction module, and output a 128-dimensional feature vector group Output.

[0093] Backpropagation training: Rearrange Output into triplet form and use the triplet loss function for backpropagation:

[0094]

[0095] Among them, represents the distance between the anchor A and the positive example P, represents the distance between the anchor A and the negative example N, and α (alpha) is a hyperparameter set by humans (set to 0.2 in this patent). The goal of the triplet loss function is to make the inter-class distance greater than the intra-class distance by alpha, as Fig.17 shown. After training, the distance between the anchor A and the positive example P in the figure gradually decreases, while the distance between the anchor A and the negative example N gradually increases.

[0096] Train in this way until convergence to obtain a feature extraction model based on simulated finger veins.

[0097] 1.4. Transfer learning scheme for feature extraction module:

[0098] The transfer learning training framework adopted in this patent is based on the CycleGAN structure.

[0099] The source domain of transfer learning is our synthetic simulated finger vein data (abbreviated as Xs), such as Fig.16 The target domain is the true finger vein (Xt for short) collected by our equipment, as shown in Fig.18 Based on the trained feature extraction model for simulated finger veins (Ms for short), the model weights are transferred and trained to the feature extraction model for real finger veins (Mt for short) using the transfer learning architecture improved based on CycleGAN.

[0100] The full name of the GAN network is Generative Adversarial Network, which consists of a generator and a discriminator. The generator (abbreviated as G) is responsible for forging more realistic data, and the discriminator (abbreviated as D) is responsible for distinguishing forged data from real data, outputting 1 for real data and 0 for fake data. As the training progresses, the forging ability of the generator and the discrimination ability of the discriminator are enhanced, until the discriminator can no longer tell the difference between forged data and real data, and the training is completed. At this time, we will get a generator that can generate very realistic data, and it can generate a kind of data very well, which means that it can well reflect the laws of the data itself.

[0101] The loss function of the GAN network is shown in the following formula:

[0102]

[0103] Among them, X represents real data, and Z represents random noise. When interpreting this loss function, we need to look at it separately. When focusing on the discriminator D, the generator G is fixed. According to the loss function, the discriminator D will try to maximize V(G,D), requiring D(x) to be maximized and D(G(z)) to be minimized, that is, all real data are judged as 1, and all fake data are judged as 0; when focusing on the generator G, the discriminator D is fixed, and the generator will try to minimize V(G,D), requiring D(G(z)) to be maximized, that is, all fake data are judged as 1. We refer to the above loss as the adversarial loss of the GAN network.

[0104] CycleGAN is a classic image translation GAN network that aims to transfer image features from one domain to another.

[0105] The CycleGAN network consists of two mirror-symmetric GAN networks, forming a ring network. The CycleGAN network structure is as follows: Fig. 20 As shown in Figure 2, there are images from two domains with different features. CycleGAN attempts to use the generator GA-＞B Convert the data in domain A to domain B, and then use the generator G B-＞A Convert domain B back to domain A. There is also a reverse direction, using generator G B-＞A Convert from B to A, and then use generator G A-＞B Convert A back to B. The loss function is divided into three parts:

[0106]

[0107] Part 1: The above loss is the generator G A-＞B and the discriminator D B The adversarial loss, L adv It is an abbreviation for Against Loss.

[0108]

[0109] Part 2: The above loss is the generator G B-＞A and the discriminator D A against losses.

[0110]

[0111] Part 3: The above is the cycle consistency loss of CycleGAN, including: converting the A domain sample to the B domain and then converting it back to the A domain, the distance comparison loss between the new sample and the original sample, and the distance comparison loss in the other direction B->A->B. The sum of the two losses is the cycle consistency loss.

[0112] Based on CycleGAN, we transformed our transfer learning model, the structure is as follows Fig.21 As shown in the figure. We call domain A the source domain and name it S. Domain B is called the target domain and name it T. We add feature extraction models for the source domain and the target domain respectively. The feature extraction model for the source domain is the “feature extraction module benchmark model” described above. The two feature extraction models have exactly the same structure but different parameters. They are trained based on the same loss function, referred to as L. task , which is the "ternary pair loss function" described above. In the network structure of transfer learning, the cycle consistency loss in CycleGAN is removed, and CycleGAN and the feature extraction model are jointly trained. The specific training steps are as follows:

[0113] The network input is: source domain dataset X s (Simulated finger vein), target domain dataset X t (real finger vein), pre-trained feature extraction model M for the source domain (simulated finger vein) S

[0114] The network output is: the trained feature extraction model M for the target domain (true finger vein) T .

[0115] Step 1: X s and X t Sampling

[0116] Step 2: Use the joint loss function formula 1 to T→S ,D S ,M S Conduct joint training.

[0117]

[0118] Joint loss function formula 1

[0119] First line of the formula: standard CycleGAN adversarial loss, discriminator-generator adversarial loss for converting target domain to source domain data.

[0120] Second row: The loss of the source domain feature extraction model trained using source domain data and labels.

[0121] Third row: Using CycleGAN generator G T→S The target domain image X t Mapped to source domain data X s ', then use X s ' and its original label, the loss of training the source domain feature extraction model.

[0122] Step 2 described above is joint training, which is back-propagation training based on the loss sum of the joint loss function formula 1.

[0123] Step 3: Use the joint loss function formula 1 to S→T ,D T ,M T Conduct joint training.

[0124]

[0125] Joint loss function formula 1

[0126] First line of the formula: Standard CycleGAN adversarial loss, discriminator-generator adversarial loss for converting source domain to target domain data.

[0127] The second line of the formula: The loss of the target domain feature extraction model. The loss of the target domain feature extraction model is trained using the target domain data and labels.

[0128] Formula line 3: Using CycleGAN generator G S→T The source domain image XS Mapped to simulation target domain data X t ', then use X t ' and its original label, training the loss of the target domain feature extraction model.

[0129] Step 3 described above is joint training, which is back-propagation training based on the loss sum in the joint loss function formula 2.

[0130] Step 4: Repeat steps 1, 2, and 3 until the target domain feature extraction model converges.

[0131] Through the above transfer learning process, the feature extraction model of the target domain (true finger vein) is finally obtained.

[0132] 2. Image acquisition preprocessing module:

[0133] The image acquisition preprocessing module of the VARS system is as follows: Fig. 22 As shown, including 201, 202:

[0134] Image acquisition unit 201: responsible for acquiring finger vein images. Fig.23 As shown in the figure, the unit is mainly composed of a near-infrared light platform and a charge-coupled device (CCD) preprocessor camera. After the finger is placed flat on the near-infrared light platform, the near-infrared light irradiates the entire collection area. The hemoglobin in the blood vessels of the finger will absorb more near-infrared light, while other tissues absorb less. The near-infrared image is acquired by the CCD camera, and the blood vessel part of the final image will appear as a darker line, forming a light and dark contrast with other parts. The final image size acquired is 400*200.

[0135] Preprocessing unit 202: responsible for cropping the finger vein region of interest (ROI), unifying the size, and normalizing the brightness. After receiving the image from the image acquisition unit 201, find the upper and lower edges of the finger contour, fit the center line of the finger direction, rotate the finger direction to horizontal, and crop according to the inner tangent line of the finger edge. The samples after cropping are of different sizes, and the size is unified to 320*128. The effect after cropping is as follows Fig.19 As shown. Then use the following formula to normalize the image's luminance:

[0136] x * =(xu) / σ,

[0137] where x * is the normalized value. x is the original pixel value, u is the mean pixel value of the whole image, and σ is the variance of the pixel value of the whole image. After completion, the final image result is output.

[0138] 3. Sample quality assessment module:

[0139] The sample quality assessment module of the VARS system is as follows: Fig.24 Shown:

[0140] Quality evaluation unit 301: Based on the contrast and information entropy of the image, the image is scored respectively, and then the two scores are used for comprehensive scoring. Images with less than 70 points are judged as having poor image quality.

[0141] Contrast quality score: We mainly use the mean square error of the grayscale of the image to judge the contrast score.

[0142] The grayscale mean square error C of the image is based on the following formula:

[0143]

[0144] Among them, C is the grayscale mean square error of the image, x i is the gray value of each pixel in the image, N is the total number of pixels in the image, is the average grayscale value of the image.

[0145] Based on the finger vein data collected by us, when the grayscale mean square error of the image is greater than 45 (this threshold may change with different collected image libraries), the finger vein quality is high and normal finger vein recognition can be performed. The solution method for the contrast quality score is:

[0146] S1=S_contract=100 when C>45

[0147] S1=S_contract=C / 45*100 when C≤45

[0148] Information entropy quality score:

[0149] The information entropy of an image represents the aggregation characteristics of the grayscale distribution of the image. The larger the information entropy of an image, the more information the image contains and the richer it is. For finger vein images, the more significant the texture features of the image, the greater the information entropy of the image. The method for calculating the information entropy of an image is:

[0150]

[0151] Among them, p(i) is the probability of each gray level appearing in the image, and S is the image information entropy.

[0152] Based on the finger vein data collected by us, when the image information entropy is greater than 5 (this threshold may change with different collection image libraries), the quality of the finger vein image is high and normal finger vein recognition can be performed. Therefore, the solution method for the information entropy quality score S_comentropy is:

[0153] S2=S_comentropy=100 when S_comentropy>5

[0154] S2=S_comentropy=S / 5*100 when S_comentropy≤5

[0155] Overall evaluation score:

[0156] After obtaining the contrast quality score and information entropy quality score according to the above method, it is necessary to comprehensively consider the parameter scores of these two aspects and add them up according to a certain weight to obtain the final overall evaluation score.

[0157] From the solution methods of the above two indicators, it can be seen that the lower the score of each indicator, the worse the image quality. When calculating the overall evaluation score, we need to first consider the impact of a single indicator on the total score, and the weight of a single indicator should also be affected by other indicators. Based on this consideration, the solution method for the finger vein comprehensive quality score S_vein in this article is:

[0158]

[0159] When the comprehensive score is less than 70 points, the quality evaluation unit 301 determines that the picture is of poor quality.

[0160] 4. Deception Detection Module:

[0161] like Fig.25 As shown, the deception detection module of the VARS system includes 401, 402:

[0162] Feature extraction unit 401: responsible for extracting 128-dimensional features of the target finger vein image, with a preprocessed finger vein image of 320*128 size as input and a 128-dimensional feature vector as output. The network structure of feature extraction unit 401 is a series structure of shallow feature extraction unit 101 and deep feature extraction unit 102 of the feature extraction module, which will not be described in detail here.

[0163] True and false vein classification unit 402: The input is a 128-dimensional feature vector. Based on the SVM classifier, it determines whether the sample is from a true vein or a forged finger vein, and outputs the determination result, 0 represents a forged finger vein, and 1 represents a true finger vein.

[0164] Training scheme for the deception detection module:

[0165] Prepare true finger vein images: directly use the collected true finger vein images with the label 1.

[0166] Prepare to forge a finger vein image: Use a scan of a real vein image, print it on matte coated paper at a resolution of 2400dpi, and then scan it into an electronic image as a forged finger vein image with a label of 0.

[0167] The true and false finger vein sets are divided into a training set and a test set at an 8:2 ratio. First, the feature extraction unit 401 is trained with the loss function of Softmaxloss until the loss converges to obtain the trained feature extraction unit 401. The feature extraction unit 401 is used to generate a corresponding number of 128-dimensional feature vectors for the training set, and the true and false vein classification unit 402 is trained with the 128-dimensional feature vectors and the corresponding true and false finger vein labels to perform SVM model training until the accuracy of the SVM reaches the optimal level. Finally, the test set is tested, and the classification accuracy of the test set can reach 99.993%. The weight parameters of the feature extraction unit 401 and the classification unit 402 are solidified, and the deception detection model is trained at this point.

[0168] 5. Identity authentication module:

[0169] like Fig.26 As shown, the identity authentication module of the VARS system includes 501:

[0170] Identity matching unit 501: Receives identity feature distribution information of all existing internal personnel from the data storage unit, where each identity corresponds to the center point coordinates of a circle (128 dimensions) and the radius of the circle; receives a 128-dimensional vector from the feature extraction model, i.e., the feature extraction result from a finger vein image, which is a feature point (128 dimensions).

[0171] Based on the Euclidean distance, the coverage of the feature point and each circle is judged, which is divided into the following three cases: If the point is not covered by any circle, then the sample is judged as an unknown identity (outsider) and -1 is output. If the point is only covered by one circle, then the sample is judged as the identity corresponding to the circle, and the output is the number and name corresponding to the identity. If the point is covered by multiple circles, then it is judged as the category corresponding to the circle with the closest center to the point, and the number and name corresponding to the identity are output.

[0172] 6. Data storage module:

[0173] The data storage module includes 601-602:

[0174] Data processing unit 601: receives input and processes data from other modules, and writes it into storage unit 602.

[0175] Storage unit 602: data storage location, storage medium is Micro SD card

[0176] The workflow of the data storage module is as follows:

[0177] Storing encrypted images: The data processing unit 601 receives the finger vein image output from the sample quality assessment module, encrypts the image using the self-developed garbled code encryption technology, and then writes it into the storage unit 602.

[0178] Storing identity feature distribution information: The data processing unit 601 receives 50 128-dimensional feature vectors of the newly added users from the feature extraction module, performs kmeans clustering, clusters them into two clusters, and retains the main cluster. Treat each feature vector as a point in a 128-dimensional space, find the center point of these points, and then find the distance from each point to the center point. Take the median of these distances multiplied by the radius ratio (hyperparameter, which can be adjusted manually) as the radius to form a circle to approximate the normal distribution of the newly added user. The center point and radius of the circle are used as the feature distribution information of the identity to generate an identity number. The feature distribution information, identity number and identity name entered by the user are packaged together and written into the storage unit 602.

[0179] Storage system identification records: The data processing unit 601 receives the forged input records from the fraud detection module and the admission / rejection records from the identity authentication module, and directly writes them into the storage unit 602 without processing.

[0180] Read identity feature distribution information: The data processing unit 601 reads the feature distribution information of all internal personnel from the storage unit 602 and outputs it to the identity authentication module.

[0181] Reading system identification records: The data processing unit 601 reads the existing system identification records from the storage unit 602 and outputs them to the system user interface.

[0182] This example is only used to illustrate the present invention. The implementation of the VARS device and system is not limited to this example. For researchers in this field, it is entirely possible to modify and optimize the device and system described in the present invention. The essence of these principles is the same as the content of the present invention, and the changes in different forms of expression all fall within the scope of protection of the present invention.

Claims

1. A finger vein accurate identification system (VARS) based on transfer learning and GAN network, characterized in that: Contains the following modules: (1) Image acquisition and preprocessing module: responsible for the acquisition of finger veins and the preprocessing of the acquired images; after the finger vein acquisition device images, a finger vein image with a size of 400*200 is obtained, and the ROI is cropped according to the finger contour, and then the image is resized and normalized; (2) Sample quality assessment module: responsible for assessing the quality of collected and preprocessed images; The input is the finger vein image preprocessed by the image acquisition preprocessing module. After quality evaluation, if the image quality meets the standard, the input finger vein image is output; If it is determined that the image quality is not good enough and the vein information is insufficient, the current workflow is terminated here and no output is sent to the next module; (3) Deception detection module: responsible for determining whether the input image is a fake finger vein or a real finger vein; The input is the finger vein image evaluated by the sample quality assessment module. If the image is determined to be a real finger vein, the input finger vein image is output; If it is determined to be a forged finger vein, otherwise the current workflow is terminated here, the system alarms, and the forged input record and timestamp are output to the data storage module as a system identification record; (4) Feature extraction module: This module is trained through transfer learning and is responsible for extracting finger vein features from images. It outputs a 128-dimensional finger vein feature vector for each image. The input is the finger vein image output by the sample quality assessment module. After passing through the feature extraction model, a 128-dimensional finger vein feature vector is output. The feature extraction process supports batch processing; The benchmark model training method of the feature extraction module of the module (4) specifically includes: (1) Triple pair grouping: The input is the normalized sample image and the corresponding category label. According to the grouping rule, three samples are grouped into one input method. The combination rule of the three samples is: a finger vein sample, another sample of the same category, and a sample of a different category. These three samples are referred to as anchors (Anchor, abbreviated as A), positive examples (Positive, abbreviated as P), and negative examples (N). The grouping rule used by the module is: Randomly sample from the sample set and separate by category, called the Sample set; use the mapping model to map all sampled samples to obtain the mapping set Embedding; Starting from each category, two by two are paired to form (A, P) pairs first; remove the samples of the same category as A and P in the mapping set, calculate the distance of the remaining samples for A, and find all negative sample sets that meet the condition of inter-class distance - intra-class distance α<(alpha), where α(alpha) is a manually set hyperparameter; then randomly select a sample from the negative sample set to form the final (a, p, n) triple pair; repeat until the entire sample set is traversed, and obtain a series of triple pairs, with the image size still being 320*128, named Triplets; (2) Feature extraction: Input the flattened Triplets into the feature extraction module, and output a 128-dimensional feature vector group Output; Back propagation training: Rearrange the Output into a triplet form, and use the triplet loss function for back propagation: in, represents the distance between the anchor point A and the positive example P, Represents the distance between the anchor point A and the negative example N, α (alpha) is a hyperparameter set artificially; (5) Identity authentication module: responsible for determining the identity of the input feature vector; this module has two inputs: a. 128-dimensional finger vein features output by the feature extraction module, b. identity feature distribution information of all internal personnel output by the data storage module; based on the Euclidean distance comparison between a and b, the identity to be verified is determined; output -1 indicates an unknown outsider, and output the corresponding identity number indicates a known person; the determination result and timestamp will be transmitted to the data storage module as a system identification record; (6) Data storage module: responsible for storing and reading finger vein feature information, encrypting finger vein samples and system identification records; this module has five tasks:

1. Storing encrypted images: receiving the finger vein images output by the sample quality assessment module, encrypting the images, and storing the encrypted image files; 2. Storing identity feature distribution information: receiving multiple feature vectors of newly added users from the feature extraction module, generating feature distribution information of the identity after calculation and processing, and packaging and storing it with the user input name; 3. Store system identification records: records of forged input from the deception detection module and access / rejection records from the identity authentication module, including timestamps. This module stores the original records line by line; 4. Read identity feature distribution information: read the feature distribution information of all internal personnel and output it to the identity authentication module; 5. Read system identification records: read existing system identification records and output them to the system user interface for system administrators to view.

2. The VARS according to claim 1, characterized in that: The network structure of the feature extraction module of module (4) specifically includes: (1) The 2D convolutional BN layer is used to replace the ordinary 2D convolutional layer to improve the training convergence speed; the 2D convolutional BN layer is composed of 2D convolution + BatchNormalization layer + RELU; (2) It consists of a shallow feature extraction unit, a deep feature extraction unit, a fully connected feature mapping unit, and an L2 regularization unit; the shallow feature extraction unit consists of 6 convolutional BN layers and 1 3*3 maximum pooling layer, where the stride of the first and last convolutional BN layers is 2; the 102 deep feature extraction unit consists of 5 A-type inception residual blocks, A-type feature distillation blocks, 10 B-type inception residual blocks, B-type feature distillation blocks, 6 C-type inception residual blocks, average pooling layers, and Dropout layers; (3) Composition of type A inception residual block: The first branch B1 consists of a convolutional BN layer T1 with a 1*1 convolution kernel, the second branch B2 consists of a convolutional BN layer T2 with a 1*1 convolution kernel and a convolutional BN layer T3 with a 3*3 convolution kernel, and the third branch B3 consists of a 1*1 convolutional BN layer T4 and two 3*3 convolutional BN layers T5 and T6; the fusion layer T7 fuses the outputs of T1, T3 and T6, and the output of T7 passes through the 1*1 convolutional BN layer T8. The fusion layer T9 fuses the output of T8 with the original input X. , and then pass through the Relu layer as the final output; (4) Composition of the B-type inception residual block: The first branch B1 consists of a convolutional BN layer T1 with a 1*1 convolution kernel, and the second branch B2 consists of a 1*1 convolutional BN layer T2, a 1*7 convolutional BN layer T3 and a 7*1 convolutional BN layer T4. The fusion layer T5 fuses the outputs of T1 and T4, and the output of T5 passes through the 1*1 convolutional BN layer T6. The fusion layer T7 fuses the output of T6 with the original input X, and then passes through the Relu layer as the final output; (5) Composition of the C-type inception residual block: The first branch B1 consists of a convolutional BN layer T1 with a 1*1 convolution kernel, and the second branch B2 consists of a 1*1 convolutional BN layer T2, a 1*3 convolutional BN layer T3, and a 3*1 convolutional BN layer T4. The fusion layer T5 fuses the outputs of T1 and T4, and the output of T5 passes through a 1*1 convolutional BN layer T6. The fusion layer T7 fuses the output of T6 with the original input X, and then passes through the Relu layer as the final output; (6) Composition of the A-type feature distillation block: The first branch B1 consists of a convolutional BN layer T1 with a 3*3 convolution kernel and a stride of 2. The second branch B2 consists of a convolutional BN layer T2 with a 1*1 convolution kernel and two convolutional BN layers T3 and T4 with a 3*3 convolution kernel, where T4 has a stride of 2. The third branch B3 consists of a maximum pooling layer T5 with a pooling size of 3*3 and a stride of 2. The fusion layer T6 fuses the outputs of T1, T4 and T5 as the final output. (7) Composition of the B-type feature distillation block: The first branch B1 consists of a 1*1 convolutional The BN layer T1 is composed of a 3*3 convolution kernel and a convolution BN layer T2 with a stride of 2. The second branch B1 is composed of a 1*1 convolution BN layer T3 and a 3*3 convolution kernel and a convolution BN layer T4 with a stride of 2. The third branch B2 is composed of a convolution BN layer T5 with a 1*1 convolution kernel and two convolution BN layers T6 and T7 with 3*3 convolution kernels, where T7 has a stride of 2. The fourth branch B3 is composed of a maximum pooling layer T8 with a pooling size of 3*3 and a stride of 2. The fusion layer T9 fuses the outputs of T2, T4, T7 and T8 as the final output. (8) Fully connected feature mapping unit: The 1792-dimensional deep feature data is input into the 128-dimensional fully connected feature mapping unit to obtain a 128-dimensional finger vein feature vector; (9) L2 regularization unit: The 128-dimensional finger vein feature vector is normalized by L2 norm to obtain a 128-dimensional L2 distance finger vein feature.

3. The VARS according to claim 1, characterized in that: The transfer learning training scheme of the feature extraction module of module (4) specifically includes: (1) Network framework: Based on CycleGAN, a transfer learning model is constructed. Domain A in CycleGAN is called the source domain and named S. Domain B is called the target domain and named T. Feature extraction models are added to the source domain and target domain respectively. The two feature extraction models have the same structure but different parameters. They are trained based on the same loss function, referred to as L. task ; Remove the cycle consistency loss in CycleGAN in the network structure of transfer learning, and jointly train CycleGAN and the feature extraction model; (2) Training method: Feature extraction model M of the target domain T ; Step 1: X s and X t Sampling Step 2: Use the joint loss function formula 1 to T→S ,D S ,M S Conduct joint training; The joint loss is the sum of three parts: the discriminator-generator adversarial loss for converting the target domain to the source domain data, the loss of training the source domain feature extraction model using the source domain data and labels, and the loss of using the CycleGAN generator G T→S The target domain image X t Mapped to source domain data X s ', then use X s ' and its original label, the loss of training the source domain feature extraction model; Step 3: Use the joint loss function formula 1 to S→T ,D T ,M T Conduct joint training; The joint loss is the sum of three parts: the discriminator-generator adversarial loss for converting source domain to target domain data, the loss of training the target domain feature extraction model using target domain data and labels, and the loss of using the CycleGAN generator G S→T The source domain image X S Mapped to simulation target domain data X t ', then use X t ' and its original label, the loss of training the target domain feature extraction model; Step 4: Repeat steps 1, 2, and 3 until the target domain feature extraction model converges; Through the above transfer learning process, the feature extraction model of the target domain is finally obtained.

4. The VARS according to claim 1, characterized in that: The module (1) image acquisition preprocessing module specifically includes: (1) Unify the sample size to 320*128 (2) The following formula normalizes the image brightness: x * =(xu) / σ, Among them, x* is the normalized value, x is the original pixel value, u is the mean pixel value of the whole image, and σ is the variance of the pixel value of the whole image.

5. The VARS according to claim 1, characterized in that: The sample quality assessment module of module (2) specifically solves: (1) Contrast quality score: The contrast score is judged by the mean square error of the grayscale of the image. Among them, C is the grayscale mean square error of the image, x i is the gray value of each pixel in the image, N is the total number of pixels in the image, is the grayscale average of the image; the solution method for the contrast quality score is: S1=S_contract=100when C>45 S1=S_contract=C / 45*100 when C≤45 (2) Information entropy quality score: Among them, p(i) is the probability of each gray level appearing in the image, and S is the image information entropy; The solution method for the information entropy quality score S_comentropy is: S2=S_comentropy=100when S_comentropy>5 S2=S_comentropy=S / 5*100when S_comentropy≤5 (3) The solution method for the comprehensive quality score of the finger vein S_vein is: 。 6. The VARS according to claim 1, characterized in that: The module (5) identity authentication module specifically performs: (1) Identity matching method: Receive the identity feature distribution information of all existing internal personnel from the data storage unit, where each identity corresponds to the coordinates of the center point of a circle and the radius of the circle; receive a 128-dimensional vector from the feature extraction model, that is, the feature extraction result from a finger vein image, which is a feature point; Based on the Euclidean distance, the coverage of the feature point and each circle is judged, which is divided into the following three cases: if the point is not covered by any circle, then the sample is judged as an unknown identity and -1 is output; if the point is only covered by one circle, then the sample is judged as the identity corresponding to the circle, and the output is the number and name corresponding to the identity; if the point is covered by multiple circles, then it is judged as the category corresponding to the circle with the center closest to the point, and the number and name corresponding to the identity are output.

7. The VARS according to claim 1, characterized in that: The module (6) data storage module specifically performs: (1) Receive 50 128-dimensional feature vectors of the new user from the feature extraction module, perform kmeans clustering, cluster them into two clusters, and retain the main cluster; regard each feature vector as a point in a 128-dimensional space, find the center point of these points, and then find the distance from each point to the center point. Take the median of these distances multiplied by the radius ratio as the radius to form a circle to approximate the normal distribution of the new user; use the center point and radius of the circle as the feature distribution information of the identity to generate an identity number.

Images