A multimodal recognition method based on fingerprint and finger vein
By introducing multimodal recognition methods of fingerprint and finger vein in the biometric system, the feature-level fusion is used to use convolutional neural network and attention mechanism to solve the problem of low performance of a single biometric recognition system and achieve high accuracy Biometric recognition.
Patent Information
- Application Number
- CN202210671731.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-06-15
AI Technical Summary
Biometric systems based on single biometrics are susceptible to noise data, have low performance, and are susceptible to theft and counterfeiting attacks.
Multimodal recognition method based on fingerprint and finger vein is used to extract features through convolutional neural networks, and feature-level fusion is introduced to fuse fingerprint and finger vein features, and the final biometric recognition result is determined using the Softmax function.
The accuracy of the biometric recognition system has been significantly improved. By mining the complementary information between different biometric features, the discriminant characteristics have been improved, reaching a recognition accuracy of 95.41%.
Smart Images

Figure CN114913610B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of biometric identification technology, and specifically relates to a multimodal identification method based on fingerprints and finger veins, aiming to improve the accuracy of a biometric feature identification system. Background Art
[0002] Traditional identity recognition technologies are generally divided into two types: one is identity authentication based on identity identifiers, such as keys, ID cards, etc.; the other is identity authentication based on identity knowledge, such as passwords, user numbers, etc. However, they both have the disadvantages of being easily lost and forgotten. With the development of science and technology, researchers have proposed biometric recognition technology based on human biological characteristics. Among the many human biological characteristics, fingerprints are one of the most widely used and oldest biological characteristics of the human body, and are also the biometric modality with the largest commercial market share. Finger vein recognition is an emerging technology that has received widespread attention in recent years. It incorporates in vivo characteristics and has the advantages of being difficult to steal, in vivo imaging, and strong stability. In actual applications, it is found that the performance of biometric recognition systems based on single biological characteristics is easily affected by noise data, and single characteristics are also susceptible to theft and counterfeit attacks.
[0003] Fingerprint or finger vein recognition technology can generally be divided into two categories: one is based on traditional image processing algorithms, and the other is related algorithms based on deep learning. The first type of algorithm generally captures the correlation of objects within a class and the irrelevance of objects between classes by describing the relationship between image pixels and surrounding pixels, such as gradients and position information, or recognizes by increasing the distance between the recognized object and other objects. This type of method mainly includes Local Binary Pattern (LBP), Histogram of Oriented Gradient (HOG), Adaptive Radius Local Binary Pattern (ADLBP), etc. The second type of algorithm generally extracts high-dimensional features from the input data through the convolution operation in the Convolutional Neural Network (CNN), and then classifies the data according to the features. CNN can better fit the sample characteristics of biometrics, so it is more and more widely used in biometric recognition.
[0004] The fusion of multimodal biometric features can be divided into three categories according to the different fusion stages. One is pixel-level fusion, which fuses the features between multiple modalities at the beginning of the neural network extracting input data. Another is feature-level fusion, which performs feature fusion during or at the end of the neural network extracting input data. The last is decision-level fusion, which performs decision fusion after the neural network extracts features and makes a decision. When pixel-level fusion is performed, each feature does not learn enough high-level discriminant features, and the recognition performance is relatively poor. Decision-level fusion is performed after each feature makes a decision, lacks interaction between features, and the recognition performance is also average. Summary of the invention
[0005] Based on the above analysis, the present invention proposes a multimodal recognition method based on fingerprints and finger veins. After extracting features through a convolutional neural network, feature-level fusion is adopted, and an attention mechanism is introduced to fuse different biometric features to improve the performance of the biometric recognition system. First, the present invention uses the collected fingerprint and finger vein images as independent inputs of the network for feature extraction and analysis. Secondly, the present invention analyzes the correlation between the fingerprint and the corresponding finger vein from the dimensions of channel and space through the attention mechanism, and fuses the fingerprint and finger vein features. On this basis, the Softmax function is used to determine the final biometric recognition result.
[0006] A multimodal recognition method based on fingerprint and finger vein, characterized in that the method comprises the following steps:
[0007] Step 1: Obtain fingerprint and finger vein image data; record the fingerprint and finger vein image data of the person being collected through the collection device;
[0008] Step 2: Extract the corresponding region of interest (ROI) from all the data obtained in step 1;
[0009] Step 3: Build a network model, perform feature extraction, fusion and classification, and use the collected data obtained in step 2 to train the model to obtain the best model parameters; the network model mainly includes two branch networks and a multi-feature fusion module;
[0010] Step 3.1: Construct a branch network; the first branch network is used to extract fingerprint features, including multiple convolution, pooling, normalization, and activation function layers; the fingerprint ROI image extracted in step 2 is input into the network to extract features; the second branch network is used to extract finger vein features, including multiple convolution, pooling, normalization, and activation function layers; the finger vein ROI image extracted in step 2 is input into the network to extract features;
[0011] Step 3.2: Construct a multi-feature fusion module; this module accepts the output of the two branch networks as input, and then uses the channel and spatial attention mechanisms to fuse the features of different modalities and output them;
[0012] Step 3.3: Use the collected fingerprint and finger vein image data to train the above network model to obtain the optimal model parameters;
[0013] Step 4: Apply the model; input the fingerprint and finger vein image to be identified into the network model, and the model outputs the corresponding identification results.
[0014] Furthermore, in step 1, the acquisition device used is a high-definition camera.
[0015] Furthermore, in step 3.1, the first branch network contains 5 convolutional layers, of which the first convolutional layer contains 1 7×7 convolutional layer, 1 normalization layer, 1 activation function layer, and 1 3×3 maximum pooling layer, and the remaining four convolutional layers each contain 1 3×3 convolutional layer, 1 normalization layer, 1 activation function layer, and 1 3×3 maximum pooling layer.
[0016] Furthermore, in step 3.1, the second branch network is used to extract the feature information matrix corresponding to the finger vein, which includes 5 convolutional layers and 1 adaptive pooling layer, wherein the first convolutional layer includes 1 7×7 convolution, 1 normalization, 1 activation function, and 1 3×3 maximum pooling, and the remaining four convolutional layers each include 1 3×3 convolution, 1 normalization, 1 activation function, and 1 3×3 maximum pooling.
[0017] Furthermore, in step 3.1, the dimensions of the outputs of the two branch networks are both 512×3×6, which respectively represent the fingerprint and finger vein feature information matrices extracted by the CNN network.
[0018] Furthermore, in step 3.2, the multi-feature fusion module includes a channel attention module, a spatial attention module and two sigmoid layers, wherein the channel attention module includes 2 1×1 convolutional layers, 1 activation function layer, and 1 adaptive pooling layer, and the spatial attention module includes 2 7×7 convolutional layers, 2 normalization layers, 1 activation function layer, and 1 sigmoid layer.
[0019] Furthermore, in step 3.2, in the multi-feature fusion module, the outputs of the two branch network modules are first taken as input, and after the initial feature aggregation IFI, the matrix F is obtained. initial , and its calculation formula is shown in formula (1):
[0020]
[0021] Among them, F fpis the fingerprint feature information matrix, F fv is the finger vein feature information matrix, stands for element-by-element addition;
[0022] F initial Input to the channel attention module to get the matrix F c ; F c After inputting sigmoid1, the fingerprint vein fusion coefficient F of the channel attention module is obtained c_final and 1-F c_final ;
[0023] F c Input to the spatial attention module to get another matrix F s ; F s After inputting sigmoid2, the fingerprint and vein fusion coefficient F obtained by the spatial attention module is obtained. s_final and 1-F s_final ;
[0024] After the above steps, the output result after the fusion of fingerprint and finger vein features is calculated according to formula (2):
[0025]
[0026] in, represents element-wise multiplication, Stands for element-wise addition.
[0027] Furthermore, in step 3.3, the Adam optimizer is used to calculate the adaptive learning rate of each parameter, and then the recognition performance of the network model is determined by the test set, the optimal parameters are determined, and the optimal network model obtained by the final training is saved.
[0028] Furthermore, in step 4, the network model includes two branch networks, a multi-feature fusion module, a fully connected layer and a Softmax classifier in sequence.
[0029] Beneficial effects of the present invention:
[0030] (1) The preprocessing operation is simple, and only the ROI of fingerprints and finger veins needs to be extracted, which can maximize the preservation of detailed features in the image and reduce the consumption of additional computer resources.
[0031] (2) Compared with the existing biometric recognition method using a single biological modality, the present invention proposes to fuse fingerprints and finger vein images, which can significantly increase the discriminative features of biometrics and improve the accuracy of the biometric recognition system by mining the complementary information between different biometrics. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is an implementation flow chart of the multimodal recognition method based on fingerprint and finger vein in an embodiment of the present invention.
[0033] Figure 2 Schematic diagram of the model architecture of the multimodal recognition network in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.
[0035] The multimodal recognition method proposed in the present invention has the following process: Figure 1 The specific implementation steps of the present invention are as follows.
[0036] Step 1: Obtain fingerprint and finger vein image data. Use a high-definition camera to collect fingerprint and finger vein information of the individual being tested through the collection device.
[0037] Step 2: Extract ROI from the original fingerprint and finger vein image data. For each original fingerprint and finger vein image, manually locate and capture its ROI area.
[0038] Step 3: Build Figure 2 The multimodal recognition network model shown in Figure 1 performs feature extraction, fusion and classification. The network model is based on a convolutional neural network and mainly consists of three parts: two branch networks and a multi-feature fusion module.
[0039] Branch network 1 is used to extract the feature information matrix corresponding to the fingerprint, which contains 5 convolutional layers. The first convolutional layer is composed of a 7×7 convolutional layer, a normalization layer, an activation function layer, and a 3×3 maximum pooling layer. The remaining four convolutional layers are composed of a 3×3 convolutional layer, a normalization layer, an activation function layer, and a 3×3 maximum pooling layer. The fingerprint ROI image obtained in step 2 is used as the input of the network flow. After passing through branch network 1, the feature information matrix corresponding to the fingerprint is output.
[0040] Branch network 2 is used to extract the feature information matrix corresponding to the finger vein, which contains 5 convolution layers and 1 adaptive pooling layer in sequence. The first convolution layer is composed of 1 7×7 convolution, 1 normalization, 1 activation function, and 1 3×3 maximum pooling. The remaining four convolution layers are composed of 1 3×3 convolution, 1 normalization, 1 activation function, and 1 3×3 maximum pooling in sequence. The finger vein ROI image obtained in step 2 is used as the input of the network flow. After passing through branch network 2, the feature information matrix corresponding to the finger vein is output.
[0041] The multi-feature fusion module accepts the output of two branch networks. The output dimension of each branch network is 512 (number of channels) × 3 (height) × 6 (width), which respectively represent the fingerprint and finger vein feature information matrices extracted by the CNN network. In the multi-feature fusion module, the output of the two branch network modules is first used as input, and after initial feature integration (IFI), the matrix F is obtained. initial The calculation formula is shown in formula (1):
[0042]
[0043] Among them, F fp is the fingerprint feature information matrix, F fv is the finger vein feature information matrix, Stands for element-wise addition.
[0044] F initial The dimension is 512 (number of channels) × 3 (height) × 6 (width), and then it passes through the adaptive pooling layer in the channel attention module, the first 1×1 convolution layer, the activation function layer, and the second 1×1 convolution layer to obtain a matrix F with a dimension of 512 (number of channels) × 3 (height) × 6 (width) c_a Finally, F initial With F c_a Multiply element by element to get the matrix F c . F c The calculation formula is shown in formula (2):
[0045]
[0046] in, Stands for element-wise multiplication.
[0047] The sigmoid function is often used as an activation function of a neural network to map variables between 0 and 1. c After inputting sigmoid1, the fingerprint vein fusion coefficient F of the channel attention module is obtained c_final and 1-F c_final , corresponding to Figure 2 The outputs are indicated by solid and dashed arrows.
[0048] F c The dimension is 512 (number of channels) × 3 (height) × 6 (width), and then it passes through the first 7×7 convolution layer, normalization layer, activation function layer, second 7×7 convolution layer, sigmoid layer in the spatial attention module in turn, and obtains a matrix F with a dimension of 512 (number of channels) × 3 (height) × 6 (width) s_a Finally, F c With Fs_a Multiply element by element to get another matrix F s . F s The calculation formula is shown in formula (3):
[0049]
[0050] F s After inputting sigmoid2, the fingerprint and vein fusion coefficient F obtained by the spatial attention module is obtained. s_final and 1-F s_final , corresponding to Figure 2 The outputs are indicated by solid and dashed arrows.
[0051] After the above steps, the output result (Output) after the fusion of fingerprint and finger vein features is calculated according to formula (4).
[0052]
[0053] in, represents element-wise multiplication, Stands for element-wise addition.
[0054] The Output is input into the fully connected layer and then passes through the Softmax classifier to obtain the final classification result.
[0055] The model is trained using the ROI data manually located and captured in step 2. During training, the Adam optimizer is used to calculate the adaptive learning rate of each parameter, and then the recognition performance of the network model is determined through the test set, the optimal parameters are determined, and the optimal network model obtained by the final training is saved.
[0056] Step 4: Apply the model. Input the fingerprint and finger vein images to be identified into the network model. First, extract the ROI area of the fingerprint and finger vein images. Then input the data into two branch networks respectively. After the multi-feature fusion module and the fully connected layer, the classification result is obtained through the Softmax classifier.
[0057] The present invention proposes a multimodal recognition method based on fingerprint and finger vein. To verify its performance, the present method is compared with multiple methods on the same data set. The comparison method is as follows.
[0058] (1) Use LBP (Local Binary Pattern) as the image feature descriptor, use feature information matrix splicing as the multi-feature fusion method, and use the nearest neighbor rule classification (KNearestNeighbors, KNN) as the classifier.
[0059] (2) Use HOG (Histogram Of Oriented Gradient) as the image feature descriptor, use feature information matrix splicing as the multi-feature fusion method, and use KNN as the classifier.
[0060] (3) Use ADLBP (Adaptive Radius Local Binary Pattern) as the image feature descriptor, use feature information matrix splicing as the multi-feature fusion method, and use KNN as the classifier.
[0061] (4) Local Coding Based Convolutional Neural Network (LCCNN) based on deep learning (Li S, Zhang B, Zhao S, et al. Local discriminant coding based convolutional feature representation for multimodal finger recognition [J]. Information Sciences, 2021, 547: 1170-1181.).
[0062] (5) Attentional Feature Fusion (AFF) (Dai Y, Gieseke F, Oehmcke S, et al. Attentional feature fusion [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2021: 3560-3569.).
[0063] The experimental results are shown in Table 1. It can be seen from Table 1 that compared with other methods, the method of the present invention has a better recognition effect and the highest recognition accuracy rate, reaching 95.41%.
[0064] Table 1 Comparison of the accuracy of different multimodal recognition methods
[0065] algorithm Accuracy (%) LBP+concatenation+KNN 87.94 HOG+concatenation+KNN 82.11 ADLBP+concatenation+KNN 89.91 LCCNN 84.17 AFF 87.18 This patented method 95.41
[0066] In summary, the method proposed in the present invention can improve the recognition accuracy by mining the complementary information between different biometric features. The method is real and effective.
[0067] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A multimodal recognition method based on fingerprint and finger vein, characterized in that: The method comprises the following steps: Step 1: Obtain fingerprint and finger vein image data; record the fingerprint and finger vein image data of the person being collected through the collection device; Step 2: Extract the corresponding region of interest (ROI) from all the data obtained in step 1; Step 3: Build a network model, perform feature extraction, fusion and classification, and use the collected data obtained in step 2 to train the model to obtain the best model parameters; the network model mainly includes two branch networks and a multi-feature fusion module; Step 3.1: Construct a branch network; the first branch network is used to extract fingerprint features, including multiple convolution, pooling, normalization, and activation function layers; the fingerprint ROI image extracted in step 2 is input into the network to extract features; The second branch network is used to extract finger vein features, including multiple convolution, pooling, normalization, and activation function layers; the finger vein ROI image extracted in step 2 is input into the network to extract features; The first branch network contains 5 convolutional layers, of which the first convolutional layer contains 1 7×7 convolutional layer, 1 normalization layer, 1 activation function layer, and 1 3×3 maximum pooling layer, and the remaining four convolutional layers each contain 1 3×3 convolutional layer, 1 normalization layer, 1 activation function layer, and 1 3×3 maximum pooling layer; The second branch network is used to extract the feature information matrix corresponding to the finger vein, which includes 5 convolutional layers and 1 adaptive pooling layer. The first convolutional layer includes 1 7×7 convolution, 1 normalization, 1 activation function, and 1 3×3 maximum pooling. The remaining four convolutional layers all include 1 3×3 convolution, 1 normalization, 1 activation function, and 1 3×3 maximum pooling. Step 3.2: Construct a multi-feature fusion module; this module accepts the output of the two branch networks as input, and then uses the channel and spatial attention mechanisms to fuse the features of different modalities and output them; The multi-feature fusion module includes a channel attention module, a spatial attention module and two sigmoid layers. The channel attention module includes two 1×1 convolutional layers, one activation function layer and one adaptive pooling layer. The spatial attention module includes two 7×7 convolutional layers, two normalization layers, one activation function layer and one sigmoid layer. In the multi-feature fusion module, the outputs of the two branch network modules are first taken as input, and after the initial feature aggregation IFI, the matrix F is obtained. initial , and its calculation formula is shown in formula (1): where F fp is the fingerprint feature information matrix, F fv is the finger vein feature information matrix, ⊕ represents element-by-element addition; F initial Input to the channel attention module to get the matrix F c ; F c After inputting sigmoid1, the fingerprint vein fusion coefficient F of the channel attention module is obtained c_final and 1-F c_final ; F c Input to the spatial attention module to get another matrix F s ; F s After inputting sigmoid2, the fingerprint and vein fusion coefficient F obtained by the spatial attention module is obtained. s_final and 1-F s_final ; After the above steps, the output result after the fusion of fingerprint and finger vein features is calculated according to formula (2): in, represents element-wise multiplication, stands for element-by-element addition; Step 3.3: Use the collected fingerprint and finger vein image data to train the above network model to obtain the optimal model parameters; Step 4: Apply the model; input the fingerprint and finger vein image to be identified into the network model, and the model outputs the corresponding identification results.
2. The multimodal recognition method based on fingerprint and finger vein according to claim 1, characterized in that: In step 1, the acquisition device used is a high-definition camera.
3. The multimodal recognition method based on fingerprint and finger vein according to claim 1, characterized in that: In step 3.1, the output dimensions of the two branch networks are both 512×3×6, which respectively represent the fingerprint and finger vein feature information matrices extracted by the CNN network.
4. The multimodal recognition method based on fingerprint and finger vein according to claim 1, characterized in that: In step 3.3, the Adam optimizer is used to calculate the adaptive learning rate of each parameter, and then the recognition performance of the network model is determined by the test set, the optimal parameters are determined, and the optimal network model obtained by the final training is saved.
5. The multimodal recognition method based on fingerprint and finger vein according to claim 1, characterized in that: In step 4, the network model includes two branch networks, a multi-feature fusion module, a fully connected layer and a Softmax classifier in sequence.
