Finger vein recognition method based on lightweight fusion of global and local feature network

Abstract

This article discloses a finger vein recognition method based on a lightweight fusion global and local feature network. 1. Construct a data set and divide it into a training set and a test set; 2. Based on a lightweight residual unit, design a lightweight fusion global and local feature network FGL‑MobileNet; 3. Design an FGL‑Net network based on the fusion of global and local features, including a backbone network based on an improved residual network and a global feature and local feature extraction module; 4. Design a network loss function; 5. Train the entire FGL‑MobileNet network model until the entire training set is iterated for several times; 6. Input the test set image into the trained FGL‑MobileNet network model to extract the finger vein features and perform recognition and comparison. The present invention builds FGL‑MobileNet by stacking the unit, so that the network can quickly and effectively expand the receptive field of the finger vein features to obtain richer finger vein detail features, making the extracted finger vein features more discriminative, and the number of network parameters is also greatly reduced.

Description

Technical Field

[0001] The invention belongs to the field of biometric feature recognition and computer vision, and mainly relates to a finger vein recognition method based on a lightweight fusion global and local feature network. Background Art

[0002] At present, finger vein recognition technology has developed rapidly and has been widely used in many industries, providing more and more people with faster and safer identity authentication services. Traditional finger vein recognition algorithms mainly extract the lines, textures, endpoints and bifurcation points of finger veins as finger vein features for recognition. There is a significant difference in the grayscale distribution between the vein area and the background area of ​​the finger vein image. This difference forms the texture of the finger vein image, which also has good recognition performance. Local binary pattern (LBP) is widely used to extract image texture. This method sets a 3×3 domain window to slide on the image, encodes the positive and negative relationship between the grayscale difference between the center pixel of the window and its domain pixel, and then converts the encoding result into a decimal number as the LBP value of the center pixel, which reflects the texture information of the local area. This method is insensitive to the linear change of illumination of the entire image, and the calculation method is simple, but its anti-noise performance is weak. Rosdi et al. proposed a local linear binary method (LLBP) based on the LBP method to extract local features of finger veins in horizontal and vertical directions. Lee et al. used support vector machines to classify LBP features, and weighted various features according to the classification results to improve the recognition rate. Fu Hua et al. combined NBP with the idea of ​​block segmentation and proposed MMNBP to extract finger vein texture features, which improved the robustness of finger vein features. However, it did not grasp the global vein information well and could not highlight the structure of veins. The vein detail point features are further extracted based on the vein topology. However, due to the influence of factors such as finger placement pressure, light intensity, and equipment exposure, the collected finger vein images will appear blurred and have low contrast, which makes it impossible to extract enough vein detail points, which has a great impact on the final recognition performance.

[0003] The above traditional finger vein features are based on artificially designed features with a shallow feature layer. Although they have good recognition performance, these traditional features usually have high requirements on image quality and are difficult to adapt to the complex finger posture changes in actual application scenarios.

[0004] In recent years, deep learning technology has developed rapidly, and deep convolutional neural networks have received extensive attention and research from many scholars. They can not only extract shallow features such as image contours and corners, but also extract deeper advanced semantic features. Their applications in face recognition, target detection and other fields have achieved fruitful results and brought huge economic benefits to society. Tao Zhiyong et al. proposed a finger vein recognition algorithm based on improved AlexNet, which accelerated the network training speed and reduced the model complexity, and achieved good classification results. However, the above methods all use classification networks, which have high accuracy in classifying fingers in the training set, but their generalization performance is poor, that is, the classification accuracy of untrained fingers will be greatly reduced. Tang et al. proposed a two-stage network, which is divided into a basic network and an extended network, and used an optimized contrast loss function to supervise the network, so as to train more discriminative finger vein features and improve the finger vein recognition performance. However, the existing finger vein datasets have few finger categories and there are no finger vein images with rich postures for training. In addition, most deep learning finger vein algorithms rely on the global features of finger veins for recognition. The network structure is not designed with full consideration of the characteristics that the overall vein structure between similar finger images changes significantly while the local significant vein structure changes slightly when the finger posture changes. As a result, the recognition rate is still low when the finger posture changes.

[0005] The rapid breakthroughs in the performance of hardware such as GPUs have promoted the continuous advancement of deep learning technology. At present, the accuracy of deep convolutional neural networks has surpassed that of humans and traditional technical solutions in many application scenarios. However, as the performance of deep convolutional neural networks becomes higher and higher, their depth becomes deeper and deeper, and the number of parameters and the amount of calculation required are also increasing significantly, accompanied by problems such as large storage space occupied by network models and high power consumption. These problems limit the application of neural networks on resource-constrained embedded devices. Finger vein recognition systems are usually developed and run based on embedded devices, so it is very necessary to compress the network model.

[0006] Therefore, the present invention will be based on the lightweight network design strategy, design a suitable lightweight finger vein feature extraction network for the finger vein recognition task, and combine the idea of ​​knowledge distillation to improve the generalization ability of the lightweight network. Summary of the invention

[0007] This paper designs a finger vein recognition method based on a lightweight fusion of global and local feature networks. First, a lightweight residual unit (SE-ErfResBlock) based on fast receptive field is designed using deep separable convolution. By stacking this unit to build FGL-MobileNet, the network can quickly and effectively expand the receptive field of finger vein features to obtain richer finger vein detail features, making the extracted finger vein features more discriminative, and the number of network parameters is greatly reduced. In order to make the network pay more attention to the learning of important features of finger veins and reduce the attention to useless information, SE-ErfResBlock adds a channel attention mechanism. Experimental results show that the proposed FGL-MobileNet effectively reduces the model size and has good recognition performance.

[0008] The finger vein recognition method based on lightweight fusion global and local feature network includes the following steps:

[0009] Step 1: Construct a data set and divide it into a training set and a test set. The divided training set is expanded by using translation, brightness change, rotation, and noise addition operations to meet the different types of veins to improve the generalization ability of the network model.

[0010] Step 2: Design a lightweight fusion global and local feature network FGL-MobileNet. The present invention constructs FGL-MobileNet by proposing a lightweight residual unit SE-ErfResBlock based on a fast receptive field.

[0011] Step 3: Design the FGL-Net network based on the fusion of global and local features, which mainly includes a backbone network based on an improved residual network and global and local feature extraction modules.

[0012] Step 4, design the network loss function. During the FGL-Net network training process, in order to improve the learning representation ability of the network model, the cross entropy loss and CurricularFace loss are used to jointly supervise the network. Compared with FGL-Net, the FGL-MobileNet network is shallower and the number of parameters is significantly reduced, so its own learning ability is limited. Directly training FGL-MobileNet will cause the generalization performance of the model to decrease, and it is difficult to learn stable finger vein features, resulting in a low recognition rate. To this end, the present invention introduces a teacher-student model, adds feature map loss and knowledge distillation loss to the loss term, and uses FGL-Net as the teacher network to impart its learned more accurate finger vein feature distribution to the student network FGL-MobileNet as much as possible, thereby improving the generalization performance of FGL-MobileNet.

[0013] Step 5: Train the entire model until the entire training set is iterated several times.

[0014] Step 6: Input the test set images into the trained model to extract finger vein features and perform recognition and comparison.

[0015] The beneficial effects of the present invention are as follows:

[0016] Firstly, a lightweight residual unit (SE-ErfResBlock) based on fast receptive field is designed by using depthwise separable convolution. By stacking this unit to build FGL-MobileNet, the network can quickly and effectively expand the receptive field of finger vein features to obtain richer finger vein detail features, making the extracted finger vein features more discriminative, and the number of network parameters is greatly reduced. In order to make the network pay more attention to the learning of important features of finger vein and reduce the attention to useless information, SE-ErfResBlock adds a channel attention mechanism. Experimental results show that the proposed FGL-MobileNet effectively reduces the model size and has good recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the specific information and division of the data set of the present invention.

[0018] Figure 2 They are finger vein images for data set enhancement.

[0019] Figure 3 The present invention proposes a lightweight residual unit (SE-ErfResBlock) based on a fast receptive field to build an FGL-MobileNet network.

[0020] Figure 4 It is the channel attention mechanism in the SE-ErfResBlock residual unit.

[0021] Figure 5 This is the overall structure diagram of the FGL-MobileNet network model.

[0022] Figure 6 Detailed network structure of FGL-MobileNet.

[0023] Figure 7 It is the improved residual block in the FGL-Net network.

[0024] Figure 8 Figure 2 is the distillation loss graph of FGL-MobileNet supervised by only the last feature classification layer of FGL-Net.

[0025] Fig. 9The feature map loss is used to guide the training of the feature maps of the corresponding stages in FGL-MobileNet through the intermediate feature maps of different stages of FGL-Net.

[0026] Fig.10 Flowchart for model training.

[0027] Fig.11 In order to verify the effectiveness of the loss function of the network FGL-MobileNet of the present invention, the Top1 ranking performance is compared under different loss function training.

[0028] Fig.12 Finger vein features are extracted for VGG-16, finger vein improved residual network, FGL-Net, MobileNetv3, FGL-MobileNet, and KD-FGL-MobileNet. The ROC curves of each network are: (a) is the ROC curve of each network on the FV-USM test set; (b) is the ROC curve of each network on the FV-Normal test set.

[0029] Fig.13 Comparison of Top1 ranking performance of different networks on the FV-USM test set and the FV-Normal test set.

[0030] Fig.14 Comparison of the model sizes of VGG-16, finger vein improved residual network, FGL-Net, MobileNetv3, and FGL-MobileNet.

[0031] Fig.15 The zero-error recognition rate and Top 1 ranking of VGG-16, finger vein improved residual network, FGL-Net, MobileNetv3, and FGL-MobileNet on the FV-Special test set. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0033] The specific implementation steps of the present invention are as follows:

[0034] S1. Constructing the dataset. The finger vein dataset used in this paper consists of three parts: the normal finger posture dataset (FV-Normal) collected by the research team, the special finger posture dataset (FV-Special) and the Universiti Teknologi Malaysia finger vein dataset (FV-USM).

[0035] (1) The FV-Normal dataset contains 4,600 finger vein images in normal positions, with 6 images for each finger, a total of 24,000 images, and an image resolution of 200×500. The first 4,000 fingers come from 500 people, with 8 finger images collected from each person, and the last 600 fingers come from 100 people, with 6 finger images collected from each person. The training set and the test set are divided into 5:1 according to the number of people. The finger vein images of the first 500 people are used for training, and are expanded by rotation, translation, etc. to obtain the expanded training set. The finger vein images of the last 100 people are used for testing.

[0036] (2) The FV-Special dataset contains 64 fingers. Each finger has 9 placement postures: normal, heavy pressure, bending, upward movement, downward movement, plane left rotation, plane right rotation, axis left rotation, and right axis rotation. Ten images are collected for each finger placement posture, for a total of 5760 images with an image resolution of 200×500. All images are used as test sets to test the recognition performance of the finger vein recognition algorithm under different finger placement postures.

[0037] (3) The fingers in the FV-USM dataset belong to 123 people, with a total of 492 fingers. Each finger has 12 finger vein images, a total of 5904 images, and the ROI image resolution is 100×300. This paper divides the training set and the test set into 4:1. The first 392 finger vein images are used for training and are expanded by rotation, translation, etc. to obtain the expanded training set. The last 100 finger vein images are used for testing. Figure 1 Detailed information of the three data sets.

[0038] from Figure 1 It can be seen that the FV-USM dataset has only 392 fingers and 4704 finger vein images for training. Although the FV-Normal dataset has 4000 fingers for training, each finger has only 6 finger vein images. The insufficient number of images in the finger vein training set brings two problems: (1) The trained finger vein feature extraction network is prone to overfitting and has insufficient generalization ability on finger vein images outside the training set; (2) Too few training samples also means that the posture of each finger is relatively simple. When using the trained model for feature extraction, the proposed finger vein feature is not robust to changes in finger posture. Therefore, it is very necessary to appropriately expand the training set to improve the generalization ability of the network model. This paper uses the following traditional image processing techniques to expand the training set.

[0039] (1) Translation transformation

[0040] The finger vein image in translation posture can be calculated with the help of transformation matrix T and the original image, as shown in formulas (1.1) and (1.2):

[0041]

[0042]

[0043] Where (x0, y0) is the pixel coordinates on the original finger vein image, (x1, y1) is the pixel coordinates after translation, (t x ,t y ) are the translation distances of the image in the up and down and left and right directions respectively. The finger vein image after translation is as follows Figure 2 shown.

[0044] (2) Brightness changes

[0045] Affected by factors such as the pressure of the user's finger placement, the intensity of surrounding light, and the exposure of the device, the collected finger veins usually have inconsistent overall brightness. The training set has insufficient diversity in brightness changes due to the small number of images, which makes the trained network model insufficient in generalization ability. Therefore, the overall brightness of the finger vein image can be changed by adjusting the grayscale scaling coefficient of the gamma change, thereby enriching the brightness diversity of the training set image. The gamma transformation formula is shown in formula (1.3):

[0046]

[0047] In the formula, g0 is the gray value of the point on the original finger vein image, and g1 represents the pixel value after brightness transformation. λ is the gray adjustment coefficient. When its value is greater than 0, the overall gray value of the image becomes smaller, and the image visual effect is darker. On the contrary, the overall gray value of the image becomes larger, and the image visual effect is brighter, achieving the purpose of brightness adjustment. Figure 2 Demonstrates the brightness changes of finger vein images.

[0048] (3) Rotation transformation

[0049] The rotation transformation is used to simulate the planar rotation posture finger vein image. Its conversion formula is the same as formula (1.1), and the transformation matrix is ​​as formula (1.4):

[0050]

[0051] In the formula, (x c ,y c ) refers to the coordinates of the center pixel of the original vein image, and θ is the rotation angle, which takes values ​​of ±2, ±4. Figure 2 The finger vein images after rotation of ±2 degrees are shown.

[0052] (4) Image Noise

[0053] During the image acquisition process, foreign matter such as dust may fall on the mirror surface of the image acquisition device, causing noise in the acquired vein image. By adding random salt and pepper noise to simulate noise such as dust, the difficulty of network training is increased, and the network's anti-noise performance and feature extraction capabilities are improved. Figure 2 Shows a finger vein image with salt and pepper noise added.

[0054] S2. Design a lightweight fusion global and local feature network FGL-MobileNet. The present invention constructs FGL-MobileNet by proposing a lightweight residual unit based on fast receptive field (SE-ErfResBlock). In order to make the network pay more attention to the learning of important features of finger veins and reduce the attention to useless information, SE-ErfResBlock adds a channel attention mechanism to construct the designed SE-ErfResBlock as follows: Figure 3 As shown, the channel attention mechanism is a SE module such as Figure 4 As shown in the figure, the SE module consists of three parts: compression operation, excitation operation and weighted operation. Assume that the input feature map is x, and its size is W×H×C1. The first step is the compression operation. GAP is used on the input x to compress the finger vein spatial features on each channel by a real number. At this time, the feature map size becomes 1×1×C1, and each real number represents the global feature of the finger vein on each channel, that is, it has a global vein receptive field. The compression operation does not learn the importance between different channels. Therefore, the second step of excitation operation is required, which includes two fully connected layers. The first fully connected layer reduces the dimension of the output of the compression operation to achieve the purpose of reducing the number of parameters. The second fully connected layer is used to restore the original dimension. This process can learn the nonlinear relationship between different feature channels and obtain the weight matrix representing the importance of each channel; the third step is the weighted operation. The channel importance weight matrix output by the excitation operation is multiplied by the corresponding channel of the input feature map for weighting, thereby enhancing the important finger vein features and suppressing the expression of useless information.

[0055] The FGL-MobileNet network structure is as follows Figure 5 The detailed network structure is shown in Figure 6As shown in the figure. FGL-MobileNet is mainly divided into the backbone network and the global and local feature extraction modules. The backbone network is mainly composed of lightweight residual units; specifically, it includes a 5-layer convolution structure. Conv_1 consists of 1 batch normalization layer, 1 Mish activation layer, and 1 3×3 standard convolution layer. Conv_2 to Conv_4 are composed of 2, 2, and 4 SE-ErfResBlocks, respectively. Conv_5 is divided into 3 branches, each of which is composed of 2 SE-ErfResBlocks, and each branch learns the global finger vein features and local significant vein features at the corresponding granularity. In order to obtain higher-resolution feature maps for block division, Branch-2 and Branch-3 cancel the downsampling operation. In the global and local feature extraction module, the global vein features are extracted by global average pooling (GAP), and the local significant vein features are extracted by global maximum pooling (GMP). The extracted global features and local features are reduced in dimension by a layer of 1×1 convolution, and the finger vein fusion feature vector is obtained by feature splicing. Five 256-dimensional local features L and three 256-dimensional global features G are spliced ​​to obtain a 2048-dimensional fusion feature vector F.

[0056] S3, design the FGL-Net network based on the fusion of global and local features, which mainly includes the backbone network based on the improved residual network and the global and local feature extraction modules. The improved residual block is as follows Figure 7 As shown in the figure. The FGL-Net network has the same design structure as the FGL-MobileNet network, except that the backbone network here uses the ResNet50 structure, which contains a 5-layer convolution structure. Conv_T1 contains a convolution layer with a convolution kernel size of 3×3, a BN layer, and a Mish activation layer. Conv_T2 to Conv_T5 are composed of 3, 4, 14, and 3 improved residual blocks, respectively. By extracting global features and local features through the global and local feature extraction module proposed in the FGL-MobileNet design, FGL-Net finally outputs a 2048-dimensional finger vein fusion feature vector.

[0057] S4. Design loss function. During the FGL-Net network training process, in order to improve the learning representation ability of the network model, the cross entropy loss and CurricularFace loss are used to jointly supervise the network. The calculation formula is as follows:

[0058] L=L CrossEntropy +L CurricularFace (4.1)

[0059] Where, LCrossEntropy is the standard cross entropy loss, L CurricularFace is the CurricularFace loss.

[0060]

[0061] Where N is the number of images in a batch in the training set, n is the number of categories of vein images in the training set, and f i is the feature vector of the i-th finger vein image, y i represents f i The corresponding category, W j represents the jth column of the weight matrix W, b j 、b yi is the bias term.

[0062] The present invention introduces CurricularFace loss, which aims to expand the inter-class difference of finger vein feature vectors and reduce the intra-class difference. At the same time, it adaptively mines special posture finger samples online, so that the network can better adapt to posture changes such as translation and axis rotation. The calculation formula is as follows:

[0063]

[0064] Where N is the number of images in a batch in the training set, s is the radius of the hypersphere, m is the angle increment penalty term, and f is i is the feature vector of the i-th finger vein image, y i represents f i The corresponding category, express With f i The angle between them, N(t,cosθ j ) represents the negative cosine similarity function, and its formula is as follows:

[0065]

[0066] t (k) =αr (k) +(1-α)t (k-1) (4.5)

[0067] Where t is the adaptive estimation parameter, t (0) =0, α is momentum, r (k) is the average value of the sine and cosine similarity of the kth mini-batch. When the finger vein image is classified correctly, it is considered to be a normal posture finger vein image. At this time, N(t,cosθ j )=cosθ j; When the finger vein image is classified incorrectly, it is considered to be a special posture finger vein image, N(t,cosθ j )=cosθ j (t+cosθ j ). In the early stage of training, t approaches 0, (t+cosθ j )<1, the weight of normal fingers is greater than that of special posture fingers, and the network pays more attention to normal fingers, which is conducive to accelerating the convergence of the model; as the training progresses, the sine and cosine similarity of the samples becomes higher and higher, that is, r (k) Continuously increasing, t (k) Continue to increase, and finally (t+cosθ j )>1, that is, the weight of special posture fingers is greater than that of normal fingers. The network pays more attention to the training of special posture fingers, so that the network can better adapt to the changes of finger translation, axis rotation and other postures.

[0068] FGL-MobileNet and FGL-Net have the same overall network framework design, so the original training loss function of FGL-MobileNet is consistent with that of FGL-Net, and its formula is as follows:

[0069] Loss Oringin =Loss CrossEntropy +Loss CurricularFace (4.6)

[0070] In the formula, Loss CrossEntropy is the cross entropy loss, Loss CurricularFace The CurricularFace loss is used. Both the global feature vector and the finger vein fusion feature vector contain global vein information and have high discrimination power. The CurricularFace loss is used for constraint. The local features of the blocks may not be aligned, so the local feature vector is constrained by the cross entropy loss to ensure that the classes can be separated.

[0071] In order to enable the network to learn global and local vein information at different granularity sizes and improve the network's recognition performance under different vein images, FGL-Net and FGL-MobileNet both contain three branches, including three global features, five local features, and one finger vein fusion feature. Figure 8As shown, the present invention will perform knowledge distillation on the classification vectors of the 9 finger vein features of FGL-MobileNet through the classification vectors of the 9 finger vein features of FGL-Net, so that FGL-MobileNet can fully imitate the output probability distribution of FGL-Net, ensuring that FGL-MobileNet can better extract the global and local features of finger veins at different granularity sizes, thereby improving the robustness of FGL-MobileNet to special posture fingers. The knowledge distillation loss formula is as follows (4.7):

[0072]

[0073] In the formula, Loss S Represents the distillation loss of two finger vein classification vectors, Loss KD represents the total loss of network knowledge distillation, G si ,G ti Represents the classification vector of the global features of FGL-MobileNet and the i-th branch of FGL-Net, G sF ,G tF Represents the classification vectors of the finger vein fusion features of FGL-MobileNet and FGL-Net, L si-j ,L ti-j Represent the classification vector of the j-th local feature in the i-th branch of FGL-MobileNet and FGL-Net, respectively.

[0074] Only the last feature classification layer of FGL-Net supervises FGL-MobileNet, which has the problem of insufficient constraints, so the performance improvement of FGL-MobileNet is limited. The backbone network is responsible for learning the mapping relationship from the original finger vein image to the high-level semantic features of the finger vein. Since the FGL-Net network has a deeper layer and more parameters, it has a stronger learning ability. Therefore, the finger vein information contained in the feature map it extracts is more accurate and rich, and can better complete this mapping. The backbone networks of FGL-Net and FGL-MobileNet contain 4 residual convolution layers from shallow to deep layers, and a total of 6 intermediate feature maps are generated, which extract the finger vein feature information at different stages of the network. Therefore, the training of the feature maps of the corresponding stages in FGL-MobileNet is guided by the intermediate feature maps of different stages of FGL-Net, which more effectively constrains FGL-MobileNet to continuously learn FGL-Net's more effective finger vein feature representation from shallow to deep layers, and better complete the mapping of the original finger vein image to the high-level semantic features of the finger vein. In order to better highlight the finger vein texture information, this paper uses L2Loss to measure the distance between feature maps. The feature map loss is as follows Fig. 9As shown, its formula is as follows (4.8):

[0075]

[0076] In the formula, Loss f represents the feature map loss, U i represents the i-th feature map of the student network, V i represents the i-th feature map of the teacher network.

[0077] Therefore, the total loss function formula of FGL-MobileNet is shown in formula (4.9):

[0078] Loss sum =α×Loss Origin +(1-α)×Loss KD +λ×Loss f (4.9)

[0079] In the formula, Loss sum Represents the total loss function, α and λ are loss weighting coefficients, which adjust the weights of different loss functions. In order for FGL-MobileNet to better learn the finger vein feature representation of the teacher network, this chapter takes α = 0.05 and λ = 1.

[0080] S5, train the entire model, such as Fig.10 , until the entire training set is iteratively trained several times.

[0081] S6. Input the test set images into the trained model to extract finger vein features and perform recognition and comparison.

[0082] The method of the present invention uses the Euclidean distance as the standard for measuring the similarity of vein features. The smaller the Euclidean distance, the higher the similarity of the two vein features. Conversely, the lower the similarity of the two vein features. The test set refers to the vein features that are compared in a 1:1 manner for different categories, and the threshold T is obtained. Similarly, the test set refers to the vein features that are compared in a 1:1 manner for the same category. When the comparison result is less than the corresponding threshold, it is considered that the comparison is successful, otherwise, the comparison fails.

[0083] The server GPU used in the experiment is NVIDIA TITANRTX, CUDA version is 10.1, the operating system is 64-bit Ubuntu16.04, the deep learning framework used is Pytorch1.7.1, and the programming language is python3.7.9. In the training stage, the batch size of the finger vein image is set to 16, and a total of 16 rounds of training are performed. The initial value of the learning rate is 0.0001, and it is decayed by 10 times every 5 rounds. The update of the network parameters is optimized by the Adaptive Momentum Estimation (ADAM) optimizer. The dimension of the finger vein feature vector is 2048, and the normalized image size is 112×112.

[0084] In order to verify the effectiveness of the designed FGL-MobileNet loss function, this paper designs three groups of comparative experiments based on the FV-USM dataset. In the first group of experiments, the network is constrained only by the original FGL-MobileNet loss. In the second group of experiments, the distillation loss is added on the basis of the first group of experiments. The third group of experiments adds both the distillation loss and the feature map loss on the basis of the first group of experiments, that is, the loss function designed in this chapter. The experimental results are shown in Figure 2. Fig.11 As shown in the figure, compared with only using the original loss, after adding the distillation loss, the network's zero-error recognition rate increased by 0.92%, and the Top1 ranking performance increased by 0.12%, indicating that after adding the knowledge distillation loss, FGL-MobileNet can learn the finger vein feature distribution output by the teacher model, thereby improving the finger vein recognition performance. After adding the knowledge distillation loss and feature map loss at the same time, the network's zero-error recognition rate increased by 2.14% compared with only using the original loss, and the Top1 ranking performance increased by 0.17%. After adding the feature map loss, the recognition performance of FGL-MobileNet was further improved, indicating that the feature map loss can more strongly constrain FGL-MobileNet from the shallow network to the deep network to approximate the teacher model, so that it can learn a more effective finger vein feature representation.

[0085] In order to verify the effectiveness of FGL-MobileNet, the present invention sets up the following groups of comparative experiments. Among them, FGL-MobileNet means that only the original loss function is used for training, and KD-FGL-MobileNet means that the loss function designed by the present invention is used for training.

[0086] (1) Performance comparison on the FV-USM test set and the FV-Normal test set

[0087] VGG-16, finger vein improved residual network, FGL-Net, MobileNetv3, FGL-MobileNet, and KD-FGL-MobileNet are used to extract finger vein features. The ROC curves of each network are shown in the following figure. Fig.11 As shown. Fig.12 From the ROC curve, we can see that when FAR is 0, FGL-MobileNet has a zero-error recognition rate that is 4.23%, 10.98%, 9.13% higher than that of VGG-16, finger vein improved residual network, and MobileNetv3 on the FV-USM test set and FV-Normal test set, and is 8.48%, 16.69%, 6.21% higher, indicating that FGL-MobileNet can better extract finger vein feature information. It is 2.2% lower and 1.90% lower than that of FGL-Net, indicating that the generalization performance of FGL-MobileNet is weakened due to the shallow network and small number of parameters. However, KD-FGL-MobileNet’s zero-error recognition rates on the FV-USM test set and FV-Normal test set were improved by 2.14% and 1.77% respectively compared with FGL-MobileNet, and were only 0.02% and 0.13% lower than FGL-Net, indicating that the loss function designed in this chapter can effectively transfer the finger vein feature distribution learned by FGL-Net to FGL-MobileNet.

[0088] from Fig.13 It can be seen that FGL-MobileNet's Top1 ranking performance on the FV-USM test set and FV-Normal test set is improved by 1.34%, 4.10%, 3.82% and 1.58%, 8.09%, 4.79% respectively compared with VGG-16, finger vein improved residual network, and MobileNetv3. KD-FGL-MobileNet's Top1 ranking performance on the FV-USM test set and FV-Normal test set is improved by 0.19% and 0.17% compared with FGL-MobileNet, and is only 0.02% and 0.00% different from FGL-Net, indicating the effectiveness of the FGL-MobileNet and its loss function designed in this chapter, which effectively improves the Top1 ranking performance of finger vein.

[0089] Depend on Fig.14 It can be seen that the model size of FGL-MobileNet is 17 times smaller than that of FGL-Net, which is only 12.25M. The number of parameters of the residual network improved by VGG-16 and finger vein is also greatly reduced, and it is only slightly larger than the MobileNetv3 model. The practicality of FGL-MobileNet is greatly improved.

[0090] (2) Performance comparison on the FV-Special test set

[0091] In order to verify the recognition performance and Top1 ranking performance of FGL-MobileNet on finger vein images with special postures, MobileNetv3, FGL-Net, FGL-MobileNet, and KD-FGL-MobileNet were used for feature extraction. The zero-false recognition rate and Top1 ranking of each network under different finger postures are shown in the following figure. Fig.15 shown.

[0092] Depend on Fig.15 It can be seen that under different finger placement postures, the average zero-error recognition rate and average Top1 ranking performance of FGL-MobileNet are respectively improved by 17.93% and 8.68% compared with MobileNetv3, indicating that the finger vein features extracted by FGL-MobileNet in this paper are more robust to changes in finger posture. Compared with FGL-Net, the average zero-error recognition rate and average Top1 ranking performance of FGL-MobileNet decreased by 6.19% and 3.78% respectively, which is a large loss. However, when the loss function designed in this chapter is used for training, the performance of KD-FGL-MobileNet is almost the same as that of FGL-Net, indicating that the loss function in this chapter can effectively transfer the robustness of FGL-Net to changes in finger posture to FGL-MobileNet, significantly improving the recognition performance of FGL-MobileNet in special posture finger vein images.

Claims

1. Finger vein recognition method based on lightweight fusion of global and local feature network, characterized by The following steps are involved: Step 1: construct a data set and divide it into a training set and a test set; expand the divided training set by using translation, brightness change, rotation, and noise addition operations; Step 2: Based on the lightweight residual unit, design a lightweight fusion global and local feature network FGL-MobileNet; Step 3: Design the FGL-Net network based on the fusion of global and local features, which mainly includes a backbone network based on an improved residual network and a global and local feature extraction module; Step 4: Design the network loss function. During the training process of FGL-Net network, cross entropy loss and CurricularFace loss are used to jointly supervise the network. For FGL-MobileNet network, the teacher-student model is introduced, and feature map loss and knowledge distillation loss are added to the loss term. FGL-Net is used as the teacher network to pass the more accurate finger vein feature distribution learned by it to the student network FGL-MobileNet, thereby improving the generalization performance of FGL-MobileNet. Step 5: Train the entire FGL-MobileNet network model until the entire training set is iterated several times; Step 6: Input the test set images into the trained FGL-MobileNet network model to extract finger vein features and perform recognition and comparison; Step 2 is implemented as follows: FGL-MobileNet is constructed based on the lightweight residual unit SE-ErfResBlock with fast receptive field, and a channel attention mechanism is added to SE-ErfResBlock. The channel attention mechanism is an SE module, which includes three parts: compression operation, excitation operation and weighted operation; SE-ErfResBlock based on the lightweight residual unit with fast receptive field is a modified residual block, which consists of a channel-by-channel convolution layer with a convolution kernel size of 3×3, a point-by-point convolution layer of 1×1, a batch normalization layer, a channel attention mechanism layer and a Mish activation layer; the input passes through a 3×3 channel-by-channel convolution layer, a batch normalization layer and a Mish activation layer, a 3×3 channel-by-channel convolution layer, a batch normalization layer and a Mish activation layer, a 1×1 point-by-point convolution layer, a batch normalization layer and a channel attention mechanism layer in sequence, and the obtained output is fused with the original input through a skip connection and then passed through a Mish activation layer to obtain the final output; The FGL-MobileNet network structure mainly includes the backbone network and global and local feature extraction modules; The backbone network is mainly composed of lightweight residual units based on fast receptive fields, including a 5-layer convolutional structure: Conv_1 consists of 1 batch normalization layer, 1 Mish activation layer, and 1 3×3 standard convolutional layer; Conv_2 to Conv_4 are composed of 2, 2, and 4 SE-ErfResBlocks respectively; Conv_5 is divided into 3 branches, each of which is composed of 2 SE-ErfResBlocks, each of which learns the global finger vein features and local significant vein features at the corresponding granularity; In the global and local feature extraction module, the global vein features are extracted by global average pooling to obtain three 256-dimensional global features G, and the local significant vein features are extracted by global maximum pooling to obtain five 256-dimensional local features L. The extracted global features and local features are reduced in dimension by a layer of 1×1 convolution, and the finger vein fusion feature vector is obtained by feature splicing; the five 256-dimensional local features L and the three 256-dimensional global features G are spliced ​​to obtain a 2048-dimensional fusion feature vector F; The FGL-Net network based on the fusion of global and local features mainly includes a backbone network based on an improved residual network and a global and local feature extraction module; the backbone network of the FGL-Net network adopts the ResNet50 structure, including a 5-layer convolution structure, Conv_T1 includes a convolution layer with a convolution kernel size of 3×3, a BN layer and a Mish activation layer, and Conv_T2 to Conv_T5 are composed of 3, 4, 14 and 3 improved residual blocks respectively; FGL-Net finally outputs a 2048-dimensional finger vein fusion feature vector; the input of the improved residual block passes through a 3×3 standard convolution layer, a batch normalization layer and a Mish activation layer, a 3×3 standard convolution layer and a batch normalization layer in sequence, and the obtained output is fused with the original input through a jump connection and then passes through a Mish activation layer to obtain the final output.

2. According to claim 1, the finger vein recognition method based on lightweight fusion global and local feature network is characterized in that The specific implementation of the SE module is as follows: assuming that the input feature map is x, its size is W×H×C1; the first step is the compression operation, using GAP on the input x to compress the finger vein spatial features on each channel by a real number, and the feature map size becomes 1×1×C1; the second step is the excitation operation, which includes two fully connected layers. The first fully connected layer reduces the dimension of the output of the compression operation, and the second fully connected layer is used to restore the original dimension to obtain a weight matrix representing the importance of each channel; the third step is the weighted operation, which multiplies the channel importance weight matrix output by the excitation operation with the corresponding channel of the input feature map for weighting, thereby enhancing the important finger vein features and suppressing the expression of useless information.

3. According to claim 1, the finger vein recognition method based on lightweight fusion global and local feature network is characterized in that The loss function is designed as follows: During the FGL-Net network training process, the cross entropy loss and CurricularFace loss are used to jointly supervise the network. The calculation formula is as follows: L=L CrossEntropy +L CurricularFace (4.1) Where, L CrossEntropy is the standard cross entropy loss, L CurricularFace for CurricularFace loss; Where N is the number of images in a batch in the training set, n is the number of categories of vein images in the training set, and f i is the feature vector of the i-th finger vein image, y i represents f i The corresponding category, W j represents the jth column of the weight matrix W, b j , is the bias term; The calculation formula of CurricularFace loss is as follows: Where N is the number of images in a batch in the training set, s is the radius of the hypersphere, m is the angle increment penalty term, and f i is the feature vector of the i-th finger vein image, y i represents f i The corresponding category, express With f i The angle between them, N(t,cosθ j ) represents the negative cosine similarity function, and its formula is as follows: t (k) =αr (k) +(1-α)t (k-1) (4.5) Where t is the adaptive estimation parameter, t (0) =0, α is momentum, r (k) is the average value of the sine and cosine similarity of the kth mini-batch; when the finger vein image is classified correctly, it is considered to be a normal posture finger vein image. At this time, N(t,cosθ j )=cosθ j ; When the finger vein image is classified incorrectly, it is considered to be a special posture finger vein image, N(t,cosθ j )=cosθ j (t+cosθ j ); In the early stage of training, t approaches 0, (t+cosθ j )<1, the weight of normal fingers is greater than that of special posture fingers, and the network pays more attention to normal fingers, which is conducive to accelerating the convergence of the model; as the training progresses, the sine and cosine similarity of the samples becomes higher and higher, that is, r (k) Continuously increasing, t (k) Continue to increase, and finally (t+cosθ j )>1, that is, the weight of the special posture finger is greater than the normal finger weight. The network pays more attention to the training of the special posture finger, so that the network can better adapt to the changes of finger translation and axis rotation posture.

4. The finger vein recognition method based on lightweight fusion global and local feature network according to claim 3 is characterized in that Since FGL-MobileNet and FGL-Net have the same overall network framework design, the original training loss function of FGL-MobileNet is consistent with that of FGL-Net, and its formula is as follows: Loss Oringin =Loss CrossEntropy +Loss CurricularFace (4.6) In the formula, Loss CrossEntropy is the cross entropy loss, Loss CurricularFace CurricularFace loss is used. Both the global feature vector and the finger vein fusion feature vector contain global vein information, and CurricularFace loss is used for constraint. Both FGL-Net and FGL-MobileNet contain three branches, including three global features, five local features and one finger vein fusion feature. The classification vectors of the nine finger vein features of FGL-Net are used to perform knowledge distillation on the classification vectors of the nine finger vein features of FGL-MobileNet, so that FGL-MobileNet fully imitates the output probability distribution of FGL-Net. The knowledge distillation loss formula is as follows: In the formula, Loss S Represents the distillation loss of two finger vein classification vectors, Loss KD represents the total loss of network knowledge distillation, G si ,G ti Represents the classification vector of the global features of FGL-MobileNet and the i-th branch of FGL-Net, G sF ,G tF Represents the classification vectors of the finger vein fusion features of FGL-MobileNet and FGL-Net, L si-j ,L ti-j Represent the classification vector of the jth local feature in the i-th branch of FGL-MobileNet and FGL-Net respectively; The backbone networks of FGL-Net and FGL-MobileNet contain 4 residual convolution layers from shallow to deep layers, generating a total of 6 intermediate feature maps, which extract the finger vein feature information at different stages of the network; therefore, the intermediate feature maps of different stages of FGL-Net are used to guide the training of the feature maps of the corresponding stages in FGL-MobileNet, which more effectively constrains FGL-MobileNet to continuously learn more effective finger vein feature representations from shallow to deep layers, and better complete the mapping of finger vein original images to finger vein high-level semantic features; the feature map loss is used to measure the distance between feature maps, and its formula is as follows (4.8): In the formula, Loss f represents the feature map loss, U i represents the i-th feature map of the student network, V i represents the i-th feature map of the teacher network; Therefore, the total loss function formula of FGL-MobileNet is shown in formula (4.9): Loss sum =α×Loss Origin +(1-a)×Loss KD +λ×Loss f (4.9) In the formula, Loss sum Represents the total loss function, α and λ are loss weighting coefficients.

Images