Finger vein image recognition method based on lightweight bilinear convolutional network

Abstract

The present invention discloses a finger vein image recognition method based on a lightweight bilinear convolutional network, comprising the following steps: S1: preprocessing finger vein images in a public data set; S2: constructing a lightweight bilinear convolutional network; S3: constructing a loss function, and using the training set constructed by S1 to train the convolutional network constructed by S2; S4: establishing a finger vein feature library of local registrants; S5: extracting the finger vein feature vector of the person to be identified; S6: obtaining the cosine similarity between the feature vector extracted by S5 and the feature vector in S4, so as to determine the identity of the person to be identified. The advantages of the present invention are: solving the disadvantage of the large feature dimension of the output of the traditional bilinear network, and further reducing the number of parameters and the amount of calculation in the network. The ability to capture fine-grained differences in finger vein images is improved, thereby improving the recognition accuracy. The time consumption and storage space loss are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of vein image recognition, and in particular to a finger vein image recognition method based on a lightweight bilinear convolutional network. Background Art

[0002] Biometric recognition technology has received widespread attention because it is more secure than traditional personal identity authentication methods while maintaining portability. Among them, the technology of identity recognition based on finger vein images has developed rapidly because it has higher security than existing biometric recognition technologies. The method of manually designing operators to express texture information in finger vein images has been gradually replaced by convolutional neural network methods with stronger feature extraction capabilities due to cumbersome processing steps and weak robustness and generalization. However, at the same time, this also increases the requirements for the computing power of the operating equipment. Therefore, there is a lightweight method for studying convolutional neural networks, which reduces the number of parameters and calculations while maintaining its recognition accuracy as much as possible. On the other hand, finger vein recognition is classified as a fine-grained recognition problem, which requires the use of detailed features in the image, so there are certain requirements for the fine-grained feature extraction capabilities of the network.

[0003] In order to extract fine-grained features of images, Lin [1] A bilinear network structure is proposed, but the feature dimension of the network output is huge, which will seriously consume the computing power and storage space of the running device in actual application scenarios. Gao [2] et al. used tensor sketches to perform kernel approximation on the output of bilinear networks, reducing the dimensionality of their output features. [3] et al. and Li [4] et al. reduced the dimensionality of fine-grained features by performing a low-rank approximation on the covariance matrix of the bilinear network. In addition, Yu [5] et al. performed dimensionality reduction projection before pooling the bilinear network to obtain the covariance matrix, which alleviated the problem of network dimensionality explosion. However, the dimensionality reduction operators constructed by these methods reduce the dimensionality of the output of the bilinear network in one step, so that the fine-grained feature expression ability retained by the network output decreases rapidly as the feature dimension decreases. In addition, the kernel approximation method and the projection method need to store the projection matrix with a large number of parameters, which will occupy the limited storage space of the recognition device.

[0004] References

[0005] [1] Lin T-Y, Roychowdhury A, Maji S. Bilinear cnn models for fine-grained visual recognition[C]. Proceedings of the IEEE international conference on computer vision, 2015:1449-1457.

[0006] [2] Gao Y, Beijbom O, Zhang N, et al. Compact bilinear pooling[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016:317-326.

[0007] [3] Kong S, Fowlkes C. Low-rank bilinear pooling for fine-grained classification[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017:365-374.

[0008] [4] Li Y, Wang N, Liu J, et al. Factorized bilinear models for image recognition[C]. Proceedings of the IEEE International Conference on Computer Vision, 2017:2079-2087.

[0009] [5] Yu C, Zhao X, Zheng Q, et al. Hierarchical bilinear pooling for fine-grained visual recognition[C]. Proceedings of the European conference on computer vision (ECCV), 2018:574-589. Summary of the Invention

[0010] In view of the defects of the prior art, the present invention provides a finger vein image recognition method based on a lightweight bilinear convolutional network. While gradually reducing the dimension of the feature vector output by the bilinear convolutional network, the fine-grained features in the finger vein image are effectively extracted, thereby improving the discrimination ability of the finger vein recognition system. On the other hand, since the number of parameters and the amount of calculation of the traditional bilinear convolutional network are effectively reduced, it is more suitable for application in edge devices with weaker computing power.

[0011] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0012] A finger vein image recognition method based on a lightweight bilinear convolutional network comprises the following steps:

[0013] S1: Preprocess finger vein images in the public dataset to obtain finger vein image datasets divided into high-quality and low-quality datasets, as well as region of interest datasets of the corresponding datasets;

[0014] S2: Build a lightweight bilinear convolutional network whose output is divided into a quality recognition branch and an image classification branch, named RQABS-Net (Recognition and Quality Assessment BilinearSpindle Net).

[0015] S3: The quality recognition branch of RQABS-Net is trained using the finger vein image dataset constructed in S1, and the image classification branch of RQABS-Net is trained using the region of interest dataset constructed in S1. During the training process, the multi-classification cross entropy loss function and the center loss function are used to quantify the gap between the network output and the label.

[0016] S4: Use a finger vein image acquisition device to obtain the finger vein image data stream of the actual registrant, input the data stream into the fully trained RQABS-Net, store the features that are judged as high-quality images based on the results of the RQABS-Net quality recognition branch, and then establish a registered person feature library consisting of multiple registrants.

[0017] S5: Use the finger vein image acquisition device to obtain the finger vein image data stream of the person to be identified, input the data stream into the same RQABS-Net as in step S4, and decide whether to use the feature vector z output by RQABS-Net to perform identity matching in the registered person feature library based on the result of the RQABS-Net quality recognition branch.

[0018] S6: Use the feature vector z output by RQABS-Net as the feature vector of the current finger vein image, calculate the cosine similarity with the feature vector in the registered person feature library, and output the matched identity information or determine that the person to be identified is a stranger based on the threshold.

[0019] Furthermore, S2 includes the following sub-steps:

[0020] S21 is based on the substructures of MobileNet-V2: the inverted residual structure and the linear bottleneck structure. It constructs a spindle-shaped convolutional network structure in which the number of feature map channels c in the first half gradually increases and the number of feature map channels c in the second half gradually decreases. It is named SpindleNet. Among them, the inverted residual structure and the linear bottleneck structure rely on depthwise separable convolution to increase and reduce the dimension of the feature map in the network.

[0021] S22 is constructed based on SpindleNet into a lightweight bilinear structure RQABS-Net, which increases the dimension of the input 224×224×3-dimensional image I to 14×14×96 dimensions through 13 inversion residual structures and linear bottleneck structures, and then reduces the dimension to 7×7×16 dimensions by repeating the inversion residual structure and linear bottleneck structure 6 times.

[0022] Furthermore, for the feature map I at this time f Perform bilinear pooling to obtain the covariance matrix φ(I f ), and then flatten φ(I from left to right and from top to bottom f ) is 256-dimensional and normalized to obtain a fine-grained feature vector z.

[0023] The image classification branch and the image quality recognition branch are connected after the feature vector z. According to the number of finger vein image categories in the public dataset, the value of the output node k of the image classification branch is set, and the output node of the image quality recognition branch is set to 2 to classify the finger vein images into high-quality and low-quality images.

[0024] Furthermore, S3 includes the following sub-steps:

[0025] S31 uses the image quality dataset constructed by S1 to train the quality recognition branch of RQABS-Net, and uses the region of interest dataset constructed by S1 to train the image classification branch of RQABS-Net, so that the network can extract features with fine-grained differentiation capabilities in finger vein images.

[0026] S32 uses Softmax Cross-Entropy Loss combined with Center Loss to quantify the gap between network output and label:

[0027] Loss=Loss softmax +λLoss center

[0028] Where λ controls the extent to which the central loss contributes to the final loss value.

[0029] Further, S4 includes the following sub-steps:

[0030] S41 uses a light source that emits 850nm near-infrared light to illuminate the registrant's fingers, and uses a near-infrared camera to collect the near-infrared light that passes through the fingers, thereby capturing the vein structure in the fingers and obtaining a finger vein image video stream.

[0031] S42 extracts the region of interest (ROI) of the finger vein image in the collected video stream in real time, and obtains the ROI image data stream by determining the upper and lower boundaries of the finger and the left and right boundaries of the second joint of the finger.

[0032] S43 inputs the data stream into the fully trained RQABS-Net. If the RQABS-Net quality recognition branch determines that the current video frame is of low quality, the registrant is prompted to change his posture and continue to judge the quality of the next frame. When a high-quality input video frame is identified, the feature vector z output by the current RQABS-Net is retained as the identity feature vector of the registrant.

[0033] S44 obtains six identity feature vectors for one registrant, and forms a registrant feature information database with the ID information of multiple registrants and the corresponding identity feature vectors.

[0034] Furthermore, S5 includes the following sub-steps:

[0035] S51 obtains the finger vein image video stream of the person to be identified in the same manner as in S41 and S42.

[0036] S52 inputs the video stream into the fully trained RQABS-Net. If the RQABS-Net quality recognition branch determines that the current video frame is of low quality, it prompts the person to be identified to change his posture and continues to judge the quality of the next frame of the image. When a high-quality input video frame is identified, the feature vector z output by the current RQABS-Net is used as the identity feature vector of the person to be identified.

[0037] Compared with the prior art, the advantages of the present invention are:

[0038] Based on the idea and structure of MobileNet-V2, a new spindle-shaped structure, SpindleNet, is proposed, which reduces the number of parameters and the amount of calculation on the basis of MobileNet-V2. Then, SpindleNet is used to form a bilinear network structure, RQABS-Net, which solves the disadvantage of large feature dimension of the traditional bilinear network output and further reduces the number of parameters and the amount of calculation in the network. Therefore, by reducing the number of parameters and the amount of calculation, the recognition speed of the finger vein recognition system is accelerated; by forming a bilinear convolutional network structure, the system's ability to capture fine-grained differences in finger vein images is improved, thereby improving the recognition accuracy of the system. On the other hand, RQABS-Net combines the recognition of image quality with the extraction and matching of fine-grained features of the image, so that the tasks that should have been completed by two convolutional networks are combined in one convolutional network, thereby using a unified network, reducing the time consumption and storage space loss caused by loading and switching of two convolutional network models. The recognition speed of the system is further improved. The present invention also verifies the performance of the finger vein recognition algorithm and achieves good test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a process diagram of a finger vein image recognition method based on a lightweight bilinear convolutional network according to an embodiment of the invention;

[0040] Figure 2 A schematic diagram of high-quality and low-quality images in a public data set for an embodiment of the invention: wherein a is a high-quality image in SDUMLA; b is a low-quality image in SDUMLA; c is a high-quality image in HKPolyU; d is a low-quality image in HKPolyU;

[0041] Figure 3 Schematic diagram of image preprocessing of an embodiment of the invention; wherein a is the original image; b is the finger edge map; c is the ROI contour calculated based on the finger contour; d is the ROI area of ​​the original image;

[0042] Figure 4 This is a schematic diagram of the structure of SpindleNet, an embodiment of the invention;

[0043] Figure 5 Schematic diagram of modules in MobileNet-V2 of the embodiment of the invention; a is an inverted residual structure; b is a linear bottleneck structure;

[0044] Figure 6 It is a schematic diagram of the structure of the invention embodiment RQABS-Net;

[0045] Figure 7: This is a performance diagram of the RQABS-Net quality recognition branch of the invention embodiment; wherein a is the recognition accuracy of the RQABS-Net quality recognition branch trained by SDUMLA-QD; b is the recognition accuracy of the RQABS-Net quality recognition branch trained by HKPolyU-QD;

[0046] Figure 8 It is a performance diagram of the RQABS-Net image classification branch of the embodiment of the invention; wherein a is the ROC curve and EER value of the RQABS-Net image classification branch trained by SDUMLA-ROI; b is the ROC curve and EER value of the RQABS-Net image classification branch trained by HKPolyU-ROI;

[0047] Fig. 9 : It is a performance diagram of RQABS-Net of the embodiment of the invention; wherein, a is the ROC curve and EER value of RQABS-Net trained by SDUMLA-ROI and SDUMLA-QD; b is the ROC curve and EER value of RQABS-Net trained by HKPolyU-ROI and HKPolyU-QD. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0049] like Figure 1 As shown, the finger vein image recognition method based on a lightweight bilinear convolutional network of the present invention comprises the following steps:

[0050] Step S1: Preprocess the finger vein images in the public dataset. The steps are as follows:

[0051] (1) We selected “high-quality” and “low-quality” finger vein image samples from the SDUMLA and HKPolyU datasets, respectively, and constructed a labeled quality recognition dataset, denoted by SDUMLA-QD and HKPolyU-QD. The finger vein images selected as “high-quality” and “low-quality” are as follows: Figure 2 As shown in Figure 2. However, the number of "high-quality" and "low-quality" images is not balanced, and the number of the latter is too small, which is not conducive to network training. To this end, the number of low-quality images is expanded by the SMOTE method, so that the number of low-quality images increases to a number similar to that of high-quality images.

[0052] (2) Use a low-pass Gaussian filter to filter SDUMLA (such as Figure 3-a), HKPolyU, SDUMLA-QD, and HKPolyU-QD datasets are smoothed, and then the Sobel operator is used to detect the finger edges (as shown in Figure 3 -b), thus determining the upper and lower boundaries. Then, based on the original size of the image, 1 / 5 and 4 / 5 of the long side are used as the left and right boundaries (as shown in Figure 3 -c). Extract the ROI of the finger vein image within the bounding box in the original image (such as Figure 3 -d).

[0053] (3) The ROI-processed data is amplified by random brightness adjustment and random cropping to train the convolutional network and obtain the SDUMLA-ROI, HKPolyU-ROI, SDUMLA-QD, and HKPolyU-QD data sets corresponding to the aforementioned data sets.

[0054] Step S2: Build a lightweight bilinear convolutional network. The steps are as follows:

[0055] (1) Based on the substructure of MobileNet-V2: the inverted residual structure and the linear bottleneck structure, a spindle-shaped convolutional network structure is constructed in which the number of feature map channels c in the first half gradually increases and the number of feature map channels c in the second half gradually decreases. It is named SpindleNet (e.g. Figure 4 , as shown in Table 1). Among them, the inverted residual structure (such as Figure 5 -a) and linear bottleneck structures (such as Figure 5 -b) relies on depthwise separable convolution to increase and reduce the dimension of feature maps in the network.

[0056] (2) Based on SpindleNet, a lightweight bilinear structure RQABS-Net (Recognition and Quality Assessment Bilinear Spindle Net) is constructed (such as Figure 6 , as shown in Table 2), the input 224×224×3 dimensional image I is upgraded to 14×14×96 dimensions through 13 times of inversion residual structure and linear bottleneck structure, and then the inversion residual structure and linear bottleneck structure are repeated 6 times to reduce the dimension to 7×7×16 dimensions.

[0057] Furthermore, for the feature map I at this time f Perform bilinear pooling. Specifically, first calculate I f The covariance matrix φ(I f ):

[0058]

[0059] Where f is the convolution, activation, etc. applied by RQABS-Net on the input image I in the inverted residual structure and linear bottleneck structure. Then flatten φ(I f ) is 256-dimensional and normalized to obtain the fine-grained feature vector z:

[0060]

[0061]

[0062] (3) Connect the image classification branch and the image quality recognition branch after the feature vector z. According to the number of finger vein image categories in the public dataset, set the value of the output node k of the image classification branch, and set the output node of the image quality recognition branch to 2, so as to classify the finger vein images into high-quality and low-quality images. Specifically, the RQABS-Net trained by SDUMLA is set to 636+2 nodes, and the RQABS-Net trained by HKPolyU is set to 312+2 nodes.

[0063] Table 1 SpindleNet structure

[0064]

[0065] Table 2 RQABS-Net structure

[0066]

[0067] Step S3: Train RQABS-Net. The steps are as follows:

[0068] (1) The SDUMLA-ROI, HKPolyU-ROI, SDUMLA-QD, and HKPolyU-QD datasets are divided into training and test sets in a ratio of 7:3. RQABS-Net trained based on SDUMLA and RQABS-Net trained based on HKPolyU are obtained respectively.

[0069] (2) Use Softmax Cross-Entropy Loss combined with Center Loss to quantify the gap between network output and label:

[0070] Loss=Loss softmax +λLoss center

[0071] Where λ controls the degree to which the central loss participates in the quantization of the loss value.

[0072] Separately, calculate the Losssoftmax The part is: the output value of the i-th node output by the image classification branch Calculate the probability P of the i-th class i :

[0073]

[0074] Then we get the loss of each category -y i log(P i ), and then sum to get the total loss value

[0075] Calculate Loss center The part is: calculate the training sample x i Its corresponding category y i The central eigenvector of The gap between:

[0076]

[0077] Among them, in the training process of each batch, the central feature vectors of each category required for the next batch training are obtained

[0078]

[0079] In the formula, is the central feature vector of the jth class feature vector in the current training batch; is the central feature vector of the jth feature vector used in the next training batch; α controls the speed of update; Δc j Represents all samples of the j-th feature in the current training batch and their central feature vector The average difference between:

[0080]

[0081] Where δ is an indicator function, which takes 1 when the value in the brackets is true, and 0 otherwise.

[0082] Step S4: Using RQABS-Net trained with a public data set, a local registrant feature database consisting of multiple registrants is established. The steps are as follows:

[0083] (1) A light source that can emit 850nm near-infrared light is used to illuminate the registrant's finger, and a near-infrared camera is used to collect the near-infrared light that passes through the finger, thereby capturing the vein structure in the finger and obtaining a finger vein image video stream.

[0084] (2) Extract the region of interest (ROI) of the finger vein image in the collected video stream in real time. Generally, the ROI image data stream is obtained by determining the upper and lower boundaries of the finger and the left and right boundaries of the second joint of the finger.

[0085] (3) The ROI image data stream is input into the fully trained RQABS-Net. If the RQABS-Net quality recognition branch determines that the current video frame is of low quality, the registrant is prompted to change his posture and continue to judge the quality of the next frame. When a high-quality input video frame is identified, the feature vector z output by the current RQABS-Net is retained as the identity feature vector of the registrant.

[0086] (4) Obtain six identity feature vectors for one registrant, and form a registrant feature information database with the ID information of multiple registrants and their corresponding identity feature vectors.

[0087] Step S5: Using RQABS-Net trained with a public dataset, extract the features of the finger veins of the person to be identified. The steps are as follows:

[0088] (1) The finger vein image video stream of the person to be identified is obtained in the same manner as in S4(1) and (2).

[0089] (2) The video stream is input into the fully trained RQABS-Net. If the quality recognition branch of RQABS-Net determines that the current video frame is of low quality, it prompts the person to be identified to change his posture and continues to judge the quality of the next frame. When a high-quality input video frame is identified, the feature vector z output by the current RQABS-Net is used as the identity feature vector of the person to be identified.

[0090] Step S6: Identify the individual's identity based on the feature vector of the person to be identified. The steps are as follows:

[0091] (1) Use the feature vector z output by RQABS-Net as the feature vector of the current finger vein image and calculate the cosine similarity with the feature vector in the registered person feature library.

[0092] (2) A threshold is set based on the evaluation results of the public data set, and the output of the matched identity information or the determination of the person to be identified as a stranger is determined based on the threshold.

[0093] The present invention can be widely used in scenarios where identity recognition is required, such as attendance, clocking in, security and other systems.

[0094] The algorithm for finger vein recognition based on the lightweight bilinear network RQABS-Net of the present invention is experimentally verified on the test sets of SDUMLA-ROI, SDUMLA-QD, HKPolyU-ROI, and HKPolyU-QD data.

[0095] The Shandong University Finger Vein Database (SDUMLA) provides finger vein images of 106 people. Each individual has images of 6 fingers, namely the index finger, middle finger and ring finger of the left and right hands. Each finger is collected 6 times, so the dataset contains a total of 3816 (106×6×6) images. All finger vein images in this dataset are grayscale images with a pixel resolution of 320×240. On this basis, ROI extraction and data expansion are performed to form SDUMLA-ROI. After expansion, SDUMLA-ROI contains 12720 images; manual quality classification is performed, and the low-quality images are amplified using the SMOTE method, and finally SDUMLA-QD with 3382 high-quality images and 3472 low-quality images is formed.

[0096] The Hong Kong Polytechnic University Finger Vein Database (HKPolyU) contains finger vein images of 156 volunteers, of which 51 were collected only once, and the other 105 were collected twice with an interval of 1 to 6 months. Each time, the index finger and middle finger of the volunteer's left hand were collected, and 6 513×256 resolution images were collected for each finger. The experiment used the collection of images collected by 156 volunteers for the first time, a total of 1872 (156×2×6) images. On this basis, ROI extraction and data expansion were performed to form HKPolyU-ROI. After expansion, HKPolyU-ROI contains 6240 images; manual quality classification was performed, and the low-quality images were amplified using the SMOTE method, and finally formed HKPolyU-QD with 1722 high-quality images and 1800 low-quality images.

[0097] The above data set is divided into training set and test set in a ratio of 7:3.

[0098] RQABS-Net image quality recognition branch experiment

[0099] Using the binary cross entropy loss function, the quality recognition branch of RQABS-Net is trained on the SDUMLA-QD and HKPolyU-QD training sets respectively. After each epoch of training, a test is performed on the test set. The change of the recognition rate of the test set with the number of training epochs is shown in the figure below. Figure 7 As shown in Figure 2, the recognition rate of image quality on SDUMLA-QD reached 95.31% (as shown in Figure 2 Figure 7-a), the recognition rate of image quality on HKPolyU-QD reached 95.64% (as shown in Figure 7 -b). Compared with other image quality judgment methods, the results are shown in Table 3. From the data, we can see that our proposed RQABS-Net has an advantage in the quality judgment ability of finger vein images.

[0100] Table 3 ACC comparison of different image quality recognition algorithms on SDUMLA-QD and HKPolyU-QD test sets

[0101]

[0102]

[0103] RQABS-Net image classification branch experiment

[0104] Using the multi-classification cross entropy loss function and the center loss function, the image classification branch of RQABS-Net was trained on the SDUMLA-ROI and HKPolyU-ROI training sets respectively. Since the output feature vector z needs to be matched in the subsequent use of RQABS-Net, the matching mode is used here to observe the performance of the image classification branch of RQABS-Net. In order to reduce the calculation time, we randomly selected 100 fingers in SDUMLA-ROI for testing. In heterogeneous matching, each image is matched with 6 images of the other 99 fingers, and 356,400 heterogeneous matching scores can be obtained accordingly. In homologous matching, each image is matched with another 5 images of the same type, thus 3,000 homologous matching scores can be obtained. The experimental results show that the equal error rate (EER) is 0.69% (such as Figure 8 -a). Similarly, 100 fingers were randomly selected in HKPolyU-ROI for testing, and the error rate was 0.63% (as shown in Figure 8 -b).

[0105] Combining RQABS-Net image quality recognition and image classification experiments

[0106] 100 fingers were randomly selected from SDUMLA-ROI and HKPolyU-ROI, and the matching mode was also adopted for testing. Since some low-quality image features will be excluded according to the quality recognition branch results, the number of feature vectors retained for each finger is different. Statistics show that SDUMLA-ROI has 315,866 heterogeneous matches and 2,658 homogeneous matches; HKPolyU-ROI has 327,888 heterogeneous matches and 2,760 homogeneous matches. The experiment shows that the EER of the model trained on SDUMLA-ROI and SDUMLA-QD is 0.64% (e.g. Fig. 9 -a); the EER of the model trained on HKPolyU-ROI and HKPolyU-QD is 0.58% (as shown in Fig. 9 -b). Compared with other finger vein recognition methods, the results are shown in Table 4. From the data, we can see that our proposed RQABS-Net has an advantage in the recognition ability of finger vein images.

[0107] Table 4 Performance comparison of different recognition methods

[0108]

[0109]

[0110] RQABS-Net operation speed experiment

[0111] Using the edge device Raspberry Pi 4b as the experimental platform, the lightweight convolutional network model was loaded using the Intel Movidius 2nd generation neural computing stick, and the forward propagation time of the common lightweight networks MobileNet-V1, MobileNet-V2 and RQABS-Net was compared, that is, the average forward propagation time of 1,000 finger vein images selected from the SDUMLA-ROI and HKPolyU-ROI datasets was counted. It should be noted that in order to eliminate the interference of the model loading time on the statistical value, the time is counted after the model is loaded. The forward propagation time of each model is shown in Table 5. It can be seen that the RQABS-Net we proposed has fewer parameters and calculations, and the forward reasoning speed is faster.

[0112] Table 5 Comparison of network inference speed

[0113]

[0114] Experiment on the transfer recognition ability of RQABS-Net

[0115] Using the RQABS-Net model trained by SDUMLA-QD, SDUMLA-ROI and HKPolyU-QD, HKPolyU-ROI, local personal identification is completed by steps S4 to S6, and the finger vein image and feature library constructed by the local individual are represented by SWUST-FV. During the construction process, a total of 720 finger vein images of 120 fingers were collected, and 514080 heterogeneous matches and 3600 homogeneous matches were obtained. Experiments show that the EER of the model trained on SDUMLA-ROI and SDUMLA-QD is 1.36%; the EER of the model trained on HKPolyU-ROI and HKPolyU-QD is 1.23%. It can be seen that cross-database recognition will lose part of the system recognition accuracy, and the network trained by HKPolyU-ROI and HKPolyU-QD has better migration to SWUST-FV. This is because the grayscale range of the finger vein image and the clarity of the veins in SWUST-FV are closer to HKPolyU.

[0116] It can be seen that the proposed RQABS-Net has achieved good results in relevant indicators on the public finger vein image datasets SDUMLA and HKPolyU, proving that the finger vein image recognition method based on lightweight bilinear convolutional network has high practical value.

[0117] The present invention proposes a finger vein image recognition method based on a lightweight bilinear convolutional network, which can avoid the problem of large network parameters and huge computation caused by the large output dimension of the traditional bilinear network. The present invention first constructs a spindle-shaped convolutional network structure-SpindleNet, then bilinearizes SpindleNet, and sets the network output to two branches, one branch is used to identify the quality of the finger vein image entering from the network input end, so as to eliminate the influence of low-quality images on the recognition rate, and the other branch is used to identify the personal identity represented by the finger vein image. The features of the bilinear network output express the fine-grained distinguishing features of the image, which improves the identity discrimination accuracy of the network. In order to train RQABS-Net, the public data sets of finger vein images SDUMLA and HKPolyU are used, and multi-classification cross entropy loss and center loss are used for training, so that the finger vein image features of the same type are more clustered in the feature space, and the features of different types are more dispersed. Finally, the RQABS-Net trained by the public data set is applied to the actual local personal identity recognition.

[0118] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A finger vein image recognition method based on a lightweight bilinear convolutional network, characterized in that: The following steps are involved: S1: Preprocess finger vein images in the public dataset to obtain finger vein image datasets divided into high-quality and low-quality datasets, as well as region of interest datasets of the corresponding datasets; S2: Build a lightweight bilinear convolutional network whose output is divided into a quality recognition branch and an image classification branch, named RQABS-Net (Recognition and Quality Assessment BilinearSpindle Net); S2 includes the following sub-steps: S21 is based on the substructures of MobileNet-V2: the inverted residual structure and the linear bottleneck structure. It constructs a spindle-shaped convolutional network structure in which the number of feature map channels c in the first half gradually increases and the number of feature map channels c in the second half gradually decreases. It is named SpindleNet. Among them, the inverted residual structure and the linear bottleneck structure rely on depthwise separable convolution to increase and reduce the dimension of the feature map in the network. S22 is constructed based on SpindleNet into a lightweight bilinear structure RQABS-Net, which increases the dimension of the input 224×224×3-dimensional image I to 14×14×96 dimensions through 13 times of inversion residual structure and linear bottleneck structure, and then reduces the dimension to 7×7×16 dimensions by repeating the inversion residual structure and linear bottleneck structure 6 times; Perform bilinear pooling on the feature map at this time to obtain the covariance matrix φ(I f ), and then flatten φ(I from left to right and from top to bottom f ) is 256-dimensional and normalized to obtain a fine-grained feature vector z; Connect the image classification branch and the image quality recognition branch after the feature vector z. According to the number of finger vein image categories in the public dataset, set the value of the output node k of the image classification branch, and set the output node of the image quality recognition branch to 2, so as to classify the finger vein images into high-quality and low-quality images. S3: Use the finger vein image dataset constructed in S1 to train the quality recognition branch of RQABS-Net, and use the region of interest dataset constructed in S1 to train the image classification branch of RQABS-Net; during the training process, multi-classification cross entropy loss function and center loss function are used to quantify the gap between network output and label; S4: Use a finger vein image acquisition device to obtain the finger vein image data stream of the actual registrant, input the data stream into the fully trained RQABS-Net, store the features that are judged as high-quality images according to the results of the RQABS-Net quality recognition branch, and then establish a registered person feature library consisting of multiple registrants; S5: Use a finger vein image acquisition device to obtain a finger vein image data stream of the person to be identified, input the data stream into the same RQABS-Net as in step S4, and decide whether to use the feature vector z output by RQABS-Net to perform identity matching in the registered person feature library according to the result of the RQABS-Net quality recognition branch; S6: Use the feature vector z output by RQABS-Net as the feature vector of the current finger vein image, calculate the cosine similarity with the feature vector in the registered person feature library, and output the matched identity information or determine that the person to be identified is a stranger based on the threshold.

2. According to claim 1, a finger vein image recognition method based on a lightweight bilinear convolutional network is characterized in that: S3 includes the following sub-steps: S31 uses the image quality dataset constructed in S1 to train the quality identification branch of RQABS-Net, and uses the region of interest dataset constructed in S1 to train the image classification branch of RQABS-Net; This enables the network to extract features with fine-grained distinguishing capabilities from finger vein images; S32 uses multi-classification cross entropy loss (Softmax Cross-Entropy Loss) combined with center loss (CenterLoss) to quantify the gap between network output and label: Loss=Loss softmax +λLoss center Where λ controls the extent to which the central loss contributes to the final loss value.

3. The finger vein image recognition method based on a lightweight bilinear convolutional network according to claim 1 is characterized in that: S4 includes the following sub-steps: S41 uses a light source that emits 850nm near-infrared light to illuminate the registrant's finger, and uses a near-infrared camera to collect the near-infrared light that passes through the finger, thereby capturing the vein structure in the finger and obtaining a finger vein image video stream; S42 extracts the region of interest (ROI) of the finger vein image in the collected video stream in real time, and obtains the ROI image data stream by determining the upper and lower boundaries of the finger and the left and right boundaries of the second joint of the finger; S43 inputs the data stream into the well-trained RQABS-Net. If the RQABS-Net quality recognition branch determines that the current video frame is of low quality, the registrant is prompted to change his posture and continue to judge the quality of the next frame. When a high-quality input video frame is identified, the feature vector z output by the current RQABS-Net is retained as the identity feature vector of the registrant. S44 obtains six identity feature vectors for one registrant, and forms a registrant feature information database with the ID information of multiple registrants and the corresponding identity feature vectors.

4. The finger vein image recognition method based on a lightweight bilinear convolutional network according to claim 3 is characterized in that: S5 includes the following sub-steps: S51 obtains a finger vein image video stream of the person to be identified in the same manner as in S41 and S42; S52 inputs the video stream into the fully trained RQABS-Net. If the RQABS-Net quality recognition branch determines that the current video frame is of low quality, it prompts the person to be identified to change his posture and continues to judge the quality of the next frame of the image. When a high-quality input video frame is identified, the feature vector z output by the current RQABS-Net is used as the identity feature vector of the person to be identified.

Images