Comparative learning finger vein recognition method based on hybrid enhancement and time-frequency feature fusion

Abstract

The invention belongs to the technical field of finger vein recognition, and particularly relates to a comparative learning finger vein recognition method based on hybrid enhancement and time-frequency feature fusion, and the method comprises the steps: S1, constructing a deep neural network for finger vein recognition, and the network comprises a backbone network and a mapping head; s2, inserting at least one time-frequency domain feature fusion module into the backbone network; s3, in the training of the deep neural network, using a mixed enhancement sample obtained by mixing two different finger vein images in the training set by a mixed enhancement module; and S4, using the trained deep neural network to perform feature representation and recognition of the finger vein image. According to the method, the technical problem that overfitting occurs on a deep learning model due to poor finger vein image quality and small and single data set is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of finger vein recognition, and in particular relates to a comparative learning finger vein recognition method based on hybrid enhancement and time-frequency feature fusion. Background Art

[0002] The existing finger vein recognition methods are mainly divided into two categories, namely traditional methods and deep learning methods. The difficulty of finger vein recognition research lies in the effective extraction of key features. In the early years, traditional algorithm research was mostly based on geometric structure, local binary features, points and other local invariant features. There are also special methods based on PCA (Principal Component Analysis), HMM (Hidden Markov Model) based signal processing and wavelet transform. Geometric features are features proposed based on the contour, direction angle and other aspects of the finger vein. For example, the repeated line tracking algorithm tracks the finger veins multiple times, identifies the local finger vein texture, and searches pixel by pixel along the texture trajectory to the position without texture, and then randomly tracks from the new position to finally obtain the texture features of the entire image. Local binary features compare pixel gray values ​​and divide pixels into different types to represent the local feature information of the finger vein. Point features refer to pixel point features with position information such as intersections, inflection points and bifurcation points selected by the algorithm. SIFT, SURF and Harris operators are commonly used to obtain point features. SIFT features are invariant to affine transformations and brightness changes, and have a certain effect in reducing interference, but they require high-quality images. Noise blur, uneven grayscale, and color changes make point feature extraction difficult, and the number of feature points to be extracted is large, which affects the matching time. Although traditional algorithms have a considerable recognition rate, they have high requirements for image quality and are sensitive to morphological changes. Complex scenes require a lot of mathematical derivations and manual parameter design, which makes the recognition system generally unavailable.

[0003] With the development of deep learning, more researchers have gradually combined it with vein recognition. Convolutional Neural Networks (CNN) do not require manual design of feature extraction methods and parameters like traditional methods. They are invariant to transformations such as rotation and translation and have strong anti-interference ability. Therefore, deep learning algorithms represented by CNN are gradually being used in the field of finger vein recognition.

[0004] With the application of more and more deep learning methods in this field, the recognition performance has been further improved. However, the quality of finger vein images (blur, uneven illumination, spatial position changes, etc.) still affects the robustness of feature extraction. At the same time, the robustness of features is also subject to the richness of finger vein data (the amount of finger vein data of the same category), the design of loss and activation functions, the rationality of network structure, and whether effective feature information is focused on and extracted. At present, most public finger vein databases have poor image quality, single and not rich changes, and more background information interference from non-finger parts; single-category images are very rare (3-6 images), which is very easy to cause network overfitting and cannot give full play to the advantages of deep learning with large data volumes. Overfitting is specifically manifested in: the model has a lower loss value and higher prediction accuracy on the training data, but has a larger loss value and lower prediction accuracy on the test data. In addition, traditional comparison methods often rely only on a single feature representation, ignoring the complementarity between different feature representations, which may limit the full utilization of sample pairs. Summary of the invention

[0005] The present invention proposes a contrastive learning finger vein recognition method based on hybrid enhancement and time-frequency feature fusion, which solves the technical problem of overfitting in the deep learning model due to poor quality of finger vein images and small and single data set.

[0006] The contrastive learning finger vein recognition method based on hybrid enhancement and time-frequency feature fusion includes the following steps:

[0007] S1 builds a deep neural network for finger vein recognition, which includes the backbone network f encoder and mapping head MLP;

[0008] S2 is on the backbone network encoder Insert at least one time-frequency domain feature fusion module;

[0009] In this module, for a given intermediate feature On the one hand, a two-dimensional fast Fourier transform is used to convert it from the time domain to the frequency domain to obtain the corresponding frequency domain representation Then, the frequency domain features X are extracted through a one-dimensional convolution block F The relevant information in the F ;

[0010] On the other hand, simple convolution is used to encode the intermediate feature X, and we get To capture dependencies in the time dimension;

[0011] Then, in X S and X′ F A linear fusion is performed between them, and the channel interaction between the features of the two domains is established as follows:

[0012]

[0013] in, represents element-wise multiplication;

[0014] Finally, for X linear Perform two-dimensional inverse fast Fourier transform to restore it to time domain features;

[0015] S3 uses the mixed enhancement samples obtained by mixing two different finger vein images in the training set by the mixed enhancement module in the training of the deep neural network;

[0016] First, the finger vein image dataset X = {x1, x2, ···, x n} is divided into X1={x1,x2,···,x n2} and X2={x n2+1 ,x n2+2 ,···,x n};

[0017] In the hybrid enhancement module, the data x is obtained i ∈X1 and data x k ∈X2 is a mixture of enhanced samples

[0018] S4 uses a trained deep neural network to represent and recognize the features of finger vein images.

[0019] Furthermore, in step S2, the number of time-frequency domain feature fusion modules inserted is two.

[0020] Furthermore, in the hybrid enhancement module, the hybrid enhancement samples is defined as follows:

[0021]

[0022] Among them, fft represents two-dimensional fast Fourier transform, ifft represents two-dimensional inverse fast Fourier transform, and λ represents a given weight;

[0023] Furthermore, the loss function used in training includes the following hybrid loss function:

[0024]

[0025] v i ,v k =f encoder (x i ,x k )

[0026] in, Represents similarity distance calculation.

[0027] Progressively, cosine similarity is used for calculation.

[0028] Furthermore, the loss function used in training also includes x i and x k The self-supervised learning loss function between:

[0029]

[0030] Here, h(·) represents the mapping head MLP of the network.

[0031] Furthermore, the overall loss function is defined as and The combination is as follows:

[0032]

[0033] Furthermore, the deep neural network is FVRAS-Net, FV-CNN, PV-CNN or MobileNet.

[0034] The principles and beneficial effects of the present invention are:

[0035] The present invention proposes a method based on contrastive learning to train a more robust feature extractor (deep neural network) to improve recognition performance without increasing the amount of data.

[0036] Among them, the role of the contrastive hybrid enhancement module is to construct positive sample pairs through the contrastive hybrid enhancement method, so that the model can learn more effective representations using the contrast method under limited training data.

[0037] The flexibility and versatility of the TFF module allow it to be easily inserted into any deep neural network (DNNs) to perform fusion learning of the features of the intermediate layer. This innovative design idea is not only simple and easy, but also shows remarkable results in practical applications. In the learning process of deep neural networks, due to their complex structure and a large number of parameters, the network often has a certain preference for certain specific frequency domain components. This preference may cause the model to perform well in certain specific scenarios, but may perform poorly in other scenarios, thereby affecting the generalization ability of the model. Through the fusion learning of the intermediate layer features by the TFF module, the feature information of different domains can be effectively integrated, so that the model can understand the inherent laws of the data more comprehensively. In this way, when facing different scenarios, the model can more flexibly adjust its parameters and outputs, thereby improving its generalization ability.

[0038] The method of the present invention combines the hybrid enhancement technique with the fusion of time-frequency domain features, fully utilizes the information of positive sample pairs and the complementarity of time-frequency domain features, and improves the finger vein recognition performance. DETAILED DESCRIPTION

[0039] This example discloses a contrastive learning finger vein recognition method based on hybrid enhancement and time-frequency domain feature fusion, called CLTF-Mixup (Contrastive Learning-based Finger-Vein Recognition with Time-Frequency Feature Fusion and Mixup). This method aims to use hybrid enhancement techniques to construct positive sample pairs and combine time-frequency domain feature fusion to improve finger vein recognition performance. The overall architecture of CLTF-Mixup is shown in the figure. Figure 1 As shown in the figure, it includes three key parts: mixed enhancement module, time-frequency feature fusion module (TFF module) and feature representation learning. The learning process of CLTF-Mixup is divided into the following steps:

[0040] Firstly, a hybrid enhancement module is used to perform linear interpolation on two different finger vein images to generate sample pairs with rich information, which can enhance the robustness and generalization ability of the model.

[0041] Secondly, the TFF module is added to the backbone network of the model to extract key features from the time domain and frequency domain of the finger vein image: the time domain features mainly reflect the spatial structure and detail information of the image, while the frequency domain features reveal the frequency components and periodic structure of the image. By fusing these two features, their complementarity can be fully utilized to enhance the discriminative ability of the features.

[0042] Finally, the fused features are input into the model to learn the feature representation.

[0043] After learning is completed, the trained model is used to perform feature representation and recognition of finger vein images.

[0044] In this example, the role of the contrastive hybrid enhancement module is to construct positive sample pairs through the contrastive hybrid enhancement method, so that the model can learn more effective representations using the contrast method with limited training data. Similar to the hybrid enhancement method in supervised learning, contrastive hybrid enhancement also merges two images into one image.

[0045] Specifically, firstly, the finger vein image dataset X = {x1, x2, ···, x n}, divided equally into X1={x1,x2,···,x n2} and X2={xn2+1 ,x n2+2 ,···,x n}, in the hybrid enhancement module, data x i ∈X1 and data x k ∈X2 Convex combination mixed data is defined as follows:

[0046]

[0047] Among them, fft represents two-dimensional fast Fourier transform, ifft represents two-dimensional inverse fast Fourier transform, and λ represents a given weight;

[0048] The mixed sample and the original sample are ready-made comparison sample pairs. Input feature learning module backbone network f encoder Learn to extract features, and the obtained features are represented as follows:

[0049]

[0050] It should be noted that x i and x k They are "negative samples" of each other, that is, samples that do not share semantics. Therefore, the feature representation of mixed samples With the feature representation v from the original image i and v k There is a certain degree of positive sample relationship. Learning to capture such a relationship is much more difficult than simply distinguishing between single positive and negative samples, which leads to more effective use of sample information.

[0051] For the generated mixed enhanced samples, the loss function is defined as follows:

[0052]

[0053] in, Refers to the similarity distance, such as cosine similarity, Indicates the mixed similarity between the mixed sample and the original input data.

[0054] x i and x k The self-supervised learning loss function can be written as follows:

[0055]

[0056] Here, h(·) represents the mapping head MLP of the network.

[0057] Finally, the total training loss is defined using the hyperparameter β as and The combination is as follows:

[0058]

[0059] After constructing the comparison samples, the vein image needs to be mapped to the feature space. In this example, a general and effective framework is given for learning the discriminative feature representation of finger veins, namely linear time-frequency feature fusion (TFF). Previous work has ignored the relationship between time domain features and frequency domain features, while the TFF proposed in this example not only utilizes time domain and frequency domain features at the same time, but also enhances representation learning in a more fine-grained manner. Unlike the sum direct connection, TFF uses linear time-frequency fusion to refine the pairwise affinity between time domain features and frequency domain features to produce a more comprehensive feature representation. The resulting fused features can further enhance the discriminability of positive sample pairs and expand the difference of negative pairs.

[0060] Specifically, if Figure 2 As shown, in the TFF module, for a given intermediate feature First, a two-dimensional fast Fourier transform is used to convert it from the time domain to the frequency domain to obtain the corresponding frequency domain representation Next, a simple convolution is used to encode the intermediate feature X to obtain To capture the dependencies in the time dimension; at the same time, the frequency domain features X are extracted through a one-dimensional convolution block F The relevant information in the F In order to further integrate the information of time domain and frequency domain, the TFF module is used in X S and X′ F A linear fusion is performed between them, and the channel interaction between the features of the two domains is established as follows:

[0061]

[0062] in, Represents element-wise multiplication.

[0063] Finally, for X linear Perform two-dimensional inverse fast Fourier transform to restore it to time domain features.

[0064] The linear process fully models X S and X F The fine-grained time-frequency affinity between them. This linear feature transfers fine-grained time-frequency affinity to obtain more discriminative feature representation.

[0065] The flexibility and versatility of the TFF module allows it to be easily inserted into any deep neural network (DNNs) to perform fusion learning of the features of the intermediate layer, and more than one TFF module can be inserted, such as Figure 1 As shown in the figure, two TFF modules are inserted in different positions in the backbone network of this example. This innovative design idea is not only simple and easy to implement, but also shows remarkable results in practical applications. In the learning process of deep neural networks, due to its complex structure and a large number of parameters, the network often has a certain preference for certain specific frequency domain components. This preference may cause the model to perform well in certain specific scenarios, but may perform poorly in other scenarios, thereby affecting the generalization ability of the model. Through the fusion learning of the intermediate layer features by the TFF module, the feature information of different domains can be effectively integrated, so that the model can understand the internal laws of the data more comprehensively. In this way, when facing different scenarios, the model can adjust its parameters and outputs more flexibly, thereby improving its generalization ability.

[0066] Experimental data show that implementing the CLTF-Mixup method given in this example based on deep neural networks such as FVRAS-Net, FV-CNN, PV-CNN, and MobileNet can effectively improve the model's recognition accuracy for finger veins.

Claims

1. A contrastive learning finger vein recognition method based on hybrid enhancement and time-frequency feature fusion, characterized in that: The following steps are involved: S1 builds a deep neural network for finger vein recognition, which includes the backbone network f encoder and mapping head MLP; S2 is on the backbone network encoder Insert at least one time-frequency domain feature fusion module; In this module, for a given intermediate feature On the one hand, a two-dimensional fast Fourier transform is used to convert it from the time domain to the frequency domain to obtain the corresponding frequency domain representation Then, the frequency domain features X are extracted through a one-dimensional convolution block F The relevant information in the F ; On the other hand, simple convolution is used to encode the intermediate feature X, and we get To capture dependencies in the time dimension; Then, in X S and X′ F A linear fusion is performed between them, and the channel interaction between the features of the two domains is established as follows: in, represents element-wise multiplication; Finally, for X linear Perform two-dimensional inverse fast Fourier transform to restore it to time domain features; S3 uses the mixed enhancement samples obtained by mixing two different finger vein images in the training set by the mixed enhancement module in the training of the deep neural network; First, the finger vein image dataset X = {x1, x2, ···, x n } is divided into X1={x1,x2,···,x n2 } and X2={x n2+1 ,x n2+2 ,···,x n }; In the hybrid enhancement module, the data x is obtained i ∈X1 and data x k ∈X2 is a mixture of enhanced samples S4 uses a trained deep neural network to represent and recognize the features of finger vein images.

2. The method according to claim 1, characterized in that In step S2, the number of time-frequency domain feature fusion modules inserted is two.

3. The method according to claim 1, characterized in that In the hybrid enhancement module, hybrid enhancement samples is defined as follows: Among them, fft represents two-dimensional fast Fourier transform, ifft represents two-dimensional inverse fast Fourier transform, and λ represents a given weight.

4. The method according to claim 3, characterized in that: The loss functions used in training include the following hybrid loss functions: v i ,v k =f encoder (x i ,x k ) in, Represents similarity distance calculation.

5. The method according to claim 4, characterized in that The similarity distance is calculated using cosine similarity.

6. The method according to claim 5, characterized in that The loss function used in training also includes x i and x k The self-supervised learning loss function between: Here, h(·) represents the mapping head MLP of the network.

7. The method according to claim 6, characterized in that The overall loss function is defined as follows using the hyperparameter β during training: and The combination is as follows:

8. The method according to claim 1, characterized in that The deep neural network is FVRAS-Net, FV-CNN, PV-CNN or MobileNet.

Images