Palm print and palm vein recognition method in missing mode based on multilevel knowledge distillation

Abstract

The invention discloses a method for recognizing palm prints and palm veins in a missing mode based on multi-level knowledge distillation, and provides three innovation points aiming at the problem that semantic information of different levels of a teacher network cannot be fully mined by a traditional distillation method: (1) knowledge distillation is carried out on three progressive levels of the teacher network, and in addition, the knowledge distillation is carried out on the three progressive levels of the teacher network; logic output is also subjected to fine-grained alignment, and it is ensured that the reasoning network fully learns overall semantic knowledge of the teacher network. (2) proposing a cross-modal feature distribution correlation distillation mechanism, and encouraging the reasoning network to learn representation correlation knowledge among modals instead of focusing on transmission of knowledge in a single modal; and (3) providing structural knowledge of distribution angles and distances of feature vectors of different samples in a high-dimensional space based on structural knowledge distillation among samples guided by classification boundaries, and meanwhile, enabling the inference network to pay more attention to the samples with classification uncertainty through a classification boundary guiding mechanism so as to refine the extracted structural knowledge.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of biometric recognition, and particularly relates to a palmprint and palm vein recognition method based on multi-level knowledge distillation under missing modalities. BACKGROUND

[0002] In terms of privacy protection and information security, a multi-modal biometric recognition system provides a safer and more accurate identity authentication scheme than a single-modal biometric recognition system. The biometric recognition technology based on palmprint and palm vein fusion has attracted more and more attention due to its rich distinguishing characteristics, non-contact and hygienic collection mode. However, in real situations, due to the failure of part of the sensors during collection or the loss of part of the data during transmission, the availability of all collected data cannot be ensured. In this case, the existing multi-modal fusion recognition method will have a serious performance decline, and therefore it is of great practical significance to design a palmprint and palm vein fusion recognition method for uncertain missing modalities.

[0003] Currently, deep learning has achieved outstanding performance in the field of multi-modal biometric recognition. For example, Zhong et al. proposed a multi-modal recognition algorithm based on palmprint and dorsal hand vein, namely Palmprint Hybridized with Dorsal hand vine (PHD). PHD utilizes deep hashing network (DHN) and biometric graph matching (BGM) to process palmprint and dorsal hand vein data, respectively (D. X. Zhong, V. G. G. H. K. Shao, and X. F. Du, “A Hand-Based Multi-Biometrics via Deep Hashing Network and Biometric Graph Matching,” Ieee Transactions on Information Forensics and Security, vol. 14, no. 12, pp. 3140-3150, Dec. 2019.). In addition, Yi et al. proposed a multi-modal fusion recognition method, which includes fingerprint and finger vein. They used asymmetric backbone network (FAB-Net) to extract intra-modality features of fingerprint and finger vein. Subsequently, they introduced a new attention-based encoder fusion network (AEF-Net) for dimensionality reduction-based fusion of multi-modal multi-level features to alleviate the performance degradation caused by information redundancy. AEF-Net also introduced channel attention to further improve performance (Huang Yiwei, G. Get al., “Multimodal Finger Recognition Based on Asymmetric Networks With Fused Similarity,” IEEE Access, vol. 11, 2023, pp. 17497-17509.). However, currently, multi-modal biometric fusion recognition technology often does not consider the damage or loss of multi-modal features in the research modal fusion process, which will affect the robustness and universality of the model. Therefore, how to fully utilize existing information during the training process and still maintain excellent recognition performance in the case of poor quality of some modal data or even loss of modal data is an important problem that needs to be solved. SUMMARY

[0004] The purpose of the present invention is to provide a palm print and vein recognition method under missing modalities based on multi-level knowledge distillation. The present invention addresses the problems existing in the current methods of using knowledge distillation to solve the missing modalities: (1) Most methods only focus on knowledge distillation at the feature level and ignore the mining of overall semantic knowledge from the teacher network, which leads to suboptimal performance; (2) Methods that rely solely on single-modal interaction cannot learn the associated knowledge of cross-modal feature distribution. (3) Insufficient mining of structural knowledge between samples and ignoring the semantic knowledge differences between different samples lead to insufficient learning of modality-specific information during the distillation process. The present invention designs a multi-level knowledge distillation framework. Specifically, the present invention proposes a cross-modal representation correlation distillation, which encourages the deployed network to learn the correlation of representation distributions between different modalities within the sample, and generates excellent cross-modal correlation representations from the teacher network. In addition, the present invention designs a structural knowledge distillation guided by classification boundaries, which focuses on transferring the distance and angle structure knowledge between samples to reconstruct the missing semantics. At the same time, the present invention introduces the concept of classification boundaries. By calculating the distance between different samples and the category boundary, weights are assigned to corresponding samples, guiding the deployed network to inherit robust inter-class boundaries. Finally, we introduce a decoupled alignment distillation strategy that decouples the logical outputs of the teacher network and the inference network into target class outputs and non-target class outputs. This strategy optimizes the inference network's ability to learn from the teacher network by weakening the inhibitory effect of target class response distillation on non-target class response distillation.

[0005] The technical solution to achieve the purpose of the present invention is: a palm print and palm vein recognition method in missing mode based on multi-level knowledge distillation, the steps are as follows:

[0006] Step 1: Divide the palm print and palm vein image database into a training set and a test set at a ratio of 1:1. Assume that some palm print images and palm vein images in the test set are missing, with a missing rate of μ, and proceed to step 2.

[0007] Step 2: Use ResNet18 to build the palm vein feature extractor of the original teacher network Palm vein feature extractor based on inference network And the palmprint feature extractor that builds the original teacher network Palmprint feature extractor based on inference network Go to step 3.

[0008] Step 3: Input the palm print and palm vein images in the training set into the original teacher network, train the original teacher network using cross entropy loss, obtain the teacher network, and proceed to step 4.

[0009] Step 4: Set the parameters in the teacher network to the frozen state, and input the palm print and palm vein images in the training set into the teacher network to obtain the palm print features of the teacher network. and palmar vein characteristics At the same time, the palm print and palm vein images input into the teacher network are introduced into the Bernoulli indicator to generate missing palm print and palm vein images, and the generated missing palm print and palm vein images are input into the inference network to obtain the palm print features of the inference network and palmar vein characteristics Simultaneously calculate the palmprint features of the teacher network and palmar vein characteristics Distribution differences and palmprint features of the inference network and palmar vein characteristics The distribution difference is calculated using the L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD , go to step 5.

[0010] Step 5: Based on the palmprint features of the teacher network and palmar vein characteristics Get the fusion feature S of the teacher's palm print feature and palm vein feature T ; Using palmprint features of inference network and palmar vein characteristics Get the fusion feature S of the inferred palm print feature and palm vein feature S , calculate the fusion feature S in a batch of the teacher network T The angle and distance between them correspond to the angle relationship matrix of the teacher network and distance relationship matrix Calculate the fusion feature S in a batch of the inference network S The angle and distance between them, and the angle relationship matrix of the inference network is obtained and distance relationship matrix The fusion features S of a batch in the teacher network T Send it to the classifier to get the teacher network logic output y T , and the teacher network logic output y T and the label l of the input palm vein image T Calculate the cross entropy between them to get the distance coefficient α, and transform the angle relationship matrix of the teacher network into and the angle relationship matrix of the inference network Element-by-element subtraction, the distance relationship matrix of the teacher network And the distance relationship matrix of the inference network Subtract each element to obtain the angle difference matrix and the distance difference matrix. Then add each column of the angle difference matrix and the distance difference matrix to obtain the angle difference vector V. A and the distance difference vector V D ; The angle difference vector V A and the distance difference vector VD respectively multiplied by the distance coefficient a, to obtain the weighted angle difference vector V α-A and the weighted distance difference vector V α-D , and the weighted difference vector distribution difference is calculated using the L2 loss to obtain the cross-sample structure distillation loss L guided by the classification boundary SKD , and step 6 is entered.

[0011] Step 6, the fusion feature S of the inference network S is sent to the classifier to obtain the logical output y of the inference network S ; at the same time, the logical output y of the teacher network T is decoupled into target class output and non-target class output The logical output y of the inference network S is decoupled into target class output and non-target class output The decoupled alignment distillation loss L is obtained by calculating the difference between the target class output and the non-target class output in the teacher network and the inference network using the KL loss DAD , and finally the cross-entropy loss is used to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, so as to obtain the trained inference network, and step 7 is entered.

[0012] Step 7, the accuracy of the trained inference network is evaluated by using the test set.

[0013] Compared with the prior art, the advantages of the present application are:

[0014] (1) The present application proposes to perform knowledge distillation on three progressive levels of the teacher network. In addition, we also perform fine alignment on the high-level semantic logical output.

[0015] (2) The present application proposes a cross-modal feature distribution correlation distillation mechanism to encourage the inference network to learn the spatial correlation knowledge between representations to deliver meaningful supervision signals.

[0016] (3) The present application proposes a sample inter-structure knowledge distillation method based on classification boundary guidance for extracting structural knowledge between different samples. At the same time, we focus on samples far from the class center to help restore the missing specific modal features in the inference network. BRIEF DESCRIPTION OF DRAWINGS

[0017] Fig. 1 is the flowchart of the missing modal palmprint and palm vein recognition method based on multi-level knowledge distillation of the present application.

[0018] Fig. 2 is the model diagram of the missing modal palmprint and palm vein recognition method based on multi-level knowledge distillation of the present application.

[0019] Fig. 3 A model diagram of the cross-sample structure knowledge distillation method based on classification boundary guidance of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application are described in further detail below.

[0021] In combination Figs. 1-3 The missing modality based palm vein and palm print recognition method based on multi-level knowledge distillation of the present application comprises the following steps:

[0022] Step 1, collect palm print and palm vein images of a plurality of individuals, collect 10 palm print images and 10 palm vein images for each individual, and establish a palm print and palm vein image database; the palm print and palm vein image database is divided into a training set and a test set according to 1:1, and the palm print images and the palm vein images in the test set are partially missing, and the missing rate is 0 ， Go to Step 2.

[0023] Step 2, use ResNet18 to construct a palm vein feature extractor of the original teacher network and a palm vein feature extractor of the inference network and a palm print feature extractor of the original teacher network and a palm print feature extractor of the inference network

[0024] Specifically, the last layer classifier of the convolutional neural network ResNet18 is removed to obtain a new convolutional neural network; the new convolutional neural network is connected with the spatiotemporal attention mechanism CSAFM to construct a palm vein feature extraction network of the original teacher network and a palm vein feature extraction network of the inference network and a palm print feature extraction network of the original teacher network and a palm print feature extraction network of the inference network Use and the classifier to construct the original teacher network, use and the classifier to construct the inference network, and go to Step 3.

[0025] Step 3, input the palm print and palm vein images in the training set into the original teacher network, train the original teacher network using the cross-entropy loss, complete the pre-training phase, obtain the teacher network, and go to Step 4.

[0026] Step 4. Existing knowledge distillation methods force the inference network to completely imitate the modality-specific features of the teacher network, which may lead to overfitting. It only focuses on partial information of a single modality and ignores the correlation knowledge of cross-modal representations in the distribution space. In order to accurately capture the representation information in the teacher network and restore the missing semantic information, this method proposes a cross-modal representation correlation distillation. This mechanism does not force the inference network to imitate the representation of a single modality in the teacher network, but learns the correlation between the semantic distributions of different modalities. Set the parameters in the teacher network to a frozen state, and input the palmprint and palm vein images in the training set into the teacher network to obtain the palmprint features of the teacher network. and palmar vein characteristics At the same time, the palm print and palm vein images input into the teacher network are introduced into the Bernoulli indicator to generate missing palm print and palm vein images, and the generated missing palm print and palm vein images are input into the inference network to obtain the palm print features of the inference network and palmar vein characteristics Simultaneously calculate the palmprint features of the teacher network and palmar vein characteristics Distribution differences and palmprint features of the inference network and palmar vein characteristics The distribution difference is calculated using the L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD , as follows:

[0027] Step 4.1: Set the parameters in the teacher network to the frozen state, and input the palm print and palm vein images in the training set into the teacher network to obtain the palm print features of the teacher network. and palmar vein characteristics At the same time, the palm print and palm vein images input into the teacher network are introduced into the Bernoulli index to generate the palm print and palm vein images missing between the modalities, and the missing palm print and palm vein images between the modalities are input into the inference network to obtain the palm print features of the inference network and palmar vein characteristics

[0028]

[0029] in, and represents the palm print and palm vein images input into the teacher network, and Represents the missing palm print and palm vein images between modalities as input to the inference network.

[0030] Step 4.2: Calculate the palmprint features of the teacher network and palmar vein characteristics Distribution differences and palmprint features of the inference network and palmar vein characteristics The distribution difference is calculated using the L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD :

[0031]

[0032] L CRCD =||R T -R S ||2 (8),

[0033] Among them, R ij Represents the cross-modal distribution correlation coefficient matrix, i represents the row of the matrix, j represents the column of the matrix, b represents the number of feature samples in a batch, R T and R S Represent the cross-modal distribution difference matrices of the teacher network and the inference network, represents the k-th row vector in matrix A, Represents the lth row vector in matrix B, and matrix A represents the palmprint features The matrix composed of the palm vein features is represented by matrix B. The matrix formed, ||.||2 is the L2 norm, go to step 5.

[0034] Step 5. In the existing knowledge distillation methods for missing tasks, the focus is mainly on the interaction of information in a single sample, but lacks a comprehensive exploration of the overall semantic structure across samples in the teacher network. At the same time, distillation methods based on inter-sample relations usually treat different sample instances equally, ignoring the semantic differences between samples. This results in the inability to transfer accurate structural information. To solve this problem, we propose a sample structure relationship distillation mechanism to transfer complete semantic structure information. This method encodes the semantic information of the teacher network in the full modality by capturing the structural relationship of distance and angle between samples, and penalizes the structural differences between the inference network and the teacher network samples. In addition, we introduce the concept of classification boundary to emphasize samples close to the boundary during information transfer. However, directly calculating the distance from the sample to the classification boundary is challenging, but the cluster center of each sample is known. Therefore, we measure the distance from the teacher network output to the category label as a weighting coefficient to re-evaluate the contribution of each sample to the distillation loss. Specifically, based on the palmprint features of the teacher network and palmar vein characteristics Get the fusion feature S of the teacher's palm print feature and palm vein feature T ; Using palmprint features of inference network and palmar vein characteristics Get the fusion feature S of the inferred palm print feature and palm vein featureS , the angles and distances between the fused features S T in a batch of the teacher network are calculated, and the angle relationship matrix and distance relationship matrix of the teacher network are obtained The angles and distances between the fused features S S in a batch of the inference network are calculated, and the angle relationship matrix and distance relationship matrix of the inference network are obtained The fused features S T in a batch of the teacher network are sent to the classifier to obtain the logical output y T of the teacher network, and the cross entropy between the logical output y T of the teacher network and the label l T of the input palmprint palm vein image is calculated to obtain the distance coefficient a, the angle relationship matrix of the teacher network and the angle relationship matrix of the inference network are subtracted element by element, the distance relationship matrix of the teacher network and the distance relationship matrix of the inference network are subtracted element by element, and the angle difference matrix and the distance difference matrix are obtained, and then each column of the angle difference matrix and the distance difference matrix is added respectively, and the angle difference vector V A and the distance difference vector V D are obtained; the angle difference vector V A and the distance difference vector V D are multiplied by the distance coefficient a respectively, and the weighted angle difference vector V α-A and the weighted distance difference vector V α-D are obtained, and the distribution difference of the weighted difference vector is calculated by using the L2 loss, and the cross-sample structure distillation loss L based on the classification boundary guidance is obtained SKD , which is as follows:

[0035] Step 5.1, the angles and distances between the fused features S T in a batch of the teacher network are calculated, and the angle relationship matrix and distance relationship matrix of the teacher network are obtained The angles and distances between the fused features S S in a batch of the inference network are calculated, and the angle relationship matrix and distance relationship matrix of the inference network are obtained

[0036] V1(n,m)=||n-m||2 (9),

[0037]

[0038] ​​​​

[0039] where Γ represents the transpose operation of a matrix, n and m represent a single element in the sample feature vector,

[0040] and respectively represent the feature vectors of the xth and yth samples in a batch of the teacher network, and respectively represent the feature vectors of the xth and yth samples in a batch of the inference network; V1 represents the distance difference coefficient, and V2 represents the angle difference coefficient.

[0041] Step 5.2, the fusion features S T of a batch in the teacher network are sent to the classifier to obtain the logical output y T of the teacher network, and the cross entropy between the logical output y T of the teacher network and the label l T of the input palm vein image is calculated to obtain the distance coefficient a:

[0042] a(x th ) = δ(-l T (x th )*log(y T )) (15),

[0043] where δ(.) represents the softmax function, and l T (x th ) represents the label corresponding to the xth sample.

[0044] Step 5.3, the angle relationship matrix of the teacher network and the angle relationship matrix of the inference network are subtracted element by element, the distance relationship matrix of the teacher network and the distance relationship matrix of the inference network are subtracted element by element, and the angle difference matrix and the distance difference matrix are obtained, then the columns of the angle difference matrix and the distance difference matrix are added respectively to obtain the angle difference vector V A and the distance difference vector V D , the angle difference matrix V A and the distance difference matrix V D are multiplied by the distance coefficient a respectively to obtain the weighted angle difference matrix V α-A and the weighted distance difference matrix V α-D , and the L2 loss is used to calculate the distribution difference of the weighted difference matrix to obtain the cross-sample structure distillation loss L SKD based on the classification boundary guidance:

[0045]

[0046]

[0047] where b represents the number of feature samples in a batch, i represents the row number of the feature matrix, and step 6 is entered.

[0048] Step 6, the fusion feature S S of the inference network is input into the classifier to obtain the logical output y S of the inference network; meanwhile, the logical output y T of the teacher network is decoupled into target class output and non-target class output The logical output y S of the inference network is decoupled into target class output and non-target class output The decoupled alignment distillation loss L DAD is obtained by calculating the difference between the target class output and the non-target class output in the teacher network and the inference network by using the KL loss; finally, the cross-entropy loss is used to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, thereby obtaining the trained inference network, as follows:

[0049] The fusion feature S T of the teacher network and the fusion feature S S of the inference network are input into respective classifiers to obtain corresponding logical outputs y T and y S ; meanwhile, the logical output y T of the teacher network is decoupled into target class output and non-target class output The logical output y S of the inference network is decoupled into target class output and non-target class output The decoupled alignment distillation loss L DAD is obtained by calculating the difference between the target class output and the non-target class output in the teacher network and the inference network by using the KL loss; finally, the cross-entropy loss is used to ensure that the maximum probability output of the classifier is consistent with the actual label probability, combined with the cross-modal feature distribution correlation distillation loss L CRCD , the cross-sample structure distillation loss L SKD and the decoupled alignment distillation loss L DAD together train, thereby obtaining the trained inference network:

[0050] y S = Classifier S (S S ) (19),

[0051] y TClassifier T (S T ) (20),

[0052]

[0053] L MKD =L ClS +λ1L CRCD +λ2L SKD +L DAD (24),

[0054] where De(.) denotes the decoupling operation, Classifier S and Classifier T denote the classifier of the inference network and the classifier of the teacher network respectively, δ is a weighted hyper-parameter for the target class logical knowledge distillation, β is a weighted hyper-parameter for the non-target class logical knowledge distillation, KL(.) denotes a KL divergence calculation function, λ1 denotes a weighted hyper-parameter of the cross-modal feature distribution correlation distillation loss L CRCD , and λ2 denotes a weighted hyper-parameter of the cross-sample structure distillation loss L SKD guided by the classification boundary, L ClS denotes a cross-entropy loss, and L MKD denotes a multi-level distillation loss, and step 7 is entered.

[0055] Step 7, the accuracy of the trained inference network is evaluated by using a test set, as follows:

[0056] Step 7.1, during testing, if both the palmprint image and the palm vein image are not missing, then the features of each modality are directly fused for recognition; if the palmprint image or the palm vein image is missing, we use 0 elements to fill the input of the missing image.

[0057] Step 7.2, during the recognition process, the correct recognition rate is used to evaluate the performance of the inference network; N True denotes the cumulative number of correctly predicted samples, N Flace denotes the cumulative number of incorrectly predicted samples, and then the correct recognition rate CRR is defined as:

[0058]

[0059] Embodiment 1

[0060] The palmprint and palm vein recognition method under missing modalities based on multi-level knowledge distillation provided by the application comprises the following steps:

[0061] Step 1, A self-made palmprint and palm vein database named CUMT was used, which contains 290 volunteers' palm images. Each volunteer provided 10 images, representing two different biometric modalities. Therefore, the entire database contains 290 different classes, a total of 5800 images. These images were collected in two specific time periods, and the time interval between the two collections was more than 10 days. In each collection, 5 palmprint images and 5 palm vein image samples were obtained from each volunteer. The original images collected were then marked with coordinate points and cropped to extract the image region of interest. In the experiment, we divided the dataset into training and test sets in a 1:1 ratio. The training set contains 2900 images, and the test set also contains 2900 images, with some palmprint and palm vein images missing.

[0062] Step 2, ResNet18 was used to build the palm vein feature extraction network of the original teacher network and the palm vein feature extraction network of the inference network as well as the palmprint feature extraction network of the original teacher network and the palmprint feature extraction network of the inference network

[0063] Step 3, Input the palmprint and palm vein images in the training set into the original teacher network, train the original teacher network using cross-entropy loss, complete the pre-training phase, and obtain the teacher network.

[0064] Step 4, Set the parameters in the teacher network to a frozen state, and input the palmprint and palm vein images in the training set into the teacher network to obtain the palmprint features and palm vein features of the teacher network. At the same time, introduce the palmprint and palm vein images input into the teacher network into the Bernoulli indicator to generate missing palmprint and palm vein images, and input the generated missing palmprint and palm vein images into the inference network to obtain the palmprint features and palm vein features of the inference network. Calculate the distribution difference of the palmprint features and palm vein features of the teacher network, as well as the distribution difference of the palmprint features and palm vein features of the inference network, and calculate the distribution difference using L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD , as follows:

[0065] Step 4.1, Set the parameters in the teacher network to a frozen state, and input the palmprint and palm vein images in the training set into the teacher network to obtain the palmprint features and palm vein features The palmprint and palm vein images input into the teacher network are introduced into the Bernoulli index to generate the inter-modal missing palmprint and palm vein images, which are input into the inference network to obtain the palmprint features of the inference network and palm vein features

[0066]

[0067] wherein, and denote the palmprint and palm vein images input into the teacher network, and denote the inter-modal missing palmprint and palm vein images input into the inference network.

[0068] Step 4.2, calculating the palmprint features of the teacher network and palm vein features of the teacher network, and calculating the distribution difference of the palmprint features and palm vein features of the inference network, and calculating the distribution difference by using L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD :

[0069]

[0070]

[0071] L CRCD =||R T -R S ||2 (8),

[0072] wherein, R ij denotes the cross-modal distribution correlation coefficient matrix, b denotes the number of samples in a batch, R T and R S respectively represent the cross-modal distribution difference matrices of the teacher network and the inference network, denotes the kth row vector in matrix A, denotes the lth row vector in matrix B, matrix A denotes a matrix composed of palmprint features , and matrix B denotes a matrix composed of palm vein features , and ||.||2 is L2 norm;

[0073] Step 5, obtaining the fusion features S T of the teacher palmprint features and palm vein features according to the palmprint features and palm vein features of the teacher network; and using the palmprint features And palm vein features Obtaining a fusion feature S of the inferred palmprint feature and the palm vein feature S , the angle and distance between the fusion features S in a batch of the teacher network are calculated, and the angle relationship matrix and the distance relationship matrix of the teacher network are obtained T The angle and distance between the fusion features S in a batch of the inference network are calculated, and the angle relationship matrix and the distance relationship matrix of the inference network are obtained S The fusion features S in a batch of the teacher network are sent to the classifier to obtain the logical output y of the teacher network T , and the cross entropy between the logical output y of the teacher network and the label l T is calculated to obtain a distance coefficient α T , the angle relationship matrix of the teacher network and the angle relationship matrix of the inference network are subtracted element by element, the distance relationship matrix of the teacher network and the distance relationship matrix of the inference network are subtracted element by element, and the angle difference matrix and the distance difference matrix are obtained, and then each column of the angle difference matrix and the distance difference matrix is added, and the angle difference vector V T and the distance difference vector V A are obtained D ; the obtained angle difference vector V A and the distance difference vector V D are multiplied by the distance coefficient α respectively to obtain the weighted angle difference vector V α-A and the weighted distance difference vector V α-D , and the L2 loss is used to calculate the distribution difference of the weighted difference vector, and the cross-sample structure distillation loss L based on the classification boundary guidance is obtained SKD , which is specifically as follows:

[0074] Step 5.1, the angle and distance between the fusion features S in a batch of the teacher network are calculated, and the angle relationship matrix and the distance relationship matrix of the teacher network are obtained T Step 5.2, the angle and distance between the fusion features S in a batch of the inference network are calculated, and the angle relationship matrix and the distance relationship matrix of the inference network are obtained S

[0075] V1(n,m)=||n-m||2 (9),

[0076]

[0077] where Γ represents the transpose operation of a matrix, n and m represent a single element in the sample feature vector,

[0078] and respectively represent the feature vectors of the xth and yth samples in a batch of the teacher network, and respectively represent the feature vectors of the xth and yth samples in a batch of the inference network; V1 represents the distance difference coefficient, and V2 represents the angle difference coefficient.

[0079] Step 5.2, the fusion features S T of a batch in the teacher network are sent to the classifier to obtain the logical output y T of the teacher network, and the cross entropy between the logical output y T of the teacher network and the label l T of the input palm vein image is calculated to obtain the distance coefficient a:

[0080] a(x th ) = δ(-l T (x th )*log(y T )) (15),

[0081] where δ(.) represents the softmax function, and l T (x th ) represents the label corresponding to the xth sample.

[0082] Step 5.3, the angle relationship matrix V of the teacher network and the angle relationship matrix V of the inference network are subtracted element by element, the distance relationship matrix V of the teacher network and the distance relationship matrix V of the inference network are subtracted element by element, and the angle difference matrix V A and the distance difference matrix V D are obtained, then each column of the angle difference matrix V A and the distance difference matrix V D is added to obtain the angle difference vector V α-A and the distance difference vector V α-D , the angle difference matrix V A and the distance difference matrix V D are multiplied by the distance coefficient a respectively to obtain the weighted angle difference matrix V α-A and the weighted distance difference matrix V α-D , and the L2 loss is used to calculate the distribution difference of the weighted difference matrix to obtain the cross-sample structure distillation loss L SKD based on the classification boundary guidance:

[0083]

[0084] Where b represents the number of samples in a batch, and i represents the number of rows in the feature matrix.

[0085] Step 6: The fusion feature S of the inference network S After being sent to the classifier, the inference network logic output y is obtained S ; At the same time, the teacher network logic output y T Decoupling to target class output and non-target class output Output the inference network logic y S Decoupling to target class output and non-target class output The KL loss is used to calculate the difference between the target class output and the non-target class output in the teacher network and the inference network to obtain the decoupled alignment distillation loss L DAD Finally, the cross entropy loss is used to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, thereby obtaining a trained inference network, as follows:

[0086] The fusion feature S of the teacher network T , the fusion feature S of the inference network S Input into each classifier to get the corresponding logical output y T 、y S ; At the same time, the teacher network logic output y T Decoupling to target class output and non-target class output Output the inference network logic y S Decoupling to target class output and non-target class output The KL loss is used to calculate the difference between the target class output and the non-target class output in the teacher network and the inference network to obtain the decoupled alignment distillation loss L DAD Finally, the cross entropy loss is used to ensure that the maximum probability output of the classifier is consistent with the probability of the actual label, combined with the cross-modal feature distribution correlation distillation loss L CRCD , cross-sample structure distillation loss L SKD and decoupled alignment distillation loss L DAD Train together to obtain a trained inference network:

[0087] y S =Classifier S (S S ) (19),

[0088] y T =Classifier T (S T ) (20),

[0089]

[0090] L MKD =L ClS +λ1L CRCD +λ2L SKD +L DAD (24),

[0091] where De(.) denotes the decoupling operation, Classifier S and Classifier T denote the classifier of the inference network and the teacher network respectively, δ is the weighting hyper-parameter of the target class logical knowledge distillation, β is the weighting hyper-parameter of the non-target class logical knowledge distillation, KL(.) denotes the KL divergence calculation function, λ1 denotes the weighting hyper-parameter of the cross-modal feature distribution correlation distillation loss L CRCD , and λ2 denotes the weighting hyper-parameter of the cross-sample structure distillation loss L SKD guided by the classification boundary, L ClS denotes the cross-entropy loss, and L MKD denotes the multi-level distillation loss.

[0092] Step 7, use the test set to evaluate the accuracy of the trained inference network.

[0093] Step 7.1, during the test, if both the palmprint image and the palm vein image are not missing, then directly fuse the modal features for recognition; if the palmprint image or the palm vein image is missing, we use 0 elements to fill the input of the missing image;

[0094] Step 7.2, during the recognition process, use the correct recognition rate to infer the performance of the network; use N True to represent the cumulative number of correctly predicted samples, and N Flace to represent the cumulative number of incorrectly predicted samples, then the correct recognition rate CRR is defined as:

[0095]

[0096] The present application uses the Python programming language and the network framework built by the pytorch framework to conduct related experiments on the Nvidia 1050Ti GPU host. We set the batch-size to 8, and use the stochastic gradient descent (SGD) algorithm to optimize the total loss. In order to speed up the network training speed, we choose the optimization strategy of Momentum as 0.9, and the weightdecay as 1e -5In the experiment, we use the Dropout layer to prevent the network from overfitting problem, and set the Dropout value to 0.5. In this experiment, we regard two modalities from the same palm as the same class, and do not perform any form of expansion on the training data to maximize the testing of the real situation of palmprint and palm vein.

[0097] The experiment is divided into the case that the palmprint is missing or the palm vein is missing in the test set, and the missing rate μ is 0, 20, 50, 80 and 100. The experimental results are shown in Tables 1 and 2, wherein 0 represents that the modalities are not missing.

[0098] Table 1 Recognition results of palmprint missing

[0099]

[0100] Table 2 Recognition results of palm vein missing

[0101]

[0102] According to the experimental results shown in Tables 1 and 2, the method proposed by the application can achieve excellent recognition effect whether the palmprint is missing or the palm vein is missing, and the missing rate has little effect on it. Even when one modality is 100% missing, it can still achieve good effect, and the model has good robustness.

Claims

1. A palm print and palm vein recognition method based on multi-level knowledge distillation in missing mode, characterized by: The following steps are involved: Step 1: Divide the palm print and palm vein image database into a training set and a test set at a ratio of 1:

1. Assume that some palm print images and palm vein images in the test set are missing, and the missing rate is μ. Then proceed to step 2. Step 2: Use ResNet18 to build the palm vein feature extractor of the original teacher network Palm vein feature extractor based on inference network And the palmprint feature extractor that builds the original teacher network Palmprint feature extractor based on inference network Go to step 3; Step 3: Input the palm print and palm vein images in the training set into the original teacher network, train the original teacher network using cross entropy loss, obtain the teacher network, and proceed to step 4; Step 4: Set the parameters in the teacher network to the frozen state, and input the palm print and palm vein images in the training set into the teacher network to obtain the palm print features of the teacher network. and palmar vein characteristics At the same time, the palm print and palm vein images input into the teacher network are introduced into the Bernoulli indicator to generate missing palm print and palm vein images, and the generated missing palm print and palm vein images are input into the inference network to obtain the palm print features of the inference network and palmar vein characteristics Simultaneously calculate the palmprint features of the teacher network and palmar vein characteristics Distribution differences and palmprint features of the inference network and palmar vein characteristics The distribution difference is calculated using the L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD , go to step 5; Step 5: Based on the palmprint features of the teacher network and palmar vein characteristics Get the fusion feature S of the teacher's palm print feature and palm vein feature T ; Using palmprint features of inference network and palmar vein characteristics Get the fusion feature S of the inferred palm print feature and palm vein feature S , calculate the fusion feature S in a batch of the teacher network T The angle and distance between them correspond to the angle relationship matrix of the teacher network and distance relationship matrix Calculate the fusion feature S in a batch of the inference network S The angle and distance between them, and the angle relationship matrix of the inference network is obtained and distance relationship matrix The fusion features S of a batch in the teacher network T Send it to the classifier to get the teacher network logic output y T , and the teacher network logic output y T and the label l of the input palm vein image T Calculate the cross entropy between them to get the distance coefficient α, and transform the angle relationship matrix of the teacher network into and the angle relationship matrix of the inference network Element-by-element subtraction, the distance relationship matrix of the teacher network And the distance relationship matrix of the inference network Subtract each element to obtain the angle difference matrix and the distance difference matrix. Then add each column of the angle difference matrix and the distance difference matrix to obtain the angle difference vector V. A and the distance difference vector V D ; The angle difference vector V A and the distance difference vector V D Multiply them by the distance coefficient α to get the weighted angle difference vector V α-A and weighted distance difference vector V α-D , and use L2 loss to calculate the weighted difference vector distribution difference to obtain the cross-sample structure distillation loss L based on classification boundary guidance SKD , go to step 6; Step 6: The fusion feature S of the inference network S After being sent to the classifier, the inference network logic output y is obtained S ; At the same time, the teacher network logic output y T Decoupling to target class output and non-target class output Output the inference network logic y S Decoupling to target class output and non-target class output The KL loss is used to calculate the difference between the target class output and the non-target class output in the teacher network and the inference network to obtain the decoupled alignment distillation loss L DAD Finally, the cross entropy loss is used to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, thereby obtaining a trained inference network and proceeding to step 7; Step 7: Use the test set to evaluate the accuracy of the trained inference network.

2. The palm print and palm vein recognition method based on multi-level knowledge distillation in missing mode according to claim 1 is characterized in that: In step 1, 0<μ<1.

3. The palm print and palm vein recognition method based on multi-level knowledge distillation in missing mode according to claim 2 is characterized in that: In step 2, the palm vein feature extractor of the original teacher network is constructed using ResNet18 Palm vein feature extractor based on inference network And the palmprint feature extractor that builds the original teacher network Palmprint feature extractor based on inference network The details are as follows: Remove the last layer classifier of the convolutional neural network ResNet18 to obtain a new convolutional neural network; connect the new convolutional neural network with the spatiotemporal attention mechanism CSAFM to construct a palm vein feature extraction network of the original teacher network Palm vein feature extraction network based on inference network And the palmprint feature extraction network that builds the original teacher network Palmprint feature extraction network based on inference network use and classifier to build the original teacher network, using and classifiers to build inference networks.

4. The palm print and palm vein recognition method based on multi-level knowledge distillation in missing mode according to claim 3 is characterized in that: In step 4, the parameters in the teacher network are set to the frozen state, and the palm print and palm vein images in the training set are input into the teacher network to obtain the palm print features of the teacher network. and palmar vein characteristics At the same time, the palm print and palm vein images input into the teacher network are introduced into the Bernoulli indicator to generate missing palm print and palm vein images, and the generated missing palm print and palm vein images are input into the inference network to obtain the palm print features of the inference network and palmar vein characteristics Simultaneously calculate the palmprint features of the teacher network and palmar vein characteristics Distribution differences and palmprint features of the inference network and palmar vein characteristics The distribution difference is calculated using the L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD , as follows: Step 4.1: Set the parameters in the teacher network to the frozen state, and input the palm print and palm vein images in the training set into the teacher network to obtain the palm print features of the teacher network. and palmar vein characteristics At the same time, the palm print and palm vein images input into the teacher network are introduced into the Bernoulli index to generate the palm print and palm vein images missing between the modalities, and the missing palm print and palm vein images between the modalities are input into the inference network to obtain the palm print features of the inference network and palmar vein characteristics in, and represents the palm print and palm vein images input into the teacher network, and Represents the missing palm print and palm vein images between modalities input to the inference network; Step 4.2: Calculate the palmprint features of the teacher network and palmar vein characteristics Distribution differences and palmprint features of the inference network and palmar vein characteristics The distribution difference is calculated using the L2 loss to obtain the cross-modal feature distribution correlation distillation loss L CRCD : L CRCD =||R T -R S ||2 (8), Among them, R ij represents the cross-modal distribution correlation coefficient matrix, b represents the number of feature samples in a batch, R T and R S Represent the cross-modal distribution difference matrices of the teacher network and the inference network, represents the k-th row vector in matrix A, Represents the lth row vector in matrix B, and matrix A represents the palmprint features The matrix composed of the palm vein features is represented by matrix B. The matrix formed by ||.||2 is the L2 norm.

5. The palm print and palm vein recognition method based on multi-level knowledge distillation in missing mode according to claim 4 is characterized in that: In step 5, the fusion feature S in a batch of the teacher network is calculated T The angle and distance between them correspond to the angle relationship matrix of the teacher network and distance relationship matrix Calculate the fusion feature S in a batch of the inference network S The angle and distance between them, and the angle relationship matrix of the inference network is obtained and distance relationship matrix The fusion features S of a batch in the teacher network T Send it to the classifier to get the teacher network logic output y T , and the teacher network logic output y T and the label l of the input palm vein image T Calculate the cross entropy between them to get the distance coefficient α, and transform the angle relationship matrix of the teacher network into and the angle relationship matrix of the inference network Element-by-element subtraction, the distance relationship matrix of the teacher network And the distance relationship matrix of the inference network Subtract each element to obtain the angle difference matrix and the distance difference matrix. Then add each column of the angle difference matrix and the distance difference matrix to obtain the angle difference vector V. A and the distance difference vector V D ; The angle difference vector V A and the distance difference vector V D Multiply them by the distance coefficient α to get the weighted angle difference vector V α-A and weighted distance difference vector V α-D , and use L2 loss to calculate the weighted difference vector distribution difference to obtain the cross-sample structure distillation loss L based on classification boundary guidance SKD , as follows: Step 5.1: Calculate the fusion feature S in a batch of the teacher network T The angle and distance between them correspond to the angle relationship matrix of the teacher network and distance relationship matrix Calculate the fusion feature S in a batch of the inference network S The angle and distance between them correspond to the angle relationship matrix of the inference network and distance relationship matrix V1(n,m)=||nm||2 (9), Where Γ represents the transpose operation of the matrix, n and m both represent single elements in the sample feature vector, and They correspond to the feature vectors of the xth and yth samples in a batch of the teacher network, and They correspond to the feature vectors of the xth and yth samples in a batch of the inference network respectively; V1 represents the distance difference coefficient, and V2 represents the angle difference coefficient; Step 5.2: The fusion feature S of a batch in the teacher network T Send it to the classifier to get the teacher network logic output y T , and the teacher network logic output y T and the label l of the input palm vein image T Calculate the cross entropy between to get the distance coefficient α: a(x th )=δ(-l T (x th )*log(y T )) (15), Among them, δ(.) represents the softmax function, l T (x th ) represents the label corresponding to the xth sample; Step 5.3: Transform the teacher network's angle relationship matrix and the angle relationship matrix of the inference network Element-by-element subtraction, the distance relationship matrix of the teacher network And the distance relationship matrix of the inference network Subtract each element to obtain the angle difference matrix and the distance difference matrix. Then add each column of the angle difference matrix and the distance difference matrix to obtain the angle difference vector V. A and the distance difference vector V D , the angle difference matrix V A and the distance difference matrix V D Multiply them with the distance coefficient α respectively to get the weighted angle difference matrix V α-A and the weighted distance difference matrix V α-D , and use L2 loss to calculate the weighted difference matrix distribution difference to obtain the cross-sample structure distillation loss L based on classification boundary guidance SKD : Where b represents the number of feature samples in a batch, and i represents the number of rows in the feature matrix.

6. The palm print and palm vein recognition method in missing mode based on multi-level knowledge distillation according to claim 5 is characterized in that: In step 6, the fusion feature S of the inference network is S After being sent to the classifier, the inference network logic output y is obtained S ; At the same time, the teacher network logic output y T Decoupling to target class output and non-target class output Output the inference network logic y S Decoupling to target class output and non-target class output The KL loss is used to calculate the difference between the target class output and the non-target class output in the teacher network and the inference network to obtain the decoupled alignment distillation loss L DAD Finally, the cross entropy loss is used to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, thereby obtaining a trained inference network, as follows: The fusion feature S of the teacher network T , the fusion feature S of the inference network S Input into each classifier to get the corresponding logical output y T 、y S ; At the same time, the teacher network logic output y T Decoupling to target class output and non-target class output Output the inference network logic y S Decoupling to target class output and non-target class output The KL loss is used to calculate the difference between the target class output and the non-target class output in the teacher network and the inference network to obtain the decoupled alignment distillation loss L DAD Finally, the cross entropy loss is used to ensure that the maximum probability output of the classifier is consistent with the probability of the actual label, combined with the cross-modal feature distribution correlation distillation loss L CRCD , cross-sample structure distillation loss L SKD and decoupled alignment distillation loss L DAD Train together to obtain a trained inference network: y S =Classifier S (S S ) (19), y T =Classifier T (S T ) (20), L MKD =L ClS +λ1L CRCD +λ2L SKD +L DAD (24), Among them, De(.) represents the decoupling operation, Classifier S and Classifier T They represent the classifiers of the inference network and the teacher network respectively, δ is the weighted hyperparameter of the target class logic knowledge distillation, β is the weighted hyperparameter of the non-target class logic knowledge distillation, KL(.) represents the KL divergence calculation function, and λ1 represents the cross-modal feature distribution correlation distillation loss L CRCD The weighted hyperparameter of λ2 represents the cross-sample structure distillation loss L based on the classification boundary guidance. SKD The weighted hyperparameter, L ClS represents the cross entropy loss, L MKD Represents multi-level distillation loss.

7. The palm print and palm vein recognition method based on multi-level knowledge distillation in missing mode according to claim 6 is characterized in that: In step 7, the test set is used to evaluate the accuracy of the trained inference network as follows: Step 7.1: During testing, if both the palm print image and the palm vein image are present, the features of each modality are directly fused for recognition. If either the palm print image or the palm vein image is missing, the missing image input is filled with zero elements. Step 7.2: During the recognition process, use the correct recognition rate to infer the performance of the network; use N True To represent the cumulative number of samples predicted correctly, N Flace To represent the cumulative number of samples with incorrect predictions, the correct recognition rate CRR is defined as:

Images