A palm print and palm vein fusion recognition method based on feature decoupling in missing modality

Abstract

The present invention discloses a palm print and palm vein fusion recognition method under missing modalities based on feature decoupling, which aims to solve the problems existing in the current multimodal fusion methods under missing modalities: (1) using a generative method, a generative model needs to be designed for each missing modality, resulting in a large overall model that is not suitable for practical applications; (2) most modality missing methods specify the missing modality, which limits flexibility. In view of the above problems, the present invention uses a simple network model, designs a palm print cross-modal generator or a palm vein cross-modal generator that decouples shared features and specific features using contrastive learning loss and inter-modality identity consistency loss, and constructs a palm print cross-modal generator or a palm vein cross-modal generator at the feature layer, and uses known modal features to complete the missing modal features. Different from the existing methods, the multimodal fusion method proposed in the present invention has excellent robustness, and can still obtain accurate recognition results when any modality is missing in the test set, meeting the needs of diverse application scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of biometric feature recognition, and in particular relates to a palm print and palm vein fusion recognition method in a missing mode based on feature decoupling. Background Art

[0002] In terms of privacy protection and information security, multimodal biometric recognition systems provide safer and more accurate identity authentication solutions than single-modal biometric recognition systems. Among them, biometric recognition technology based on palm print and palm vein fusion has received more and more attention due to its rich identification characteristics, non-contact and hygienic collection method. However, in reality, due to the failure of some sensors during collection or the loss of some data during transmission, the availability of all collected data cannot be ensured. In this case, the existing multimodal fusion recognition method will have serious performance degradation. Therefore, it is of great practical significance to design a palm print and palm vein fusion recognition method for uncertain modality loss.

[0003] At present, deep learning has achieved excellent performance in the field of multimodal biometric recognition. For example, researchers such as Zhong proposed a multimodal recognition algorithm based on palm prints and dorsal hand veins, namely Palmprint Hybridized with Dorsal handvine (PHD). PHD uses deep hashing network (DHN) and biometric graph matching (BGM) to process palm print and dorsal hand vein data respectively (DX Zhong VGG, K. Shao VGG and X. F. Du VGG "A Hand-Based Multi-Biometrics via Deep Hashing Network and Biometric Graph Matching VGG" Ieee Transactions on Information Forensics and Security VGG vol. 14 VGG no. 12 VGG pp. 3140-3150 VGG Dec. 2019.). In addition, researchers such as Yi proposed a multimodal fusion recognition method that includes fingerprints and finger veins. They used an asymmetric backbone network (FAB-Net) to extract the intra-modal features of fingerprints and finger veins. Subsequently, they introduced a novel attention-based encoder fusion network (AEF-Net) for dimensionality reduction-based fusion of multimodal multi-level features to alleviate the performance degradation caused by information redundancy. AEF-Net also introduced channel attention to further improve performance. (Huang VGG Yiwei VGGet al. "Multimodal Finger Recognition Based on Asymmetric Networks With Fused Similarity". IEEE Access VGG vol. 11 VGG 2023 VGG pp. 17497-17509.). However, at present, multimodal biometric fusion recognition technology often does not consider the damage or loss of multimodal features in the process of studying modal fusion, which affects the robustness and versatility of the model. Therefore, how to make full use of existing information during training and maintain excellent recognition performance when some modal data has poor quality or even modality is lost is an important issue that needs to be solved urgently. Summary of the invention

[0004] The present invention proposes a palm print and palm vein fusion recognition method under missing modalities based on feature decoupling, which aims to solve the problems existing in the current multimodal fusion methods under missing modalities: (1) using generative methods, a generative model needs to be designed for each missing modality, resulting in a large overall model that is not suitable for practical applications; (2) most missing modal methods specify the missing modality, which limits flexibility.

[0005] The technical solution to realize the present invention is: a palm print and palm vein fusion recognition method in missing mode based on feature decoupling:

[0006] Step 1: collect palm prints and palm vein images of n individuals, collect 2a palm print images and 2a palm vein images for each individual, and establish a palm print and palm vein image database; the palm print and palm vein image database is divided into a training set and a test set at a ratio of 1:1, and assume that some palm print images and palm vein images in the test set are missing, and the missing rate is β, then go to step 2.

[0007] Step 2: Use VGG16 combined with the spatiotemporal attention mechanism to construct palmprint feature extraction networks F p and palm vein feature extraction network F v , to extract the palmprint feature f p and palmar vein characteristics v , go to step 3.

[0008] Step 3: Use the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palmprint features f p , get the unique features of the palmprint modality f s p and the shared features of the palmprint modality f c p ; At the same time, the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palm vein features v , and obtain the unique features of the palm vein mode f s v and the shared features of the palm vein modality f c v , go to step 4.

[0009] Step 4: Use mean square error loss and designed intra-modal identity consistency loss Combined with the training set supervised training palmprint cross-modal generator G p and palm vein cross-modal generator G v , where G p Using the unique features of palm vein modality s v Generate unique features of palm printss ' p , G v Using the unique features of palmprint modality s p Generate unique features of palm veins s ' v , go to step 5.

[0010] Step 5: Use G p The unique features of the generated palm prints s ' p Combined with the shared features of palm veins c v , and obtain the palm print feature f′ completed by the palm vein feature p ; while using G v The unique features of the generated palm veins s ' v Combined with the shared features of palmprints c p , and obtain the palm vein feature f′ completed by the palm print feature v , go to step 6.

[0011] Step 6: Change the palm print feature f p and palmar vein characteristics v The fusion feature f of the palm print feature and the palm vein feature is obtained by fusion; the palm print feature f p and the palm vein feature f′ completed by the palm print feature v The fusion feature f of the palm print feature and the completed palm vein feature is obtained by fusion v ′; at the same time, the palm print feature f′ completed by the palm vein feature p and palmar vein characteristics v The fusion feature f of the completed palm print feature and palm vein feature is obtained by fusion p ′, train the palm print and palm vein fusion recognition network through cross entropy loss to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, and then obtain the trained palm print and palm vein fusion recognition network model F, and go to step 7.

[0012] Step 7: Use the test set to evaluate the accuracy of the trained palm print and palm vein fusion recognition network model F.

[0013] Compared with the prior art, the present invention has the following significant advantages:

[0014] (1) The present invention constructs a multimodal hand biometric feature extraction model, and achieves full decoupling of each hand modality by designing contrastive learning loss and inter-modality identity consistency loss to obtain corresponding shared features and specific features.

[0015] (2) The present invention proposes a palmprint cross-modal generation method at the feature layer. Through the specific features of the known modality, the mean square error loss and the intra-modal identity consistency loss designed by the present invention are used to generate the missing modality specific features. Combined with the shared features of the known modality, the missing modality features are completed in a unique way, which is simple and efficient.

[0016] (3) The present invention not only exhibits excellent recognition effect when the modality is complete and intact, but also can obtain accurate recognition results when some modalities are damaged or even uncertain modalities are missing, thus meeting the needs of different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a flowchart of a palm print and palm vein fusion recognition method in missing mode based on feature decoupling.

[0018] Figure 2 This is a model diagram of a palm print and palm vein fusion recognition method in missing mode based on feature decoupling of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The following will further introduce the specific implementation method, as well as the technical difficulties and inventive points of this invention in combination with this design example.

[0021] The present invention discloses a palm print and palm vein fusion recognition method in missing modality based on feature decoupling, which constructs a feature extraction network by combining VGG16 and spatiotemporal attention mechanism to extract deeper discriminative features of dual-modal palm print and palm vein images. The unique features of each modality and the shared features of all modalities are decoupled from the input of palm print image and palm vein image by contrast loss and inter-modality identity consistency loss. Then, when the input modality is not missing, the cross-palm print and cross-modal generator G is supervisedly trained by mean square error loss and intra-modality identity consistency loss. p Or palm vein cross-modal generator G v , so that the generated unique features are close to the real unique features, and the cross entropy loss is used to train the multimodal fusion recognition to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label. When the palm print modality or palm vein modality is missing, the unique features of the known modality are sent to the palm print cross-modal generator G p Or palm vein cross-modal generator G vThe unique features of the missing modality are obtained, combined with the common features of the known modalities, to obtain the complete missing modal features, which are then input into the classification network for prediction after modal fusion. Even when the quality of some modalities is poor or even uncertain that the modality is missing, this method can still obtain accurate recognition results to meet the needs of different application scenarios. The specific steps are as follows:

[0022] Step 1: collect palm prints and palm vein images of n individuals, collect 2a palm print images and 2a palm vein images for each individual, and establish a palm print and palm vein image database; the palm print and palm vein image database is divided into a training set and a test set at a ratio of 1:1, and assume that some palm print images and palm vein images in the test set are missing, and the missing rate is 0<β<1, then go to step 2.

[0023] Step 2: Use VGG16 combined with the spatiotemporal attention mechanism to construct palmprint feature extraction networks F p and palm vein feature extraction network F v , to extract the palmprint feature f p and palmar vein characteristics v .

[0024] Remove the last layer classifier of the convolutional neural network VGG16 to obtain a new convolutional neural network; connect the new convolutional neural network with the spatiotemporal attention mechanism CSAFM to construct the palmprint feature extraction network F p and palm vein feature extraction network F v , corresponding to the extracted palmprint features f p and palmar vein characteristics v :

[0025] f p =F P (x p ) (1)

[0026] f v =F v (x v ) (2)

[0027] Among them, x p represents the palm print image, x v Represents a palm vein image.

[0028] Go to step 3.

[0029] Step 3: In order to fully decouple the shared features and unique features of different modalities, this paper proposes the inter-modal contrast loss function and the inter-modal identity consistency loss function for the first time, realizes the adaptive separation of the shared features and unique features of different modalities, enhances the semantic information of the modal features after decoupling, and improves the decoupling effect of the shared features and unique features of different modalities. Inter Fully decouple palmprint features f p , get the unique features of the palmprint modality f s p and the shared features of the palmprint modality f c p ; At the same time, the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palm vein features v , and obtain the unique features of the palm vein mode f s v and the shared features of the palm vein modality f c v .

[0030] Step 3.1: Through contrastive learning loss, the shared features f of the palmprint modality are c p is the anchor point, and the shared feature f of the palm vein modality c v is a positive sample, the unique feature f of the palmprint modality s p is a negative sample, as shown in formula (3), which is close to the shared feature f of the palmprint modality c p and the shared features of the palm vein modality f c v distance between them, and pull away the shared features f of the palmprint modalities c p and the unique features of the palm print modality f s p The distance between the palm print features f p ;

[0031] Through contrastive learning loss, the shared features f of palm vein modality are used c v is the anchor point, and the shared feature f of the palmprint modality c p is a positive sample, the unique feature of the palm vein modality f s v is a negative sample, as shown in formula (4), which brings the shared feature f of the palm vein modality closer c v and the shared features of the palmprint modality f c pdistance between them, and pull away the shared feature f of the palm vein modality c v and the unique features of the palmar vein modality s v The distance between the palm veins is decoupled from the palm vein feature f v :

[0032]

[0033] in, represents the palmprint modality contrastive learning loss; represents the palm vein modality contrastive learning loss;

[0034] The meaning of the subscript + is: when the value in [...] is greater than 0, the value is taken as the loss, and when it is less than 0, the loss is 0; m is the number of input images; α is a hyperparameter; ||·||2 represents the L2 norm.

[0035] Step 3.2: The shared features f of the palmprint modality c p Unique features of palmar vein modality s v After combining, it is sent to the pre-trained palm vein recognition model F v ′, and at the same time, the shared feature f of the palm vein modality c v The unique features of palm print modality s p After combining, it is sent to the pre-trained palmprint recognition model F p ′, where F v ′ and F p ′ contains UpSample(·), DownSample(·) and θ(·), and the identity consistency loss L between modalities is used. Inter Zoom in F v ′ and F p The core idea of ​​the recognition results of the two models is shown in formulas (5) and (6). While exchanging the shared features of different modalities, the consistency of the identity information of the overall features of different modalities is ensured, thereby enhancing the alignment effect of the modal shared features:

[0036] f1 p =f c v +f s p (5)

[0037] f1 v =f c p +f s v (6)

[0038] x′ v =UpSample v (f1 v ) (7)

[0039] x′ p =UpSample p (f1 p ) (8)

[0040] f1′ v =DownSample v (x′ v ) (9)

[0041] f1′ p =DownSample p (x′ p ) (10)

[0042]

[0043] Among them, UpSample j (·) is used to restore features to images, DownSample j (·) is used to convert images to features, whose structure is similar to F p and F v Same, j = v, p; θ v (·) and θ p (·) represents the prediction function, which is used to transform the feature vector into the same prediction space; ||·||1 represents the L1 norm; f1 p 、f1 v , x′ p , x′ v , f1′ p and f1′ v These are intermediate variables used to simplify calculations.

[0044] Go to step 4.

[0045] Step 4: To address the problem of missing any modality in the test set, the present invention innovatively proposes a method for reconstructing any missing modal features. An intra-modal identity consistency loss function is designed to reconstruct discriminative missing modal features and enhance the robustness of the model to modal loss. The mean square error loss and the designed intra-modal identity consistency loss L are used to reconstruct the discriminative missing modal features. Intra , combined with the training set supervised training palmprint cross-modal generator G p and palm vein cross-modal generator G v , where G p Using the unique features of palm vein modality s vGenerate unique features of palm prints s ' p , G v Using the unique features of palmprint modality s p Generate unique features of palm veins s ' v .

[0046] Step 4.1: The unique features of the palmprint modality f s p Feed it into the palm vein cross-modal generator G v Unique features of palmar vein formation s ' v , and the unique features of the palmar vein modality f s v , mean square error loss through palm vein Supervised training G v , which generates the unique characteristics of palmar veins s ' v Closer to the unique features of real palm veins s v ; The unique features of the palm vein modality f s v Feed it into the palmprint cross-modal generator G p The unique features of palm prints generated in s ' p , and the unique features of the palmprint modality f s p , through the mean square error loss of the palmprint Supervised training G p , so that the unique features of the palm print are generated s ' p The unique features of the palm print mode that are closer to the real one s p , ensuring the reconstruction effect between palm print and palm vein modal features:

[0047] f s ' v =G V (f s p ) (16)

[0048] f s ' p =G p (f s v ) (17)

[0049]

[0050] Among them, m is the number of input images.

[0051] Step 4.2: The shared features f of the palmprint modality c p The unique features of palm print modality s p Combined with the palmprint modality shared feature f c p The unique features of the generated palm prints s ' p Combined, they are sent to the pre-trained palmprint recognition model F p ′, through the intra-modality identity consistency loss Zoom in on the two palmprint recognition models F p ′’s recognition result improves the discriminability of palmprint modality reconstruction features:

[0052]

[0053] x″ p =UpSample(f″ p ) (twenty two)

[0054] f1″ p =DownSample(x″ p ) (twenty three)

[0055]

[0056] Among them, UpSample(·) is used to restore features to images, and DownSample(·) is used to convert images to features; θ p (·) represents the prediction function, f p ″、x″ p 、f1″ p , and These are intermediate variables used to simplify calculations.

[0057] Step 4.3: The shared features f of the palm vein modality c v Unique features of palmar vein modality s v Combined with the shared features of the palm vein modality f c v The unique characteristics of the palm veins generated s ' v Combined, they are sent to the pre-trained palm vein recognition model F v ′, through the intra-modality identity consistency loss Zoom in on the two palm vein recognition models F v′’s recognition result improves the discriminability of palm vein modality reconstruction features:

[0058]

[0059] x v ″=UpSample(f″ v ) (29)

[0060] f1″ v =DownSample(x″ v ) (30)

[0061]

[0062]

[0063] Among them, f v ″、x v ″、f1″ v , and These are intermediate variables used to simplify calculations.

[0064] Go to step 5.

[0065] Step 5: Use G p The unique features of the generated palm prints s ' p Combining shared features of palm veins Get the palm print feature f′ completed by the palm vein feature p ; while using G v The unique features of the generated palm veins s ' v Combined with the shared features of palmprints c p , and obtain the palm vein feature f′ completed by the palm print feature v .

[0066] Step 5.1: Using the unique features of palm vein modality s v , through the palmprint cross-modal generator G p , generating the unique features of the palm print f s ' p , plus the shared features of the palm vein modality Get the palm print feature f′ completed by the palm vein feature p :

[0067] f s ' p =G p (f s v ) (35)

[0068]

[0069] Step 5.2: Using the unique features of palmprint modality f s p , through the palm vein cross-modal generator G v Generate unique features of palm vein modality f s ' v , plus the shared features of the palmprint modality f c p , and obtain the palm vein feature f′ completed by the palm print feature v :

[0070] f s ' v =G V (f s p ) (37)

[0071]

[0072] Go to step 6.

[0073] Step 6: In the existing missing modal feature reconstruction model, there is a strong correlation between multimodal data. However, there is little correlation between palm print and palm vein hand multimodal biometric features, and they only have the same identity information. Therefore, when developing a hand multimodal biometric feature recognition method under arbitrary modality loss from the perspective of feature-level missing modality reconstruction, the problem of difficulty in reconstructing the missing modality in the feature space will be faced. In order to ensure the effectiveness of the reconstructed missing modal information, the palm print feature f p and palmar vein characteristics v The fusion feature f of the palm print feature and the palm vein feature is obtained by fusion; the palm print feature f p and the palm vein feature f′ completed by the palm print feature v The fusion feature f of the palm print feature and the completed palm vein feature is obtained by fusion v ′; at the same time, the palm print feature f′ completed by the palm vein feature p and palmar vein characteristics v The fusion feature f of the completed palm print feature and palm vein feature is obtained by fusion p ′, the palm print and palm vein fusion recognition network is trained through cross entropy loss to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, so that the fusion feature f of the palm print feature and the complemented palm vein feature v ′ and the fusion feature f of the completed palm print feature and palm vein feature p ′ is closer to the real, fusion feature f of palm print feature and palm vein feature, and then the trained palm print and palm vein fusion recognition network model F is obtained.

[0074] The fusion feature f of palm print feature and palm vein feature, the fusion feature f of palm print feature and complemented palm vein feature v ′ and the fusion feature f of the completed palm print feature and palm vein feature p ′, and then sent to the fully connected layer, and then through the SoftMax layer to obtain the predicted probability distribution. The cross entropy loss L makes the predicted category of the sample equal to the true category of the sample:

[0075] f=f p +f v (39)

[0076] f v ′=f p +f′ v (40)

[0077] f p ′=f′ p +f v (41)

[0078] y i =SoftMax(FC(f)) (42)

[0079]

[0080] L=L1+L1 v +L1 p (48)

[0081] Where i is the serial number of the batch sample; B is the batch size; y i ′ indicates that the i-th sample is the true category; f indicates the modality fusion feature vector; y i Represents the predicted probability distribution; SoftMax(FC(·))

[0082] Indicates that it will first pass through the fully connected layer and then the SoftMax layer; f v ′、f p '、 L1, L1 v

[0083] and L1 p These are intermediate variables used to simplify calculations.

[0084] Go to step 7.

[0085] Step 7: Use the test set to evaluate the accuracy of the trained palm print and palm vein fusion recognition network model F.

[0086] Step 7.1: During the test, if both the palm print image and the palm vein image are present, the features of each modality are directly fused for recognition; if the palm print image or the palm vein image is missing, the known modality-specific features are used to generate the palm print cross-modality generator G. p Or palm vein cross-modal generator G v Generate missing modality-specific features, combine the shared features of known modalities, complete the missing modality features, and then fuse the features of each modality for recognition:

[0087] For palmar vein modality loss:

[0088]

[0089] in, The common feature encoder representing the palmprint, represents the unique feature encoder of palmprint; connect(·) represents the modality fusion network;

[0090] For missing palmprint modality:

[0091]

[0092] in, The common feature encoder representing the palm veins, A unique feature encoder representing palm veins;

[0093] For both palm vein modality and palm vein modality, there is no missing:

[0094] y i =SoftMax{FC[connect(F v (x v )+F p (x p ))]} (51)

[0095] Step 7.2: During the recognition process, the correct recognition rate is used to evaluate the performance of the palm print and palm vein fusion recognition network model F; True To represent the cumulative number of samples predicted correctly, N Flace To represent the number of samples with cumulative prediction errors, the correct recognition rate CRR is defined as:

[0096]

[0097] Example 1

[0098] The palm print and palm vein fusion recognition method in missing mode based on shared-specific feature decoupling of the present invention comprises the following steps:

[0099] Step 1. Create a self-made palm print and palm vein database named CUMT, which contains palm images of 290 volunteers. Each volunteer provided 10 images, representing two different biometric modalities. Therefore, the entire database contains 290 different categories and a total of 5,800 images. These images were collected in two specific time periods, and the time interval between the two collections exceeded 10.

[0100] In each acquisition, we obtained 5 palm print images and 5 palm vein image samples from each volunteer.

[0101] The acquired original image is then calibrated with coordinate points and cropped to extract the image region of interest.

[0102] In the experiment, we divided the dataset into a training set and a test set in a 1:1 ratio. The training set contains 2900 images, and the test set also contains 2900 images, in which some palm print images and palm vein images are missing.

[0103] Step 2: Use VGG16 combined with the spatiotemporal attention mechanism to construct palmprint feature extraction networks F p and palm vein feature extraction network F v , to extract the palmprint feature f p and palmar vein characteristics v .

[0104] Remove the last layer classifier of the convolutional neural network VGG16 to obtain a new convolutional neural network; connect the new convolutional neural network with the spatiotemporal attention mechanism CSAFM to construct the palmprint feature extraction network F p and palm vein feature extraction network F v , corresponding to the extracted palmprint features f p and palmar vein characteristics v :

[0105] f p =F P (x p ) (1)

[0106] f v =F v (x v ) (2)

[0107] Among them, x p represents the palm print image, x v Represents a palm vein image.

[0108] Step 3: Use the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palmprint features fp , get the unique features of the palmprint modality f s p and the shared features of the palmprint modality f c p ; At the same time, the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palm vein features v , and obtain the unique features of the palm vein mode f s v Shared features with palm vein modality

[0109] Step 3.1: Through contrastive learning loss, the shared features f of the palmprint modality are c p is the anchor point, the shared features of the palm vein modality is a positive sample, the unique feature f of the palmprint modality s p is a negative sample, decoupling the palmprint feature f p ; Through contrastive learning loss, the shared features of palm vein modality is the anchor point, and the shared feature f of the palmprint modality c p is a positive sample, the unique feature of the palm vein modality f s v is a negative sample, decoupling the palm vein feature f v :

[0110]

[0111] in, represents the palmprint modality contrastive learning loss; represents the palm vein modality contrastive learning loss;

[0112] The meaning of the subscript + is: when the value in [...] is greater than 0, the value is taken as loss, and when it is less than 0, the loss is 0; m

[0113] is the number of input images; α is a hyperparameter; ||·||2 represents the L2 norm;

[0114] Step 3.2: The shared features f of the palmprint modality c p Unique features of palmar vein modality s v After combining, it is sent to the pre-trained palm vein recognition model F v ′, and the shared features of the palm vein modality The unique features of palm print modality s p After combining, it is sent to the pre-trained palmprint recognition model F p′, where F v ′ and F p ′ contains UpSample(·), DownSample(·) and θ(·), and the identity consistency loss L between modalities is used. Inter Zoom in F v ′ and F p ′Recognition results of the two models:

[0115]

[0116] x v ′=UpSample v (f1 v ) (7)

[0117] x′ p =UpSample p (f1 p ) (8)

[0118] f1′ v =DownSample v (x′ v ) (9)

[0119] f1′ p =DownSample p (x′ p ) (10)

[0120]

[0121] Among them, UpSample j (·) is used to restore features to images, DownSample j (·) is used to convert images to features, whose structure is similar to F p and F v Same, j = v, p; θ v (·) and θ p (·) represents the prediction function, which is used to transform the feature vector into the same prediction space; ||·||1 represents the L1 norm; f1 p 、f1 v , x′ p , x′ v , f1′ p and f1′ v These are intermediate variables used to simplify calculations.

[0122] Step 4: Use mean square error loss and designed intra-modal identity consistency loss L Intra , combined with the training set supervised training palmprint cross-modal generator G pand palm vein cross-modal generator G v , where G p Using the unique features of palm vein modality s v Generate unique features of palm prints s ' p , G v Using the unique features of palmprint modality s p Generate unique features of palm veins s ' v .

[0123] Step 4.1: The unique features of the palmprint modality f s p Feed it into the palm vein cross-modal generator G v Unique features of palmar vein formation s ' v , and the unique features of the palmar vein modality f s v , mean square error loss through palm vein Supervised training G v ; The unique features of the palm vein modality f s v Feed it into the palmprint cross-modal generator G p

[0124] The unique features of palm prints generated in s ' p , and the unique features of the palmprint modality f s p , through the mean square error loss of the palmprint Supervised training G p :

[0125] f s ' v =G V (f s p ) (16)

[0126] f s ' p =G p (f s v ) (17)

[0127]

[0128]

[0129] Among them, m is the number of input images.

[0130] Step 4.2: The shared features f of the palmprint modalityc p The unique features of palm print modality s p Combined with the palmprint modality shared feature f c p The unique features of the generated palm prints s ' p Combined, they are sent to the pre-trained palmprint recognition model F p ′, through the intra-modality identity consistency loss Zoom in on the two palmprint recognition models F p 'Recognition results:

[0131]

[0132] x″ p =UpSample(f″ p ) (twenty two)

[0133] f1″ p =DownSample(x″ p ) (twenty three)

[0134]

[0135] Among them, UpSample(·) is used to restore features to images, and DownSample(·) is used to convert images to features; θ p (·) represents the prediction function, f p ″、x″ p 、f1″ p , and These are intermediate variables used to simplify calculations.

[0136] Step 4.3: Shared features of palm vein modality Unique features of palmar vein modality s v Combined with the shared features of palm vein modalities The unique characteristics of the palm veins generated s ' v Combined, they are sent to the pre-trained palm vein recognition model F v ′, through the intra-modality identity consistency loss Zoom in on the two palm vein recognition models F v 'Recognition results:

[0137]

[0138] x v ″=UpSample(f″v ) (29)

[0139] f1″ v =DownSample(x″ v ) (30)

[0140]

[0141] Among them, f v ″、x v ″、f1″ v , and These are intermediate variables used to simplify calculations.

[0142] Step 5: Use G p The unique features of the generated palm prints s ' p Combining shared features of palm veins Get the palm print feature f′ completed by the palm vein feature p ; while using G v The unique features of the generated palm veins s ' v Combined with the shared features of palm prints c p , and obtain the palm vein feature f′ completed by the palm print feature v .

[0143] Step 5.1: Using the unique features of palm vein modality s v , through the palmprint cross-modal generator G p , generating the unique features of the palm print f s ' p , plus the shared features of the palm vein modality Get the palm vein feature completed palm print feature f′ p :

[0144] f s ' p =G p (f s v ) (35)

[0145]

[0146] Step 5.2: Using the unique features of palmprint modality f s p , through the palm vein cross-modal generator G v Generate unique features of palm vein modality f s ' v , plus the shared features of the palmprint modality fc p , and obtain the palm vein feature f′ completed by the palm print feature v :

[0147] f s ' v =G V (f s p ) (37)

[0148]

[0149] Step 6: Change the palm print feature f p and palmar vein characteristics v The fusion feature f of the palm print feature and the palm vein feature is obtained by fusion; the palm print feature f p and the palm vein feature f′ completed by the palm print feature v The fusion feature f of the palm print feature and the completed palm vein feature is obtained by fusion v ′; at the same time, the palm print feature f′ completed by the palm vein feature p and palmar vein characteristics v The fusion feature f of the completed palm print feature and palm vein feature is obtained by fusion p ′, the palm print and palm vein fusion recognition network is trained through cross entropy loss to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, and then the trained palm print and palm vein fusion recognition network model F is obtained.

[0150] The fusion feature f of palm print feature and palm vein feature, the fusion feature f of palm print feature and complemented palm vein feature v ′ and the fusion feature f of the completed palm print feature and palm vein feature p ′, and then sent to the fully connected layer, and then through the SoftMax layer to obtain the predicted probability distribution. The cross entropy loss L makes the predicted category of the sample equal to the true category of the sample:

[0151] f=f p +f v (39)

[0152] f v ′=f p +f′ v (40)

[0153] f p ′=f′ p +f v (41)

[0154] y i =SoftMax(FC(f)) (42)

[0155]

[0156] L=L1+L1 v +L1 p (48)

[0157] Where i is the serial number of the batch sample; B is the batch size; y i ′ indicates that the i-th sample is the true category; f indicates the modality fusion feature vector; y i Represents the predicted probability distribution; SoftMax(FC(·))

[0158] Indicates that it will first pass through the fully connected layer and then the SoftMax layer; f v ′、f p '、 L1, L1 v

[0159] and L1 p These are intermediate variables used to simplify calculations.

[0160] Step 7: Use the test set to evaluate the accuracy of the trained palm print and palm vein fusion recognition network model F.

[0161] Step 7.1: During the test, if both the palm print image and the palm vein image are present, the features of each modality are directly fused for recognition; if the palm print image or the palm vein image is missing, the known modality-specific features are used to generate the palm print cross-modality generator G. p Or palm vein cross-modal generator G v Generate missing modality-specific features, combine the shared features of known modalities, complete the missing modality features, and then fuse the features of each modality for recognition:

[0162] For palmar vein modality loss:

[0163]

[0164] in, The common feature encoder representing the palmprint, represents the unique feature encoder of palmprint; connect(·) represents the modality fusion network;

[0165] For missing palmprint modality:

[0166]

[0167] in, The common feature encoder representing the palm veins, A unique feature encoder representing palm veins;

[0168] For both palm vein modality and palm vein modality, there is no missing:

[0169] y i =SoftMax{FC[connect(F v (x v )+F p (x p ))]} (51)

[0170] Step 7.2: During the recognition process, the correct recognition rate is used to evaluate the performance of the palm print and palm vein fusion recognition network model F; True To represent the cumulative number of samples predicted correctly, N Flace To represent the number of samples with cumulative prediction errors, the correct recognition rate CRR is defined as:

[0171]

[0172] The method of the present invention is to conduct relevant experiments on a network framework built with Python programming language and pytorch framework on an Nvidia 1050Ti GPU host. We set the batch-size to 8 and use the stochastic gradient descent (SGD) algorithm to optimize the total loss. In order to speed up the network training, we chose an optimization strategy with a momentum of 0.9 and a weight decay of 1e -5 Palmprint feature extraction network F p and palm vein feature extraction network F v (Based on VGG16) Pre-trained on ImageNet with an initial learning rate of 1e -3 In the experiment, we used the Dropout layer to prevent the network from overfitting, and set the Dropout value to 0.5. In this experiment, we considered the two modalities from the same palm as the same category, without any form of expansion of the training data, in order to maximize the test of the true situation of palm prints and palm veins.

[0173] This experiment is divided into palm print missing or palm vein missing in the test set. The experimental results when the missing rate β (%) is 0, 20, 50, 80, and 100 are shown in Tables 1 and 2, where β = 0% means that the modality is not missing.

[0174] Table 1 Recognition results of missing palm prints

[0175]

[0176] Table 2 Identification results of missing palmar veins

[0177]

[0178] According to the experimental results in Table 1 and Table 2, the method proposed in the present invention can achieve excellent recognition effect regardless of palm print missing or palm vein missing, and the missing rate has little effect on it. Even when a certain mode is 100% missing, it can still achieve good results, and the model has good robustness.

Claims

1. A palm print and palm vein fusion recognition method based on feature decoupling in missing mode, characterized in that: The following steps are involved: Step 1: collect palm prints and palm vein images of n individuals, collect 2a palm print images and 2a palm vein images for each individual, and establish a palm print and palm vein image database; the palm print and palm vein image database is divided into a training set and a test set at a ratio of 1:1, and assume that some palm print images and palm vein images in the test set are missing, and the missing rate is β, and then proceed to step 2; Step 2: Use VGG16 combined with the spatiotemporal attention mechanism to construct palmprint feature extraction networks F p and palm vein feature extraction network F v , to extract the palmprint feature f p and palmar vein characteristics v , go to step 3; Step 3: Use the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palmprint features f p , get the unique features of the palmprint modality f s p and the shared features of the palmprint modality f c p ; At the same time, the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palm vein features v , and obtain the unique features of the palm vein mode f s v and the shared features of the palm vein modality f c v , go to step 4; Step 4: Use mean square error loss and designed intra-modal identity consistency loss L Intra , combined with the training set supervised training palmprint cross-modal generator G p and palm vein cross-modal generator G v , where G p Using the unique features of palm vein modality s v Generate unique features of palm prints s ' p , G v Using the unique features of palmprint modality s p Generate unique features of palm veins s ' v , go to step 5; Step 5: Use G p The unique features of the generated palm prints s ' p Combined with the shared features of palm veins c v , and obtain the palm print feature f′ completed by the palm vein feature p ; while using G v The unique features of the generated palm veins s ' v Combined with the shared features of palmprints c p , and obtain the palm vein feature f′ completed by the palm print feature v , go to step 6; Step 6: Change the palm print feature f p and palmar vein characteristics v The fusion feature f of the palm print feature and the palm vein feature is obtained by fusion; the palm print feature f p and the palm vein feature f′ completed by the palm print feature v The fusion feature f′ of the palm print feature and the completed palm vein feature is obtained by fusion v ; At the same time, the palm print feature f′ completed by the palm vein feature p and palmar vein characteristics v The fusion feature f′ of the completed palm print feature and palm vein feature is obtained by fusion p , train the palm print and palm vein fusion recognition network through cross entropy loss to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, and then obtain the trained palm print and palm vein fusion recognition network model F, and go to step 7; Step 7: Use the test set to evaluate the accuracy of the trained palm print and palm vein fusion recognition network model F.

2. According to the method of palm print and palm vein fusion recognition in missing mode based on feature decoupling in claim 1, it is characterized in that: In step 1, 200 <n<500,2<a<15,0<β<1。 3. The palm print and palm vein fusion recognition method based on feature decoupling in missing mode according to claim 2 is characterized in that: In step 2, VGG16 is used in combination with the spatiotemporal attention mechanism to construct the palmprint feature extraction network F p and palm vein feature extraction network F v , to extract the palmprint feature f p and palmar vein characteristics v , as follows: Remove the last layer classifier of the convolutional neural network VGG16 to obtain a new convolutional neural network; connect the new convolutional neural network with the spatiotemporal attention mechanism CSAFM to construct the palmprint feature extraction network F p and palm vein feature extraction network F v , corresponding to the extracted palmprint features f p and palmar vein characteristics v : f p =F P (x p ) (1) f v =F v (x v ) (2) Among them, x p represents the palm print image, x v Represents a palm vein image.

4. According to claim 3, the palm print and palm vein fusion recognition method in missing mode based on feature decoupling is characterized in that: In step 3, the designed contrastive learning loss and inter-modal identity consistency loss L are used Inter Fully decouple palmprint features f p , get the unique features of the palmprint modality f s p and the shared features of the palmprint modality f c p ; At the same time, the designed contrastive learning loss and inter-modal identity consistency loss L Inter Fully decouple palm vein features v , and obtain the unique features of the palm vein mode f s v and the shared features of the palm vein modality f c v , as follows: Step 3.1: Through contrastive learning loss, the shared features f of the palmprint modality are c p is the anchor point, and the shared feature f of the palm vein modality c v is a positive sample, the unique feature f of the palmprint modality s p is a negative sample, decoupling the palmprint feature f p ; Through contrastive learning loss, the shared features f of palm vein modality are c v is the anchor point, and the shared feature f of the palmprint modality c p is a positive sample, the unique feature of the palm vein modality f s v is a negative sample, decoupling the palm vein feature f v : in, represents the palmprint modality contrastive learning loss; represents the palm vein modality contrast learning loss; the subscript + means: when the value in […] is greater than 0, the value is taken as loss, and when it is less than 0, the loss is 0; m is the number of input images; α is a hyperparameter; ||·||2 represents the L2 norm; Step 3.2: The shared features f of the palmprint modality c p Unique features of palmar vein modality s v After combining, it is sent to the pre-trained palm vein recognition model F′ v , and the shared features f of the palm vein modality c v The unique features of palm print modality s p After combining, it is sent to the pre-trained palmprint recognition model F′ p , where F′ v and F′ p Both include UpSample(·), DownSample(·) and θ(·), and use the inter-modal identity consistency loss L Inter Zoom in F′ v and F′ p Recognition results of the two models: x′ v =UpSample v (f1 v ) (7) x′ p =UpSample p (f1 p ) (8) f1′ v =DownSample v (x′ v ) (9) f1′ p =DownSample p (x′ p ) (10) Among them, UpSample j (·) is used to restore features to images, DownSample j (·) is used to convert images to features, whose structure is similar to F p and F v Same, j = v, p; θ v (·) and θ p (·) represents the prediction function, which is used to transform the feature vector into the same prediction space; ||·||1 represents the L1 norm; f1 p 、f1 v , x′ p , x′ v , f1′ p and f1′ v These are intermediate variables used to simplify calculations.

5. The palm print and palm vein fusion recognition method based on feature decoupling in missing mode according to claim 4 is characterized in that: In step 4, the mean square error loss and the designed intra-modal identity consistency loss L are used Intra , combined with the training set supervised training palmprint cross-modal generator G p and palm vein cross-modal generator G v , where G p Using the unique features of palm vein modality s v Generate unique features of palm prints s ' p , G v Using the unique features of palmprint modality s p Generate unique features of palm veins s ' v , as follows: Step 4.1: The unique features of the palmprint modality f s p Feed it into the palm vein cross-modal generator G v Unique features of palmar vein formation s ' v , and the unique features of the palmar vein modality f s v , mean square error loss through palm vein Supervised training G v ; The unique features of the palm vein modality f s v Feed it into the palmprint cross-modal generator G p The unique features of palm prints generated in s ' p , and the unique features of the palmprint modality f s p , through the mean square error loss of the palmprint Supervised training G p : f s ′ v =G V (f s p ) (16) f s ′ p =G p (f s v ) (17) Where m is the number of input images; Step 4.2: The shared features f of the palmprint modality c p The unique features of palm print modality s p Combined with the palmprint modality shared feature f c p The unique features of the generated palm prints s ' p Combined, they are sent to the pre-trained palmprint recognition model F′ p , through the intra-modality identity consistency loss Close the two palmprint recognition models F′ p The recognition result is: f p =f c p +f s p (20) f″ p =f c p +f s ′ p (21) x″ p =UpSample(f″ p ) (22) f1″ p =DownSample(x″ p ) (23) Among them, UpSample(·) is used to restore features to images, and DownSample(·) is used to convert images to features; θ p (·) represents the prediction function, f″ p , x″ p 、f1″ p , and They are all intermediate variables used to simplify calculations; Step 4.3: The shared features f of the palm vein modality c v Unique features of palmar vein modality s v Combined with the shared features of the palm vein modality f c v The unique characteristics of the palm veins generated s ' v Combined, they are sent to the pre-trained palm vein recognition model F′ v , through the intra-modality identity consistency loss Closely compare the two palm vein recognition models F′ v The recognition result is: f v =f c v +f s v (27) f″ v =f c v +f s ′ v (28) x v ″=UpSample(f″ v ) (29) f1″ v =DownSample(x″ v ) (30) Among them, f″ v , x″ v 、f1″ v , and These are intermediate variables used to simplify calculations.

6. The palm print and palm vein fusion recognition method based on feature decoupling in missing mode according to claim 5 is characterized in that: In step 5, use G p The unique features of the generated palm prints s ' p Combined with the shared features of palm veins c v , and obtain the palm print feature f′ completed by the palm vein feature p ; while using G v The unique features of the generated palm veins s ' v Combined with the shared features of palm prints c p , and obtain the palm vein feature f′ completed by the palm print feature v , as follows: Step 5.1: Using the unique features of palm vein modality s v , through the palmprint cross-modal generator G p , generating the unique features of the palm print f s ' p , plus the shared feature f of the palm vein modality c v , and obtain the palm print feature f′ completed by the palm vein feature p : f s ′ p =G p (f s v ) (35) f′ p =f s ′ p +f c v (36) Step 5.2: Using the unique features of palmprint modality f s p , through the palm vein cross-modal generator G v Generate unique features of palm veins s ' v , plus the shared features of the palmprint modality f c p , and obtain the palm vein feature f′ completed by the palm print feature v : f s ′ v =G V (f s p ) (37) f′ v =f s ′ v +f c p (38)。 7. The palm print and palm vein fusion recognition method based on feature decoupling in missing mode according to claim 6 is characterized in that: In step 6, the palm print feature f p and palmar vein characteristics v The fusion feature f of the palm print feature and the palm vein feature is obtained by fusion; the palm print feature f p and the palm vein feature f′ completed by the palm print feature v The fusion feature f′ of the palm print feature and the completed palm vein feature is obtained by fusion v ; At the same time, the palm print feature f′ completed by the palm vein feature p and palmar vein characteristics v The fusion feature f′ of the completed palm print feature and palm vein feature is obtained by fusion p , the palm print and palm vein fusion recognition network is trained through cross entropy loss to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, and then the trained palm print and palm vein fusion recognition network model F is obtained, as follows: The fusion feature f of palm print feature and palm vein feature, and the fusion feature f′ of palm print feature and complemented palm vein feature v And the fusion feature f′ of the completed palm print feature and palm vein feature p , respectively, are sent to the fully connected layer, and then through the SoftMax layer to obtain the predicted probability distribution. The cross entropy loss L makes the predicted category of the sample equal to the true category of the sample: f=f p +f v (39) f′ v =f p +f′ v (40) f′ p =f′ p +f v (41) y i =SoftMax(FC(f)) (42) L=L1+L1 v +L1 p (48) Where i is the serial number of the batch sample; B is the batch size; y i ′ indicates that the i-th sample is the true category; f indicates the modality fusion feature vector; y i Represents the predicted probability distribution; SoftMax(FC(·)) It means that it will first pass through the fully connected layer and then the SoftMax layer; f′ v , f′ p , L1, L1 v and L1 p These are intermediate variables used to simplify calculations.

8. The palm print and palm vein fusion recognition method based on feature decoupling in missing mode according to claim 7 is characterized in that: In step 7, the test set is used to evaluate the accuracy of the trained palm print and palm vein fusion recognition network model F, as follows: Step 7.1: During the test, if both the palm print image and the palm vein image are present, the features of each modality are directly fused for recognition; if the palm print image or the palm vein image is missing, the known modality-specific features are used to generate the palm print cross-modality generator G. p Or palm vein cross-modal generator G v Generate missing modality-specific features, combine the shared features of known modalities, complete the missing modality features, and then fuse the features of each modality for recognition: For palmar vein modality loss: in, The common feature encoder representing the palmprint, represents the unique feature encoder of palmprint; connect(·) represents the modality fusion network; For missing palmprint modality: in, The common feature encoder representing the palm veins, A unique feature encoder representing palm veins; For both palm vein modality and palm vein modality, there is no missing: y i =SoftMax{FC[connect(F v (x v )+F p (x p ))]} (51) Step 7.2: During the recognition process, the correct recognition rate is used to evaluate the performance of the palm print and palm vein fusion recognition network model F; True To represent the cumulative number of samples predicted correctly, N Flace To represent the number of samples with cumulative prediction errors, the correct recognition rate CRR is defined as:

Images