An identity recognition method based on palm print and palm vein multimodal fusion

Abstract

The present invention discloses an identity recognition method based on multimodal fusion of palm prints and palm veins, and relates to the field of biometric identification technology. Specifically, the following steps are included: setting pairing labels for palm images in a database, setting the labels of palm images belonging to the same person to 1, and setting the labels to 0 if they are different; constructing and training an initial unimodal recognition network of palm prints and palm prints; constructing and training an initial cross-modal recognition network of palm prints and palm veins; constructing and training an initial overall recognition network through the trained unimodal recognition network of palm prints and palm prints and the trained cross-modal recognition network of palm prints and palm veins. The present invention innovatively proposes a hybrid identity recognition method that combines unimodal recognition of palm prints and palm prints with cross-modal recognition of palm prints and palm veins. This method can greatly improve accuracy and make matching closer to ideal.

Description

Technical Field

[0001] The present invention belongs to the technical field of biometric identification, and in particular relates to an identity recognition method based on multi-modal fusion of palm prints and palm veins. Background Art

[0002] Compared with traditional identification methods, biometric identification technology can provide more secure, reliable and fast identification and authentication results. The current single biometric identification technology has its own advantages and disadvantages. Palm prints are more universal and stable, and have rich feature information, but the anti-counterfeiting property of palm prints is not particularly good. They are easily obtained and forged illegally, and it is also difficult to verify the liveness. Affected by unfavorable factors such as the environment, noise and deception attacks, the security of single biometric identification technology needs to be strengthened, and the recognition accuracy needs to be improved. The challenge is: how to find a low-cost, efficient, stable and reliable biometric identification technology to solve the problem.

[0003] In addition to the palm print features, the palm vein features can also be used for identity recognition. Vein recognition is to illuminate the human palm with a near-infrared light source, and use the strong absorption characteristics of hemoglobin in the blood to near-infrared light, so that the palm image after near-infrared light irradiation will show darker lines at the subcutaneous veins. The vein features can be used for identity recognition. The main feature of vein recognition is non-invasive image acquisition, which has strong concealment and anti-counterfeiting properties, and also has a live verification function. Both palm prints and palm veins are located in the palm, with high correlation, and can be collected at the same time. The biometric recognition technologies with their unique advantages are fused to obtain comprehensiveness that a single biometric recognition technology does not have, and improve the accuracy of identity recognition. The inventor found that there is a disadvantage in the biometric fusion recognition technology. When the feature information of one of the modalities is leaked, the overall recognition performance will be worse than that of a single biometric recognition technology, affecting the overall recognition accuracy and reliability. Summary of the invention

[0004] The purpose of the present invention is to provide an identity recognition method based on the multimodal fusion of palm print and palm vein, using the idea of ​​"learning from each other's strengths and complementing each other's weaknesses" to overcome the shortcomings of a single biometric recognition technology in a certain aspect, as well as the problems of security, stability and reliability in the current biometric fusion recognition technology. The present invention performs cross-modal matching and recognition of palm print and palm vein while performing palm print and palm print matching. This hybrid identity recognition can improve the accuracy and reliability of recognition, help to more comprehensively and effectively guarantee and improve the overall performance of the system, improve the overall recognition efficiency, make the authentication and recognition process more accurate and safe, thereby meeting the needs of different application scenarios, and can be more deeply applied to industries with higher security requirements.

[0005] The technical solution to achieve the purpose of the present invention is: an identity recognition method based on palm print and palm vein multimodal fusion, comprising the following steps:

[0006] Step 1: Collect palm images of n persons, collect 2a palm print images and 2a palm vein images for each person, establish an image database, and divide the database into a training set and a test set. The ratio of the number of palm images in the training set and the test set is 1:1.

[0007] Step 2: Set pairing labels for the palm images in the training set, including pairing label M1 for palm print images and palm print images, pairing label M2 for palm print images and palm vein images, and pairing label M for two palm print images and one palm vein image. The pairing labels of palm images belonging to the same person, i.e., palm images of the same category, are set to 1, and the pairing labels of palm images of different categories are set to 0.

[0008] Step 3: Set pairing labels for the palm images in the test set, including pairing labels N for two palm print images and one palm vein image. The pairing labels of palm images belonging to the same person, i.e., the same category, are set to 1, and the pairing labels of palm images of different categories are set to 0.

[0009] Step 4: Construct the unimodal recognition network of the initial palmprint and the palmprint, and use the difference between the predicted value R1 output by the unimodal recognition network of the initial palmprint and the palmprint and the paired label M1 of the palmprint image and the palmprint image in the training set to construct the unimodal loss function L unimodal , the initial palmprint and the single-modal recognition network of the palmprint are trained to obtain the trained palmprint and the single-modal recognition network of the palmprint.

[0010] Step 5: Construct an initial palm print and palm vein cross-modal recognition network, and use the difference between the predicted value R2 output by the initial palm print and palm vein cross-modal recognition network and the paired label M2 of the palm print image and palm vein image in the training set to construct a cross-modal loss function L crossmodal , the initial palm print and palm vein cross-modal recognition network is trained to obtain a trained palm print and palm vein cross-modal recognition network.

[0011] Step 6: Construct an initial overall recognition network based on the trained palm print and palm print unimodal recognition network and the trained palm print and palm vein cross-modal recognition network, and construct an overall loss function L using the difference between the predicted value R output by the initial overall recognition network and the paired labels M of the two palm print images and one palm vein image in the training set. total , train the initial overall recognition network to obtain a trained overall recognition network.

[0012] Step 7: Calculate the recognition accuracy of the trained overall recognition network on the test set.

[0013] Compared with the prior art, the advantages of the present invention are:

[0014] 1) Compared with the problems of low recognition rate and relatively narrow application scope of single biometric recognition technology, the present invention innovatively proposes cross-modal recognition of palm prints and palm veins on the basis of single-modal recognition, integrating the two biometric recognition technologies, matching and identifying palm prints and palm prints while also matching and identifying palm prints and palm veins, thereby improving the recognition range, accuracy and efficiency, improving the performance of the overall system, and meeting the needs of different application scenarios.

[0015] 2) In view of some shortcomings of biometric fusion recognition technology, such as complex recognition, long recognition time, and if the feature information of one of the modalities is leaked during the feature fusion process, the overall recognition performance will be worse than that of a single biometric recognition technology. The present invention innovatively proposes a method for matching and recognizing palm prints and palm veins at the same time as palm print matching and recognition. Not only the feature correlation within the modality, but also the feature correlation between the modalities is considered, which can improve the recognition efficiency and accuracy, enhance the security of the system, and has strong generalization ability and robustness.

[0016] 3) This paper proposes a hybrid identity recognition method, which transforms the single-modal loss function L unimodal , the cross-modal loss function L crossmodal , the overall loss function L total Integrated in a unified deep learning framework, the prediction results of the trained palmprint and palmprint unimodal recognition network and the trained palmprint and palm vein cross-modal recognition network are subjected to a voting mechanism to construct an initial overall recognition network and train it to obtain a trained overall recognition network. The trained palmprint and palmprint unimodal recognition network may sometimes have recognition errors. For example, forged palmprints may be successfully recognized. The addition of the trained palmprint and palm vein cross-modal recognition network will reduce the number of such recognition errors, greatly improving the anti-counterfeiting degree of the trained overall recognition network. As long as the prediction result of one of the trained recognition networks is 0, the prediction result of the trained overall recognition network will eventually be 0, increasing the security of the trained overall recognition network, with double protection, making the authentication and recognition process more accurate and secure, so that it can be more deeply applied to industries with higher security requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The figure is a flow chart of an identity recognition method based on multimodal fusion of palm print and palm vein.

[0018] Figure 2Framework diagram for training initial palmprint and unimodal palmprint recognition network.

[0019] Figure 3 Framework diagram for training the cross-modal recognition network for initial palm prints and palm veins.

[0020] Figure 4 Framework diagram for training the initial holistic recognition network.

[0021] Figure 5 Framework diagram of the overall recognition network trained for testing. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0023] Combination Figure 1 to Figure 5 The present invention provides an identity recognition method based on palm print and palm vein multimodal fusion, comprising the following steps:

[0024] Step 1: Collect palm images of n persons, 200<n<500, collect 2a palm print images and 2a palm vein images 10<a<40 for each person, establish an image database, and divide the database into a training set and a test set. The ratio of the number of palm images in the training set and the test set is 1:1. The training set X=[x p 11 ,x p 12 ,x p 13 …x p ij ; x v 11 ,x v 12 ,x v 13 …x v ij ], where i represents the number of the person, i = 1, 2, 3 ... n, j represents the number of the collected palm image, j = 1, 2, 3 ... a, p represents the collected palm print image, v represents the collected palm vein image, x p ij is the jth palm print image of the i-th person in the training set, x v ij is the jth palm vein image of the i-th person in the training set; the test set Y = [y p 11 , y p 12 ,y p 13 …y p ij ;yv 11 ,y v 12 ,y v 13 …y v ij ],y p ij is the jth palm print image of the i-th person in the test set, y v ij is the jth palm vein image of the i-th person in the test set.

[0025] Step 2: Set pairing labels for the palm images in the training set, including pairing labels M1 for palm print images and palm print images, M2 for palm print images and palm vein images, and M for two palm print images and one palm vein image. The pairing labels of palm images belonging to the same person, i.e., the same category, are set to 1, and the pairing labels of palm images of different categories are set to 0:

[0026] Step 2-1, set the pairing label M1 for the two palm print images. First, set the pairing label of the positive sample image. Randomly extract two palm print images of the same person from the training set, and set the label to 1, that is, i represents the number of the person, i = 1, 2, 3 ... n, r, d represent the numbers of the two palm print images of the i-th person, r = 1, 2, 3 ... a, d = 1, 2, 3 ... a, r ≠ d, is the rth palmprint image of the i-th person randomly selected, is the dth palmprint image of the i-th person randomly selected; after setting the pairing label of the positive sample image, set the pairing label of the negative sample image, randomly select a palmprint image of one person from the training set, and then randomly select a palmprint image of another person. The two palmprint images do not belong to the same category, and the label is set to 0, that is, i' represents the number of the person, i'=1,2,3…n, i≠i', is the dth palmprint image of the i'th person randomly selected.

[0027] Step 2-2, set the pairing label M2 for the palm print image and palm vein image. First, set the pairing label of the positive sample image. Randomly extract a palm print image and a palm vein image of a person from the training set, and set the label to 1, that is, is the dth palm vein image of the randomly selected person i; after setting the paired labels of the positive sample images, set the paired labels of the negative sample images, randomly select a palm print image of one person from the training set, and then randomly select a palm vein image of another person. The two palm images do not belong to the same category, and the labels are set to 0, that is, is the d-th palm vein image of the i'th person randomly selected.

[0028] Step 2-3, set the pairing label M for the two palm print images and one palm vein image. First, set the pairing label of the positive sample image. Randomly extract two palm print images and one palm vein image of a person from the training set. The three palm images belong to the same person, that is, the same category, and the label is set to 1, that is, σ, μ both represent the number of the palm print image of the i-th person extracted, γ represents the number of the palm vein image of the i-th person extracted, σ=1,2,3…a, μ=1,2,3…a, γ=1,2,3…a, σ≠μ, is the randomly selected palm print images of the ith person, is the μth palmprint image of the i-th person randomly selected, is the γth palm vein image of the i-th person randomly selected; after setting the pairing label of the positive sample image, set the pairing label of the negative sample image, randomly select two palm print images and one palm vein image that do not belong to the same person from the training set, and set the label to 0, that is, i” represents the number of a person, i”=1,2,3…n, i≠i'≠i”, is the μth palmprint image of the i'th person randomly selected, is the γth palm vein image of the randomly selected i-th person.

[0029] Step 3: Set the pairing labels N for the two palm print images and one palm vein image. First, the pairing labels for the positive sample images are selected. Two palm print images and one palm vein image of a person are randomly selected from the test set. The three palm images belong to the same category, and the labels are set to 1, that is, i represents the number of the person, i = 1, 2, 3 ... n, σ, μ both represent the number of the palm print image of the i-th person extracted, γ represents the number of the palm vein image of the i-th person extracted, σ ≠ μ, represents the σth palmprint image of the i-th person randomly selected, represents the μth palmprint image of the i-th person randomly selected, Denotes the γth palm vein image of the randomly selected person i; after setting the pairing label of the positive sample image, set the pairing label of the negative sample image, randomly select two palm print images and one palm vein image that do not belong to the same person from the test set, and set the label to 0, that is, i', i" represent the number of the person, i'= 1,2,3...n, i"=1,2,3...n, i≠i'≠i", is the μth palmprint image of the i'th person randomly selected, is the γth palm vein image of the randomly selected i-th person.

[0030] Step 4: Construct the unimodal recognition network of the initial palmprint and the palmprint, and use the difference between the predicted value R1 output by the unimodal recognition network of the initial palmprint and the palmprint and the paired label M1 of the palmprint image and the palmprint image in the training set to construct the unimodal loss function L unimodal , train the initial palmprint and the single-mode recognition network of the palmprint to obtain the trained palmprint and the single-mode recognition network of the palmprint:

[0031] Step 4-1, constructing an initial palm print and a unimodal recognition network of the palm print, wherein the initial palm print and the unimodal recognition network of the palm print include a dual-branch feature extraction module and a feature discrimination module, wherein the dual-branch feature extraction module is used to extract feature information of the input palm image, and the feature discrimination module outputs a prediction value by comparing the similarity between the extracted feature information.

[0032] Step 4-2: Randomly extract two palm print images from the training set Input into the dual-branch feature extraction module, where k1, k2∈[1, n], l1, l2∈[1, a], use the same convolutional neural network CNN to extract the feature information of the two palmprint images and obtain the feature vectors h' of the two palmprint images p1 、h' p2 ;

[0033] Using cosine similarity To calculate the similarity between the feature information of two palmprint images, set the ternary loss function Where i represents the person's ID, i = 1, 2, 3...n, f represents the negative sample image, z represents the positive sample image, and the purpose is to calculate the cosine similarity Sim of two palm print images of the same person. z Greater than the cosine similarity Sim calculated from two palm print images from different people f .

[0034] Step 4-3: Substitute the Sim(h' p1 , h' p2 ) is input into the feature discrimination module, passes through the fully connected layer FC, and then passes through a Sigmoid activation function to obtain the initial palm print and the prediction result R1 of the palm print unimodal recognition network, and set the contrast loss function The initial palm print and the unimodal recognition network of the palm print are learned by the gradient descent method, and the parameters in the unimodal recognition network of the initial palm print and the palm print are updated by back propagation so that the value of the unimodal loss function L unimodal =L c1 + L ide1 Decreases, when the predicted value is closer to the true value, i.e. the label value, L unimodalThe closer the value is to 0, the closer the network is to stability. The purpose of our training is to hope that the initial palm print and the palm print unimodal recognition network can predict as much as possible whether the two input palm print images belong to the same person, that is, R1=0, 1. Finally, we get the trained palm print and palm print unimodal recognition network. If the two input palm print images belong to the same person, the output result is 1. If the two input palm print images do not belong to the same person, the output result is 0.

[0035] Step 5: Construct an initial palm print and palm vein cross-modal recognition network, and use the difference between the predicted value R2 output by the initial palm print and palm vein cross-modal recognition network and the paired label M2 of the palm print image and palm vein image in the training set to construct a cross-modal loss function L crossmodal , train the initial palm print and palm vein cross-modal recognition network to obtain the trained palm print and palm vein cross-modal recognition network:

[0036] Step 5-1, constructing an initial cross-modal recognition network for palm prints and palm veins, wherein the initial cross-modal recognition network for palm prints and palm veins includes a dual-branch feature extraction module and a feature discrimination module, wherein the dual-branch feature extraction module is used to extract feature information of an input palm image, and the feature discrimination module outputs a prediction value by comparing the similarity between the extracted feature information.

[0037] Step 5-2: Randomly extract a palm print image from the training set and a palm vein image The feature extraction module is input into the dual-branch feature extraction module, and the feature information of the palm print image and the palm vein image is extracted using the same convolutional neural network CNN to obtain the feature vectors h of the palm print image and the palm vein image. p 、h v ;

[0038] Using cosine similarity To calculate the similarity between the feature information of palm print image and palm vein image, set the ternary loss function Where i represents the person's ID, i = 1, 2, 3...n, f* represents the negative sample image, z* represents the positive sample image, and the purpose is to calculate the cosine similarity Sim between the palm print image and the palm vein image of the same person. z* It can be greater than the cosine similarity Sim calculated from palm print images and palm vein images from different people f* .

[0039] Step 5-3: Substitute the Sim(h p ,h v) is input into the feature discrimination module, passes through the fully connected layer FC, and then passes through a Sigmoid activation function to obtain the prediction result R2 of the cross-modal recognition network of the initial palm print and palm vein, and sets the contrast loss function The initial palm print and palm vein cross-modal recognition network uses the gradient descent method to learn, and back propagates to update the parameters in the initial palm print and palm vein cross-modal recognition network so that the value of the cross-modal loss function L crossmodal = L c2 +L ide2 Decreases, when the predicted value is closer to the true value, i.e. the label value, L crossmodal The closer the value of is to 0, the closer the network is to stability. The purpose of our training is to hope that the initial palm print and palm vein cross-modal recognition network can predict as much as possible whether the input palm print image and palm vein image belong to the same person, that is, R1=0, 1. Finally, the trained palm print and palm vein cross-modal recognition network is obtained. If the input palm print image and palm vein image belong to the same person, the output result is 1. If the input palm print image and palm vein image do not belong to the same person, the output result is 0.

[0040] Step 6: Construct an initial overall recognition network based on the trained palm print and palm print unimodal recognition network and the trained palm print and palm vein cross-modal recognition network, and construct an overall loss function L using the difference between the predicted value R output by the initial overall recognition network and the paired labels M of the two palm print images and one palm vein image in the training set. total , train the initial overall recognition network to obtain the trained overall recognition network:

[0041] Step 6-1: Perform a voting mechanism on the prediction results of the initial palm print and palm print unimodal recognition network and the initial palm print and palm vein cross-modal recognition network to construct an initial overall recognition network.

[0042] Step 6-2: Randomly extract two palm print images and one palm vein image from the training set, input them into the initial overall recognition network, obtain a prediction result R = R1 × R2, and construct the overall loss function Calculate the difference between the predicted value output by the initial overall recognition network and the paired label, i represents the person's number, i = 1, 2, 3...n, λ1, λ2 are balance parameters, train the initial overall recognition network, use the gradient descent method to learn, back propagate and update the parameters in the initial overall recognition network to reduce the value of the overall loss function. When the predicted value is closer to the true value, that is, the label value, L total The closer the value is to 0, the closer the network is to being stable, and finally a trained overall recognition network is obtained. If the three input palm images belong to the same person, the output prediction result is 1, otherwise the output prediction result is 0, and finally a trained overall recognition network is obtained.

[0043] Step 7: Randomly extract a palm print image from the test set, and then randomly extract a palm print image and a palm vein image, input them into the trained overall recognition network, determine whether the three input palm images belong to the same person, and output the prediction result R test = 0, 1, based on the paired labels N of two palm print images and one palm vein image in step 3, if the output prediction result R test The same as label N, the prediction result of the trained overall recognition network is correct, otherwise it is incorrect. The cumulative number of correct predictions is m1, and the cumulative number of incorrect predictions is m2. Finally, the recognition accuracy of the trained overall recognition network on the test set is

Claims

1. An identity recognition method based on palm print and palm vein multimodal fusion, characterized in that: The following steps are involved: Step 1: collect palm images of n persons, collect 2a palm print images and 2a palm vein images for each person, establish an image database, and divide the database into a training set and a test set, with the ratio of palm images in the training set to that in the test set being 1:1; Step 2, set pairing labels for the palm images in the training set, including pairing labels M1 for palm print images and palm print images, M2 for palm print images and palm vein images, and M for two palm print images and one palm vein image. The pairing labels of palm images belonging to the same person, i.e., the same category, are set to 1, and the pairing labels of palm images of different categories are set to 0. Step 3, set pairing labels for the palm images in the test set, including pairing labels N for two palm print images and one palm vein image. The pairing labels of palm images belonging to the same person, i.e., the same category, are set to 1, and the pairing labels of palm images of different categories are set to 0. Step 4: Construct the unimodal recognition network of the initial palmprint and the palmprint, and use the difference between the predicted value R1 output by the unimodal recognition network of the initial palmprint and the palmprint and the paired label M1 of the palmprint image and the palmprint image in the training set to construct the unimodal loss function L unimodal , training the initial palmprint and the single-mode recognition network of the palmprint to obtain the trained palmprint and the single-mode recognition network of the palmprint; Step 5: Construct an initial palm print and palm vein cross-modal recognition network, and use the difference between the predicted value R2 output by the initial palm print and palm vein cross-modal recognition network and the paired label M2 of the palm print image and palm vein image in the training set to construct a cross-modal loss function L crossmodal , training the initial palm print and palm vein cross-modal recognition network to obtain a trained palm print and palm vein cross-modal recognition network; Step 6: Construct an initial overall recognition network based on the trained palm print and palm print unimodal recognition network and the trained palm print and palm vein cross-modal recognition network, and construct an overall loss function L using the difference between the predicted value R output by the initial overall recognition network and the paired labels M of the two palm print images and one palm vein image in the training set. total , train the initial overall recognition network to obtain a trained overall recognition network; the details are as follows: Step 6-1, the prediction results of the initial palm print and palm print unimodal recognition network and the initial palm print and palm vein cross-modal recognition network are subjected to a voting mechanism to construct an initial overall recognition network; Step 6-2, randomly extract two palm print images and one palm vein image from the training set, input them into the initial overall recognition network, and obtain a prediction result R = R1 × R2; Constructing the overall loss function Calculate the difference between the predicted value output by the initial overall recognition network and the paired label, i represents the person's number, i = 1, 2, 3...n, λ1, λ2 are balance parameters, train the initial overall recognition network, use the gradient descent method to learn, back propagate and update the parameters in the initial overall recognition network, so that the value of the overall loss function decreases, and finally obtain the trained overall recognition network. If the three input palm images belong to the same person, the output prediction result is 1, otherwise the output prediction result is 0, and finally obtain the trained overall recognition network; Step 7: Calculate the recognition accuracy of the trained overall recognition network on the test set.

2. According to claim 1, the identity recognition method based on palm print and palm vein multimodal fusion is characterized in that: In step 1, palm images of n persons are collected. For each person, 2a palm print images and 2a palm vein images are collected to establish an image database. The database is divided into a training set and a test set. The ratio of the number of palm images in the training set and the test set is 1:1, as follows: Collect palm images of n individuals, where 200 < n < 500. For each individual, collect 2a palmprint images and 2a palm vein images, where 10 < a < 40. Establish an image database and divide the database into a training set and a test set. The ratio of the number of palm images in the training set to the test set is 1:

1. The training set X = [x p 11 , x p 12 , x p 13 … x p ij ; x v 11 , x v 12 , x v 13 … x v ij , where i represents the individual number, i = 1, 2, 3… n, j represents the palm image number collected, j = 1, 2, 3… a, p represents the collected palmprint image, v represents the collected palm vein image, x p ij is the j-th palmprint image of the i-th individual in the training set, and x v ij is the j-th palm vein image of the i-th individual in the training set; the test set Y = [y p 11 , y p 12 , y p 13 … y p ij ; y v 11 , y v 12 , y v 13 … y v ij , y p ij is the j-th palmprint image of the i-th individual in the test set, and y v ij is the j-th palm vein image of the i-th individual in the test set.

3. The identity recognition method based on palm print and palm vein multimodal fusion according to claim 2 is characterized in that: In step 2, pairing labels are set for the palm images in the training set, including pairing labels M1 for palm print images and palm print images, M2 for palm print images and palm vein images, and M for two palm print images and one palm vein image. The pairing labels of palm images belonging to the same person, i.e., palm images of the same category, are set to 1, and the pairing labels of palm images of different categories are set to 0, as follows: Step 2-1, set the pairing label M1 for the two palm print images. First, set the pairing label of the positive sample image. Randomly extract two palm print images of the same person from the training set, and set the label to 1, that is, i represents the number of the person, i = 1, 2, 3 ... n, r, d represent the numbers of the two palm print images of the i-th person, r = 1, 2, 3 ... a, d = 1, 2, 3 ... a, r ≠ d, is the rth palmprint image of the i-th person randomly selected, is the dth palmprint image of the i-th person randomly selected; after setting the pairing label of the positive sample image, set the pairing label of the negative sample image, randomly select a palmprint image of one person from the training set, and then randomly select a palmprint image of another person. The two palmprint images do not belong to the same category, and the label is set to 0, that is, i′ represents the number of the person, i'=1,2,3…n, i≠i′, is the d-th palmprint image of the i′th person randomly selected; Step 2-2, set the pairing label M2 for the palm print image and palm vein image. First, set the pairing label of the positive sample image. Randomly extract a palm print image and a palm vein image of a person from the training set, and set the label to 1, that is, is the dth palm vein image of the randomly selected person i; after setting the paired labels of the positive sample images, set the paired labels of the negative sample images, randomly select a palm print image of one person from the training set, and then randomly select a palm vein image of another person. The two palm images do not belong to the same category, and the labels are set to 0, that is, is the dth palm vein image of the i′th person randomly selected; Step 2-3, set the pairing label M for the two palm print images and one palm vein image. First, set the pairing label of the positive sample image. Randomly extract two palm print images and one palm vein image of a person from the training set. The three palm images belong to the same person, that is, the same category, and the label is set to 1, that is, Where σ, μ both represent the number of the palm print image of the i-th person extracted, γ represents the number of the palm vein image of the i-th person extracted, σ=1,2,3…a, μ=1,2,3…a, γ=1,2,3…a, σ≠μ, is the randomly selected palm print images of the ith person, is the μth palmprint image of the oth person randomly selected, is the γth palm vein image of the oth person randomly selected; after setting the pairing label of the positive sample image, set the pairing label of the negative sample image, randomly select two palm print images and one palm vein image that do not belong to the same person from the training set, and set the label to 0, that is, i” represents the number of a person, i”=1,2,3…n, i≠i′≠i”, is the μth palmprint image of the i′th person randomly selected, is the γth palm vein image of the randomly selected i-th person.

4. The identity recognition method based on palm print and palm vein multimodal fusion according to claim 3 is characterized in that: In step 3, pairing labels are set for the palm images in the test set, including pairing labels N for two palm print images and one palm vein image. The pairing labels of palm images belonging to the same person, i.e., the same category, are set to 1, and the pairing labels of palm images of different categories are set to 0, as follows: Set the pairing label N for two palm print images and one palm vein image. First, the pairing label of the positive sample image is selected. Two palm print images and one palm vein image of a person are randomly selected from the test set. The three palm images belong to the same category, and the label is set to 1, that is, i represents the number of the person, i = 1, 2, 3 ... n, σ, μ both represent the number of the palm print image of the i-th person extracted, γ represents the number of the palm vein image of the i-th person extracted, σ ≠ μ, represents the σth palmprint image of the i-th person randomly selected, represents the μth palmprint image of the i-th person randomly selected, Denotes the γth palm vein image of the randomly selected person i; after setting the pairing label of the positive sample image, set the pairing label of the negative sample image, randomly select two palm print images and one palm vein image that do not belong to the same person from the test set, and set the label to 0, that is, i′ and i″ both represent the number of a person, i′=1,2,3…n, i″=1,2,3…n, i≠i′≠i″, is the μth palmprint image of the i′th person randomly selected, is the γth palm vein image of the randomly selected i-th person.

5. The identity recognition method based on palm print and palm vein multimodal fusion according to claim 4 is characterized in that: In step 4, a unimodal recognition network of the initial palmprint and the palmprint is constructed, and the difference between the predicted value R1 output by the unimodal network of the initial palmprint and the palmprint and the paired label M1 of the palmprint image and the palmprint image in the training set is used to construct the unimodal loss function L unimodal , the initial palmprint and the single-mode recognition network of the palmprint are trained to obtain the trained palmprint and the single-mode recognition network of the palmprint, as follows: Step 4-1, constructing an initial palm print and a unimodal palm print recognition network, wherein the initial palm print and the unimodal palm print recognition network include a dual-branch feature extraction module and a feature discrimination module, wherein the dual-branch feature extraction module is used to extract feature information of an input palm image, and the feature discrimination module outputs a prediction value by comparing the similarity between the extracted feature information; Step 4-2: Randomly extract two palmprint images from the training set Input into the dual-branch feature extraction module, where k1, k2 ∈ [1, n], l1, l2 ∈ [1, a], use the same convolutional neural network CNN to extract the feature information of the two palmprint images and obtain the feature vectors h′ of the two palmprint images p1 , h′ p2 ; Using cosine similarity To calculate the similarity between the feature information of two palmprint images, set the ternary loss function Where i represents the person's ID, i = 1, 2, 3...n, f represents the negative sample image, z represents the positive sample image, and the purpose is to calculate the cosine similarity Sim of two palm print images of the same person. z Greater than the cosine similarity Sim calculated from two palm print images from different people f ; Step 4-3: Substitute the Sim(h' p1 , h′ p2 ) is input into the feature discrimination module, passes through the fully connected layer FC, and then passes through a Sigmoid activation function to obtain the initial palm print and the prediction result R1 of the palm print unimodal recognition network, and set the contrast loss function , the initial palm print and the palm print unimodal recognition network use the gradient descent method to learn, and back propagate to update the parameters in the initial palm print and the palm print unimodal recognition network so that the value of the unimodal loss function L unimodal =L c1 +L ide1 Descend, and finally get the trained palm print and palm print unimodal recognition network. If the two input palm print images belong to the same person, the output result is 1. If the two input palm print images do not belong to the same person, the output result is 0.

6. The identity recognition method based on palm print and palm vein multimodal fusion according to claim 5 is characterized in that: In step 5, an initial palm print and palm vein cross-modal recognition network is constructed, and the cross-modal loss function L is constructed using the difference between the predicted value R2 output by the palm print and palm vein cross-modal recognition network and the paired label M2 of the palm print image and palm vein image in the training set. crossmodal , the initial palm print and palm vein cross-modal recognition network is trained to obtain the trained palm print and palm vein cross-modal recognition network, as follows: Step 5-1, constructing an initial palm print and palm vein cross-modal recognition network, wherein the initial palm print and palm vein cross-modal recognition network comprises a dual-branch feature extraction module and a feature discrimination module, wherein the dual-branch feature extraction module is used to extract feature information of an input palm image, and the feature discrimination module outputs a prediction value by comparing the similarity between the extracted feature information; Step 5-2: Randomly extract a palm print image from the training set and a palm vein image The feature extraction module is input into the dual-branch feature extraction module, and the feature information of the palm print image and the palm vein image is extracted using the same convolutional neural network CNN to obtain the feature vectors h of the palm print image and the palm vein image. p 、h v ; Using cosine similarity To calculate the similarity between the feature information of palm print image and palm vein image, set the ternary loss function Where i represents the number of the person, i = 1, 2, 3...n, f* represents the negative sample image, z* represents the positive sample image, and the purpose is to calculate the cosine similarity Sim between the palm print image and the palm vein image of the same person. z* It can be greater than the cosine similarity Sim calculated from palm print images and palm vein images from different people f* ; Step 5-3: Substitute the Sim(h p ,h v ) is input into the feature discrimination module, passes through the fully connected layer FC, and then passes through a Sigmoid activation function to obtain the prediction result R2 of the cross-modal recognition network of the initial palm print and palm vein, and sets the contrast loss function The initial palm print and palm vein cross-modal recognition network uses the gradient descent method to learn, and back propagates to update the parameters in the initial palm print and palm vein cross-modal recognition network so that the value of the cross-modal loss function L crossmodal =L c2 +L ide2 Finally, we get a trained cross-modal recognition network for palm prints and palm veins. If the input palm print image and palm vein image belong to the same person, the output result is 1. If the input palm print image and palm vein image do not belong to the same person, the output result is 0.

7. The identity recognition method based on palm print and palm vein multimodal fusion according to claim 1 is characterized in that: In step 7, the trained overall recognition network model is tested, and the recognition accuracy of the trained overall recognition network on the test set is calculated, as follows: Randomly extract a palm print image from the test set, then randomly extract a palm print image and a palm vein image, input them into the trained overall recognition network, judge whether the three input palm images belong to the same person, and output the prediction result R test = 0, 1, based on the paired labels N of two palm print images and one palm vein image in step 3, if the output prediction result R test The same as label N, the prediction result of the trained overall recognition network is correct, otherwise it is incorrect. The cumulative number of correct predictions is m1, and the cumulative number of incorrect predictions is m2. Finally, the recognition accuracy of the trained overall recognition network on the test set is

Images