A palm print and palm vein fusion recognition method in missing modality based on cross-modal generation

Abstract

The present invention discloses a palm print and palm vein fusion recognition method in the missing modality based on cross-modal generation. Aiming at the difficulty of generating missing modalities in pixel space and the redundancy of multi-modal fusion information, the present invention proposes two innovations: (1) A semi-supervised cross-modal generation method is proposed to gradually realize texture style transfer and texture structure generation between cross-modal images, and complete the generation of arbitrary missing modality images. (2) An adaptive sparse modality fusion model is proposed to reduce the redundant information of multi-modal fusion features and enhance the robustness of the model to dynamic changes in modality feature quality. In the case of missing images in any modality in the test set, a generator G is used to generate palm print images from palm vein images. vp and a generator G that generates a palm print image from a palm vein image pv Combined with the adaptive sparse modal fusion model, a robust recognition method for palm vein and palm print dual-modal hand biometrics is formed in the absence of any modality.

Description

Technical Field

[0001] The present invention belongs to the technical field of biometric feature recognition, and in particular relates to a palm print and palm vein fusion recognition method in a missing modality based on cross-modality generation. Background Art

[0002] In recent years, the application of biometric recognition technology can be seen everywhere. In life, fingerprint recognition on access control systems, fingerprint recognition on mobile phones, and face recognition applications; at work, the application of fingerprint clocking in and out. Among them, biometric recognition technology based on palm print and palm vein fusion has received more and more attention due to its rich identification characteristics, non-contact and hygienic collection methods. However, in practical applications, due to equipment failure, information loss, personal physical defects and other reasons, some data is lost during data collection, transmission or storage. In this case, the existing multimodal fusion recognition method will have serious performance degradation. Therefore, it is of great practical significance to design a cross-modal generation model to complete the missing modality and then perform palm print and palm vein fusion recognition method. At present, deep learning has achieved excellent performance in the field of multimodal biometric recognition. Wang et al. added a self-attention mechanism to the multimodal finger biometric fusion network and obtained better recognition efficiency. Lv et al. input the multi-level features of the three-stream network model into the channel attention mechanism to model the common feature space so that the network focuses on salient targets. Yu et al. proposed to introduce modular attention mechanism into a common attention layer composed of autoencoder network, self-attention unit and guided attention unit to help collaboratively model the interaction between modalities and modalities, and deeply superimpose the attention layer by using encoding strategy. However, in the field of biometric recognition, in-depth research on the problem of modal missing has not been fully carried out, which provides an area to be explored for the further development of biometric recognition technology. The lack of a deep understanding and solution to the modal missing situation may hinder the robustness and comprehensiveness of biometric recognition technology in practical applications. Therefore, in-depth research on the problem of modal missing can not only fill the research gap in this field, but also provide new methods and strategies for biometric recognition technology, so as to achieve more reliable and effective recognition results when facing challenges in practical scenarios. This invention will open up new possibilities for the future development of biometric recognition technology. Summary of the invention

[0003] The purpose of the present invention is to provide a palm print and palm vein fusion recognition method under missing modalities based on cross-modal generation, aiming at the problem that some modal data may be lost during data collection, transmission or storage due to equipment failure, information loss, personal physical defects and other reasons in practical applications, and the existing multimodal fusion method, giving each modality the same weight during modal fusion, without considering the pros and cons of the modality and the redundancy of modal information. The present invention is composed of a cross-modal generation module and a multimodal fusion recognition module. First, the cross-modal generation module combines the inter-modal and intra-modal comparison learning strategies designed by the present invention in a semi-supervised manner, and uses the complete modality to complete the missing modality, and then sends the obtained complete modal data to the multimodal fusion recognition module. After feature extraction by VGG16 combined with the spatiotemporal attention mechanism, the adaptive sparse modal fusion module proposed by the present invention performs multimodal fusion, and the module is trained based on the mean square error loss and cross entropy loss to obtain accurate recognition results. Even in the case where the quality of some modalities is poor or even missing, the method can still obtain accurate recognition results to meet the needs of different application scenarios.

[0004] The technical solution to achieve the purpose of the present invention is: a palm print and palm vein fusion recognition method in missing mode based on cross-modal generation, comprising the following steps:

[0005] Step 1: collect palm prints and palm vein images of n individuals, establish a palm print and palm vein image database, divide the above database into a training set and a test set at a ratio of 1:1, both the palm print images and the palm vein images in the test set are missing, and the missing rate is β, then go to step 2.

[0006] Step 2: Construct a generator G that generates palm print images from palm vein images vp , a generator G that generates a palm print image from a palm vein image pv , cross-modal palmprint generator G p , Palmprint Identifier D p Palm vein generator G v and palm vein detector D v , go to step 3.

[0007] Step 3: Train the generator G that generates palm print images from palm vein images in a semi-supervised manner vp and a generator G that generates a palm vein image from a palm print image pv , combining content, texture, color consistency loss and intra-modality contrast loss in a supervised manner to train a supervised cross-modal palmprint generator G p , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm print images from palm vein images vp and palmprint discriminator D p , where Gp and G vp The weights are shared and the generator G that generates palm print images from palm vein images is trained in parallel in a semi-supervised manner. vp At the same time, the supervised cross-modal palm vein generator G is trained by combining content, texture, color consistency loss and intra-modal contrast loss in a supervised manner v , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm vein images from palm print images pv and palm vein detector D v , where G v and G pv The weights are shared and the generator G that generates palm vein images from palm print images is trained in parallel in a semi-supervised manner. pv , go to step 4.

[0008] Step 4: Build and train the adaptive sparse modal fusion model F r First, VGG16 combined with the spatiotemporal attention mechanism is used to extract features from palm print images and palm vein images to obtain the palm print modal feature f p and palmar vein modal characteristics f v , and then the palmprint modal feature f p and palmar vein modal characteristics f v Adaptive sparse fusion obtains the fusion feature f of palm print and palm vein, and finally uses mean square error loss and cross entropy loss to train the adaptive sparse modal fusion model F r To ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, through this training process, the trained adaptive sparse modal fusion module model F is obtained. r to proceed to step 5.

[0009] Step 5: Combine the trained palm vein image with the palm print image generator G vp and a generator G that generates a palm print image from a palm vein image pv and adaptive sparse modality fusion model F r , construct the overall palm print and palm vein fusion recognition model F, and go to step 6.

[0010] Step 6: Input the palm print image and palm vein image in the test set into the trained overall palm print and palm vein fusion recognition model F, and use the correct recognition rate (CRR) to verify the accuracy of the network, and use the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to verify the quality of the missing modality image generated across modalities.

[0011] Compared with the prior art, the present invention has the following significant advantages:

[0012] 1) In the absence of real sample images, the present invention can use the existing modality images in a semi-supervised manner to generate high-quality missing modality images across modalities by combining intra-modality and inter-modality contrast losses with content, texture, and color consistency losses to complete the dataset.

[0013] 2) In the process of modal screening and fusion, the random variable r is obtained by random sampling according to the probability distribution p of each modality each time. j =[r j p ,r j v ] T ,j=1,2,…,512:,r j The value of is a category in p, and the process of selecting only one sample each time is equivalent to the dropout operation, which can prevent the model from overfitting and improve the robustness of the model.

[0014] 3) The present invention not only exhibits excellent recognition effect when the modality is complete and intact, but also can obtain accurate recognition results when some modalities are damaged or even missing, thus meeting the needs of different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The present invention is a flowchart of a palm print and palm vein fusion recognition method in a missing modality based on cross-modal generation.

[0016] Figure 2 This is an overall model diagram of a palm print and palm vein fusion recognition method in missing modality based on cross-modal generation of the present invention.

[0017] Figure 3 This is a cross-modal generation model diagram in a palm print and palm vein fusion recognition method in a missing modality based on cross-modal generation of the present invention.

[0018] Figure 4 This is a diagram of an adaptive sparse modality fusion model in a palm print and palm vein fusion recognition method in a missing modality based on cross-modality generation of the present invention.

[0019] Figure 5 The present invention provides a palm vein image cross-modally generated from a palm print image in a palm print and palm vein fusion recognition method in a missing modality based on cross-modal generation.

[0020] Figure 6 A palmprint image is cross-modally generated from a palm vein image in a palmprint and palm vein fusion recognition method in a missing modality based on cross-modal generation of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The following will further introduce the specific implementation method, as well as the technical difficulties and inventive points of this invention in combination with this design example.

[0023] Combination Figure 1-2 The present invention discloses a palm print and palm vein fusion recognition method in missing mode based on cross-modal generation, which combines content, texture, color consistency loss and intra-modal contrast loss in a supervised manner to train a supervised cross-modal palm print generator G. p or cross-modal palm vein generator G v , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm print images from palm vein images vp and palmprint discriminator D p Or a generator G that generates a palm vein image from a palm print image pv and palm vein detector D v , the overall semi-supervised way is used to train the generator G that generates palm print images from palm vein images in parallel vp Or a generator G that generates a palm vein image from a palm print image pv , in order to improve the quality of the missing modality images completed by cross-modal generation. Then, an adaptive sparse modality fusion model F is constructed and trained. r , by combining VGG16 and spatiotemporal attention mechanism to construct a feature extraction network to extract deeper discriminative features of dual-modal palm prints and palm vein images, and then fuse multimodal features through adaptive sparse modal fusion to reduce redundant information of multimodal fusion features and enhance the robustness of the model to dynamic changes in modal feature quality. Cross entropy loss is used to train multimodal fusion recognition to ensure that the maximum probability of the classifier is equal to the probability of its corresponding label. When the palm print modality or palm vein modality is missing, the image of the known modality is sent to the palm vein image generator G to generate the palm print image. vp Or a generator G that generates a palm vein image from a palm print image pv , get the missing modality image, and send the known modality image and the completed missing modality image to the adaptive sparse modality fusion model F r , after modal fusion, the model is input into the classification network for prediction. Even when the quality of some modalities is poor or even uncertain, the model is missing, this method can still obtain accurate recognition results to meet the needs of different application scenarios. The specific steps are as follows:

[0024] Step 1: collect palm prints and palm vein images of n individuals, establish a palm print and palm vein image database, and divide the above database into a training set and a test set at a ratio of 1:1. Both the palm print images and the palm vein images in the test set are missing, and the missing rate is 0<β<1, then go to step 2.

[0025] Step 2: Construct a generator G that generates palm print images from palm vein images vp , a generator G that generates a palm print image from a palm vein image pv , cross-modal palmprint generator G p , Palmprint Identifier D p Palm vein generator G v and palm vein detector D v .

[0026] Based on the residual blocks in ResNet, we construct the generator G that generates palm print images from palm vein images. vp , a generator G that generates a palm vein image from a palm print image pv , cross-modal palmprint generator G p , Palmprint Identifier D p , cross-modal palm vein generator G v , and palm vein detector D v encoder and decoder.

[0027] Go to step 3.

[0028] Step 3: The present invention proposes for the first time to use a combination of supervised and unsupervised training methods, and overall training in a semi-supervised manner to improve the quality of the missing modality images generated across modalities, such as Figure 3 The generator G that generates palm print images from palm vein images is trained in a semi-supervised manner. vp and a generator G that generates a palm vein image from a palm print image pv , combining content, texture, color consistency loss and intra-modality contrast loss in a supervised manner to train a supervised cross-modal palmprint generator G p , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm print images from palm vein images vp and palmprint discriminator D p , where G p and G vp The weights are shared and the generator G that generates palm print images from palm vein images is trained in parallel in a semi-supervised manner. vp At the same time, the supervised cross-modal palm vein generator G is trained by combining content, texture, color consistency loss and intra-modal contrast loss in a supervised manner v, in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm vein images from palm print images pv and palm vein detector D v , where G v and G pv The weights are shared and the generator G that generates palm vein images from palm print images is trained in parallel in a semi-supervised manner. pv .

[0029] Step 3.1: Palm vein image generates palm print image generator G vp Using a complete palm print and palm vein database with labels, we train a cross-modal palm print generator G in a supervised manner by combining content, texture, color consistency loss and intra-modal contrast loss. p , the palm vein image x v Input to G p In the above example, a fake palmprint image x′ is generated. p , using L content Make the fake palmprint image generated by the generator closer to the real palmprint image; use L texture Make the fake palmprint texture generated by the generator closer to the real palmprint texture; use L color Make the fake palm print image color generated by the generator closer to the real palm print image color; use unlabeled palm vein images In an unsupervised way, we combine the inter-modality contrast loss with the content consistency loss to train the generator G that generates palm print images from palm vein images. vp and palmprint discriminator D p , the unlabeled palm vein image Input to G vp Generate a fake palmprint image Then the fake palm print image The palm vein image is sent to the palm print image generator G. pv Generate a fake palm vein image Finally, generate a fake palm print image And real palm print images of different categories Feed D p :

[0030] x′ p =G(x v ) (1)

[0031]

[0032]

[0033] in, Indicates true but Unlabeled palmprint images of different categories; score p for The probability of a real image. If the output is 1, it means that it is 100% a real palm print image, and if the output is 0, it means It cannot be a real palm print image; m is the number of input images; i represents the sequence number of the input image; L content Indicates content loss; L texture Represents texture loss; VGG16 z (·) represents the output of the zth layer of VGG16; z represents the number of layers;

[0034] Sobel(·) means using the Sobel operator to extract texture information; L color Represents color loss; C represents the number of channels of the image;

[0035] H represents the height of the image; W represents the width of the image; hsv(·) represents the conversion of the image from RGB color mode to HSV color mode; L contra Indicates intra-modal contrast loss; the subscript + means: when the value in [...] is greater than 0, the loss is taken as the value, and when it is less than 0, the loss is 0; α is a hyperparameter; represents the L2 norm.

[0036] Step 3.2: Train the generator G that generates palm vein images from palm print images in a semi-supervised manner pv Using a complete palm vein database with labels, we train a cross-modal palm vein generator G in a supervised manner by combining content, texture, color consistency loss and intra-modal contrast loss. v , the palm print image x p Input to G v In the example above, a fake palm vein image x is generated. v ′; Use unlabeled palmprint images In an unsupervised way, we combine the inter-modality contrast loss with the content consistency loss to train the generator G that generates palm vein images from palm print images. pv and palm vein detector D v , the unlabeled palmprint image Input to G pv Generate a fake palm vein image Then the fake palm vein image The palm vein image is sent to the generator G which generates the palm vein image from the palm print image. vp Generate a fake palmprint image Finally, generate a fake palm vein image And the real palm vein images of different categories Feed D v :

[0037] x v ′=G(x p ) (9)

[0038]

[0039]

[0040] in, Indicates true but Unlabeled palm vein images of different categories; score v for The probability of a real image. If the output is 1, it means that it is 100% a real palm vein image, and if the output is 0, it means It cannot be a true palm vein image.

[0041] Go to step 4.

[0042] Step 4: Build and train the adaptive sparse modal fusion model F r ,like Figure 4 As shown in the figure, firstly, VGG16 is combined with the spatiotemporal attention mechanism to extract the features of palm print image and palm vein image to obtain the palm print modal feature f p and palmar vein modal characteristics f v , and then the palmprint modal feature f p and palmar vein modal characteristics f v Adaptive sparse fusion obtains the fusion feature f of palm print and palm vein, and finally uses mean square error loss and cross entropy loss to train the adaptive sparse modal fusion model F r To ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, through this training process, the trained adaptive sparse modal fusion module model F is obtained. r .

[0043] Step 4.1, remove the last layer classifier of the convolutional neural network VGG16 to obtain a new convolutional neural network;

[0044] The new convolutional neural network is connected with the spatiotemporal attention mechanism CSAFM to construct the palmprint feature extraction network F p and palm vein feature extraction network F v , corresponding to the extracted palmprint features f p and palmar vein characteristics v :

[0045] f p =F p (x p ) (17)

[0046] f v =Fv (x v ) (18)

[0047] Among them, x p represents the palm print image, x v Represents a palm vein image.

[0048] Step 4.2: Optimize the parameters of the feature extraction network by single-modal classification learning based on palm print modality and palm vein modality; transform the feature vector f of the palm print image into p and the feature vector f of the palm vein image v , respectively input into the palmprint unimodal classifier C p and palm vein unimodal classifier C v Then, the Sigmoid activation function and the Max function are used to obtain the maximum predicted probability P of the palmprint modality corresponding to the category of the i-th image. 1,i and the maximum prediction probability of palm vein mode P 2,i :

[0049] P p,i =Sigmoid(C p (f p )) (19)

[0050] P v,i =Sigmoid(C v (f v )) (20)

[0051] P 1,i =Max(P p,i ) (twenty one)

[0052] P 2,i =Max(P v,i ) (twenty two)

[0053] Among them, Sigmoid(·) represents the sigmoid activation function, Max(·) represents the maximum value, P p,i and P v,i They are all intermediate variables, which are convenient for simplifying calculations.

[0054] Step 4.3: Take the maximum predicted probability P of the palmprint modality corresponding to the category of the i-th image 1,i and the maximum prediction probability of palm vein mode P 2,i , get the screening probability P of the palm print modality and palm vein modality of the category corresponding to the i-th image i =[P i p ,P i v ]:

[0055]

[0056] Among them, P i p is the screening probability of the palmprint modality of the i-th image, P i v is the screening probability of the palm vein modality of the i-th image.

[0057] Step 4.4: Screening probability P based on palm print modality and palm vein modality i Perform random sampling to obtain the random variable r j =[r j p ,r j v ] T , j = 1, 2, ..., 512:

[0058] r j ~Multinomial(1,p)(25)

[0059] Among them, j is the number of screening features; r j The value of represents a certain category in the input, and only one modal feature is selected each time, which satisfies r j Only one value in is equal to 1, and the rest are equal to 0. T means transpose.

[0060] Step 4.5: Random variable r of palmprint p =[r1 p ,r2 p ,...,r 512 p ] T and the palmprint feature vector f p Perform element-by-element multiplication to filter out the palmprint feature vector f′ p At the same time, the random variable r of the palm vein v =[r1 v ,r2 v ,...,r 512 v ] T and the palm vein feature vector f v Perform element-by-element multiplication to filter out the palm vein feature vector f′ v , and filter out the palmprint feature vector f′ p And filter out the palm vein feature vector f′ v Add them together to get the fusion feature vector f of palm print modality and palm vein modality:

[0061] f′ p =r p ·f p (26)

[0062] f′ v =r v ·f v (27)

[0063] f=f′ p +f′ v (28)

[0064] Step 4.6: Palm print modality and palm vein modality fusion feature vector f, palm print feature vector f p and the palm vein feature vector f v They are sent to the fully connected layer respectively, and then pass through the SoftMax layer to obtain the predicted probability distribution. The cross entropy loss L is used to make the predicted category of the sample equal to the true category of the sample:

[0065] y i =SoftMax(FC(f)) (29)

[0066] y i p =SoftMax(FC(f p )) (30)

[0067] y i v =SoftMax(FC(f v )) (31)

[0068]

[0069] L2=L2_1+L2_2 (35)

[0070] L=L1+α·L2 (36)

[0071] Among them, SoftMax (FC(·)) means that it will pass through the fully connected layer first and then the SoftMax layer; B represents the batch size; α is the balance parameter of the loss function; y i ′ represents the true category of the i-th image; L1, L2_1, L2_2 and L2 are all intermediate variables, which are convenient for simplifying calculations.

[0072] Go to step 5.

[0073] Step 5: Combine the trained palm vein image with the palm print image generator G vp and a generator G that generates a palm print image from a palm vein image pv and adaptive sparse modality fusion model F r , construct the overall palm print and palm vein fusion recognition model F.

[0074] Step 6: Input the palm print image and palm vein image in the test set into the trained overall palm print and palm vein fusion recognition model F, and use the correct recognition rate (CRR) to verify the accuracy of the network, and use the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to verify the quality of the missing modality image generated across modalities.

[0075] Step 6.1: During the test, if both the palm print image and the palm vein image are not missing, they are directly sent to the adaptive sparse modal fusion model F r , for recognition; if the palm print image or palm vein image is missing, the known modality image is used to generate the palm print image through the palm vein image generator G vp Or a generator G that generates a palm print image from a palm vein image pv , generate the missing modality image, and then send the generated missing modality image and the known modality image into the adaptive sparse modality fusion model F r , for identification:

[0076] For both palm vein images and palm vein images are not missing:

[0077] y i =SoftMax{FC[connect(r p ·F P (x p )+r v ·F v (x v ))]} (37)

[0078] Where, connect(·) represents the modality fusion network; where, y i Represents the predicted category of the i-th image;

[0079] For missing palmprint modality:

[0080] y i =SoftMax{FC[connect(r p ·F P (G vp (x v ))+r v ·F v (x v ))]} (38)

[0081] For palmar vein modality loss:

[0082] y i =SoftMax{FC[connect(r p ·F P (x p )+r v ·Fv (G pv (x p )))]} (39)

[0083] Step 6.2: During the generation process, use PSNR and SSIM to evaluate the generator G that generates palm print images from palm vein images. vp and a generator G that generates a palm print image from a palm vein image pv performance;

[0084] In the recognition process, CRR is used to evaluate the adaptive sparse modal fusion model F r Performance:

[0085]

[0086] Among them, MAX I Indicates the maximum value of the color of the image point; MSE represents the mean square error between the real image and the generated image; μ x represents the mean of the real image; μ y represents the mean of the generated image; σ x represents the variance of the real image; σ y represents the variance of the generated image; σ xy represents the covariance between the real image and the generated image; c1 and c2 represent two constants to avoid the denominator being 0.

[0087] Example 1

[0088] The present invention discloses a palm print and palm vein fusion recognition method in missing mode based on cross-modal generation, comprising the following steps:

[0089] Step 1. Create a palm print and palm vein database named TongJi, which contains palm images of 600 volunteers. Each volunteer provided 40 images, representing two different biometric modalities. Therefore, the entire database contains 600 different categories, totaling 12,000 images. In each acquisition, we obtained 20 palm print images and 20 palm vein image samples from each volunteer. The collected original images were then calibrated with coordinate points and cropped to extract the image area of ​​interest. In the experiment, we divided the dataset into a training set and a test set in a 1:1 ratio. The training set contains 6,000 images, and the test set also contains 6,000 images, in which some palm print images and palm vein images are missing.

[0090] Step 2: Construct a generator G that generates palm print images from palm vein images vp , a generator G that generates a palm print image from a palm vein image pv , cross-modal palmprint generator G p, Palmprint Identifier D p Palm vein generator G v and palm vein detector D v .

[0091] Based on the residual blocks in ResNet, we construct the generator G that generates palm print images from palm vein images. vp , a generator G that generates a palm vein image from a palm print image pv , cross-modal palmprint generator G p , Palmprint Identifier D p , cross-modal palm vein generator G v , and palm vein detector D v encoder and decoder.

[0092] Step 3: Train the generator G that generates palm print images from palm vein images in a semi-supervised manner vp and a generator G that generates a palm vein image from a palm print image pv , combining content, texture, color consistency loss and intra-modality contrast loss in a supervised manner to train a supervised cross-modal palmprint generator G p , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm print images from palm vein images vp and palmprint discriminator D p , where G p and G vp The weights are shared and the generator G that generates palm print images from palm vein images is trained in parallel in a semi-supervised manner. vp At the same time, the supervised cross-modal palm vein generator G is trained by combining content, texture, color consistency loss and intra-modal contrast loss in a supervised manner v , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm vein images from palm print images pv and palm vein detector D v , where G v and G pv The weights are shared and the generator G that generates palm vein images from palm print images is trained in parallel in a semi-supervised manner. pv .

[0093] Step 3.1: Palm vein image generates palm print image generator G vp Using a complete palm print and palm vein database with labels, we train a cross-modal palm print generator G in a supervised manner by combining content, texture, color consistency loss and intra-modal contrast loss. p , the palm vein image x v Input to G p In the above example, a fake palmprint image x′ is generated. p; Using unlabeled palm vein images In an unsupervised way, we combine the inter-modality contrast loss with the content consistency loss to train the generator G that generates palm print images from palm vein images. vp and palmprint discriminator D p , the unlabeled palm vein image Input to G vp Generate a fake palmprint image Then the fake palm print image The palm vein image is sent to the palm print image generator G. pv Generate a fake palm vein image Finally, generate a fake palm print image And real palm print images of different categories Feed D p :

[0094] x′ p =G(x v ) (1)

[0095]

[0096]

[0097] in, Indicates true but Unlabeled palmprint images of different categories; score p for The probability of a real image. If the output is 1, it means that it is 100% a real palm print image, and if the output is 0, it means It cannot be a real palm print image; m is the number of input images; i represents the sequence number of the input image; L content Indicates content loss; L texture Represents texture loss; VGG16 z (·) represents the output of the zth layer of VGG16; z represents the number of layers; Sobel (·) represents the use of the Sobel operator to extract texture information; L color represents color loss; C represents the number of channels of the image; H represents the height of the image; W represents the width of the image; hsv(·) represents the conversion of the image from RGB color mode to HSV color mode; L contra Indicates intra-modal contrast loss; the subscript + means: when the value in [...] is greater than 0, the loss is taken as the value, and when it is less than 0, the loss is 0; α is a hyperparameter; represents the L2 norm;

[0098] Step 3.2: Train the generator G that generates palm vein images from palm print images in a semi-supervised manner pvUsing a complete palm vein database with labels, we train a cross-modal palm vein generator G in a supervised manner by combining content, texture, color consistency loss and intra-modal contrast loss. v , the palm print image x p Input to G v In the example above, a fake palm vein image x is generated. v ′; Use unlabeled palmprint images In an unsupervised way, we combine the inter-modality contrast loss with the content consistency loss to train the generator G that generates palm vein images from palm print images. pv and palm vein detector D v , the unlabeled palmprint image Input to G pv Generate a fake palm vein image Then the fake palm vein image The palm vein image is sent to the generator G which generates the palm vein image from the palm print image. vp Generate a fake palmprint image Finally, generate a fake palm vein image And the real palm vein images of different categories Feed D v :

[0099] x v ′=G(x p ) (9)

[0100]

[0101] in, Indicates true but Unlabeled palm vein images of different categories; score v for The probability of a real image. If the output is 1, it means that it is 100% a real palm vein image, and if the output is 0, it means It cannot be a true palm vein image.

[0102] Step 4: Build and train the adaptive sparse modal fusion model F r First, VGG16 combined with the spatiotemporal attention mechanism is used to extract features from palm print images and palm vein images to obtain the palm print modal feature f p and palmar vein modal characteristics f v , and then the palmprint modal feature f p and palmar vein modal characteristics f v Adaptive sparse fusion obtains the fusion feature f of palm print and palm vein, and finally uses mean square error loss and cross entropy loss to train the adaptive sparse modal fusion model F rTo ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, through this training process, the trained adaptive sparse modal fusion module model F is obtained. r .

[0103] Step 4.1, remove the last layer classifier of the convolutional neural network VGG16 to obtain a new convolutional neural network;

[0104] The new convolutional neural network is connected with the spatiotemporal attention mechanism CSAFM to construct the palmprint feature extraction network F p and palm vein feature extraction network F v , corresponding to the extracted palmprint features f p and palmar vein characteristics v :

[0105] f p =F p (x p ) (17)

[0106] f v =F v (x v ) (18)

[0107] Among them, x p represents the palm print image, x v Indicates palm vein image;

[0108] Step 4.2: Optimize the parameters of the feature extraction network by single-modal classification learning based on palm print modality and palm vein modality; transform the feature vector f of the palm print image into p and the feature vector f of the palm vein image v , respectively input into the palmprint unimodal classifier C p and palm vein unimodal classifier C v Then, the Sigmoid activation function and the Max function are used to obtain the maximum predicted probability P of the palmprint modality corresponding to the category of the i-th image. 1,i and the maximum prediction probability of palm vein mode P 2,i :

[0109] P p,i =Sigmoid(C p (f p )) (19)

[0110] P v,i =Sigmoid(C v (f v )) (20)

[0111] P 1,i =Max(P p,i ) (twenty one)

[0112] P 2,i =Max(P v,i ) (twenty two)

[0113] Among them, Sigmoid(·) represents the sigmoid activation function, Max(·) represents the maximum value, P p,i and P v,i They are all intermediate variables, which are convenient for simplifying calculations;

[0114] Step 4.3: Take the maximum predicted probability P of the palmprint modality corresponding to the category of the i-th image 1,i and the maximum prediction probability of palmar vein modality P 2,i , get the screening probability P of the palm print modality and palm vein modality of the category corresponding to the i-th image i =[P i p ,P i v ]:

[0115]

[0116] Among them, P i p is the screening probability of the palmprint modality of the i-th image, P i v is the screening probability of the palm vein modality of the i-th image;

[0117] Step 4.4: Screening probability P based on palm print modality and palm vein modality i Perform random sampling to obtain the random variable r j =[r j p ,r j v ] T , j = 1, 2, ..., 512:

[0118] r j ~Multinomial(1,p)(25)

[0119] Among them, j is the number of screening features; r j The value of represents a certain category in the input, and only one modal feature is selected each time, which satisfies r j Only one value is equal to 1, and the rest are equal to 0. T means transpose;

[0120] Step 4.5: Random variable r of palmprint p =[r1 p ,r2 p ,...,r 512 p ]T and the palmprint feature vector f p Perform element-by-element multiplication to filter out the palmprint feature vector f′ p At the same time, the random variable r of the palm vein v =[r1 v ,r2 v ,...,r 512 v ] T and the palm vein feature vector f v Perform element-by-element multiplication to filter out the palm vein feature vector f′ v , and filter out the palmprint feature vector f′ p And filter out the palm vein feature vector f′ v Add them together to get the fusion feature vector f of palm print modality and palm vein modality:

[0121] f′ p =r p ·f p (26)

[0122] f′ v =r v ·f v (27)

[0123] f=f′ p +f′ v (28)

[0124] Step 4.6: Palm print modality and palm vein modality fusion feature vector f, palm print feature vector f p and the palm vein feature vector f v They are sent to the fully connected layer respectively, and then pass through the SoftMax layer to obtain the predicted probability distribution. The cross entropy loss L makes the predicted category of the sample equal to the true category of the sample:

[0125] y i =SoftMax(FC(f)) (29)

[0126] y i p =SoftMax(FC(f p )) (30)

[0127] y i v =SoftMax(FC(f v )) (31)

[0128]

[0129] L2=L2_1+L2_2 (35)

[0130] L=L1+α·L2 (36)

[0131] Among them, SoftMax (FC(·)) means that it will pass through the fully connected layer first and then the SoftMax layer; B represents the batch size; α is the balance parameter of the loss function; y i ′ represents the true category of the i-th image; L1, L2_1, L2_2 and L2 are all intermediate variables, which are convenient for simplifying calculations.

[0132] Step 5: Combine the trained palm vein image with the palm print image generator G vp and a generator G that generates a palm print image from a palm vein image pv and adaptive sparse modality fusion model F r , construct the overall palm print and palm vein fusion recognition model F.

[0133] Step 6: Input the palm print image and palm vein image in the test set into the trained overall palm print and palm vein fusion recognition model F, and use the correct recognition rate (CRR) to verify the accuracy of the network, and use the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to verify the quality of the missing modality image generated across modalities.

[0134] Step 6.1: During the test, if both the palm print image and the palm vein image are not missing, they are directly sent to the adaptive sparse modal fusion model F r , for recognition; if the palm print image or palm vein image is missing, the known modality image is used to generate the palm print image through the palm vein image generator G vp Or a generator G that generates a palm print image from a palm vein image pv , generate the missing modality image, and then send the generated missing modality image and the known modality image into the adaptive sparse modality fusion model F r , for identification:

[0135] For both palm vein images and palm vein images are not missing:

[0136] y i =SoftMax{FC[connect(r p ·F P (x p )+r v ·F v (x v ))]} (37)

[0137] Where, connect(·) represents the modality fusion network; where, y i Represents the predicted category of the i-th image;

[0138] For missing palmprint modality:

[0139] y i =SoftMax{FC[connect(r p ·F P (G vp (x v ))+r v ·F v (x v ))]} (38)

[0140] For palmar vein modality loss:

[0141] y i =SoftMax{FC[connect(r p ·F P (x p )+r v ·F v (G pv (x p )))]} (39)

[0142] Step 6.2: During the generation process, use PSNR and SSIM to evaluate the generator G that generates palm print images from palm vein images. vp and a generator G that generates a palm print image from a palm vein image pv performance;

[0143] In the recognition process, CRR is used to evaluate the adaptive sparse modal fusion model F r Performance:

[0144]

[0145] Among them, MAX I Indicates the maximum value of the color of the image point; MSE represents the mean square error between the real image and the generated image; μ x represents the mean of the real image; μ y represents the mean of the generated image; σ x represents the variance of the real image; σ y represents the variance of the generated image; σ xy represents the covariance between the real image and the generated image; c1 and c2 represent two constants to avoid the denominator being 0.

[0146] The method of the present invention is to conduct relevant experiments on a network framework built with Python programming language and pytorch framework on an Nvidia 1050 Ti GPU host. We set the batch-size to 8 and use the stochastic gradient descent (SGD) algorithm to optimize the total loss. In order to speed up the network training, we selected an optimization strategy with a momentum of 0.9 and a weight decay of 1e -5 Palmprint feature extraction network F p and palm vein feature extraction network F v (Based on VGG16) Pre-trained on ImageNet with an initial learning rate of 1e -3 In the experiment, we used the Dropout layer to prevent the network from overfitting, and set the Dropout value to 0.5. In this experiment, we considered the two modalities from the same palm as the same category, without any form of expansion of the training data, in order to maximize the test of the true situation of palm prints and palm veins.

[0147] In order to better demonstrate the effect of the proposed algorithm on the completion of missing modal images, this section designs a visualization experiment of cross-modal generation. By visualizing the missing modal images completed by the cross-modal generation model for each existing modality, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to judge the quality of the missing modal images completed by the cross-modal generation model. Figure 5 As shown in the figure, the first column represents the real palm print image, the second column represents the palm vein image generated cross-modally, the third column represents the real palm vein image, and the second column represents the palm vein image generated cross-modally. The average PSNR is 40.24 and the average SSIM is 0.959. The palm print image generated cross-modally by the palm vein image is shown in the figure. Figure 6 As shown in the figure, the first column represents the real palm vein image, the second column represents the cross-modal generated palm print image, the third column represents the real palm print image, and the second column represents the cross-modal generated palm print image. The average PSNR is 34.40 and the average SSIM is 0.881. Visually, both the cross-modal generated palm vein image and the cross-modal generated palm print image are extremely similar to their corresponding real palm vein image and palm print image in structure, color, and texture, and the generated image quality is excellent.

[0148] This experiment is divided into palm print missing or palm vein missing in the test set. The experimental results when the missing rate β (%) is 10, 20, 50, 80, and 100 are shown in Tables 1 and 2, where β = 0% means that the modality is not missing.

[0149] Table 1 Recognition results of missing palm prints

[0150]

[0151] Table 2 Identification results of missing palmar veins

[0152]

[0153] According to the experimental results in Table 1 and Table 2, the method proposed in the present invention can achieve excellent recognition effect regardless of palm print missing or palm vein missing, and the missing rate has little effect on it. Even when a certain mode is 100% missing, it can still achieve good results, and the model has good robustness.

Claims

1. A palm print and palm vein fusion recognition method in missing mode based on cross-modal generation, characterized in that: Including the following steps: Step 1: Collect the palmprint and palmar vein images of n individuals, establish a palmprint and palmar vein image database, divide the above database into a training set and a test set at a ratio of 1:

1. Both the palmprint image and the palmar vein image in the test set are missing, and the missing rate is β. Then go to Step 2; Step 2: Construct a generator G that generates palm print images from palm vein images vp , a generator G that generates a palm vein image from a palm print image pv , cross-modal palmprint generator G p , Palmprint Identifier D p , Palm Vein Generator G v and palm vein detector D v , as follows: Based on the residual blocks in ResNet, we construct the generator G that generates palm print images from palm vein images. vp , a generator G that generates a palm vein image from a palm print image pv , cross-modal palmprint generator G p , Palmprint Identifier D p , cross-modal palm vein generator G v , and palm vein detector D v encoder and decoder; Go to Step 3; Step 3: Train the generator G that generates palm print images from palm vein images in a semi-supervised manner vp and a generator G that generates a palm vein image from a palm print image pv , combining content, texture, color consistency loss and intra-modality contrast loss in a supervised manner to train a supervised cross-modal palmprint generator G p , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm print images from palm vein images vp and palmprint discriminator D p , where G p and G vp Shared weights, the overall semi-supervised way of parallel training of the palm vein image generator G to generate palm print images vp At the same time, the supervised cross-modal palm vein generator G is trained by combining content, texture, color consistency loss and intra-modal contrast loss in a supervised manner v , in an unsupervised way, combining inter-modality contrast loss with content consistency loss, trains the generator G that generates palm vein images from palm print images pv and palm vein detector D v , where G v and G pv The weights are shared and the generator G that generates palm vein images from palm print images is trained in parallel in a semi-supervised manner. pv , as follows: Step 3.1: Palm vein image generates palm print image generator G vp Using a complete palm print and palm vein database with labels, we train a cross-modal palm print generator G in a supervised manner by combining content, texture, color consistency loss and intra-modal contrast loss. p , the palm vein image x v Input to G p In the above example, a fake palmprint image x′ is generated. p ; Using unlabeled palm vein images In an unsupervised way, we combine the inter-modality contrast loss with the content consistency loss to train the generator G that generates palm print images from palm vein images. vp and palmprint discriminator D p , the unlabeled palm vein image Input to G vp Generate a fake palmprint image Then the fake palm print image The palm vein image is sent to the generator G which generates the palm vein image from the palm print image. pv Generate a fake palm vein image Finally, generate a fake palm print image And real palm print images of different categories Feed D p : x′ p =G(x v ) (1) in, Indicates true but Unlabeled palmprint images of different categories; score p for The probability of a real image. If the output is 1, it means that it is 100% a real palm print image, and if the output is 0, it means It cannot be a real palm print image; m is the number of input images; i represents the sequence number of the input image; L content Indicates content loss; L texture Represents texture loss; VGG16 z (·) represents the output of the zth layer of VGG16; z represents the number of layers; Sobel(·) means using the Sobel operator to extract texture information; L color represents color loss; C represents the number of channels of the image; H represents the height of the image; W represents the width of the image; hsv(·) represents the conversion of the image from RGB color mode to HSV color mode; L contra Indicates intra-modal contrast loss; the subscript + means: when the value in [...] is greater than 0, the loss is taken as the value, and when it is less than 0, the loss is 0; α is a hyperparameter; represents the L2 norm; Step 3.2: Train the generator G that generates palm vein images from palm print images in a semi-supervised manner pv Using a complete palm vein database with labels, we train a cross-modal palm vein generator G in a supervised manner by combining content, texture, color consistency loss and intra-modal contrast loss. v , the palm print image x p Input to G v In the example above, a fake palm vein image x is generated. v ′; Use unlabeled palmprint images In an unsupervised way, we combine the inter-modality contrast loss with the content consistency loss to train the generator G that generates palm vein images from palm print images. pv and palm vein detector D v , the unlabeled palmprint image Input to G pv Generate a fake palm vein image Then the fake palm vein image The palm vein image is sent to the generator G which generates the palm vein image from the palm print image. vp Generate a fake palmprint image Finally, generate a fake palm vein image And the real palm vein images of different categories Feed D v : x v ′=G(x p ) (9) in, Indicates true but Unlabeled palm vein images of different categories; score v for The probability of a real image, if the output is 1, it means that it is 100% a real palm vein image, and the output is 0, it means It cannot be a true palm vein image; Go to Step 4; Step 4: Build and train the adaptive sparse modal fusion model F r First, VGG16 combined with the spatiotemporal attention mechanism is used to extract features from palm print images and palm vein images to obtain the palm print modal feature f p and palmar vein modal characteristics f v , and then the palmprint modal feature f p and palmar vein modal characteristics f v Adaptive sparse fusion obtains the fusion feature f of palm print and palm vein, and finally uses mean square error loss and cross entropy loss to train the adaptive sparse modal fusion model F r To ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, through this training process, the trained adaptive sparse modal fusion module model F is obtained. r , to proceed to step 5; Step 5: Combine the trained palm vein image with the palm print image generator G vp and a generator G that generates a palm vein image from a palm print image pv and adaptive sparse modality fusion model F r , construct the overall palm print and palm vein fusion recognition model F, and go to step 6; Step 6: Input the palmprint image and the palmar vein image in the test set into the trained overall palmprint and palmar vein fusion recognition model F, use CRR to verify the accuracy of the network, and use PSNR and SSIM to verify the quality of the missing modality images generated across modalities.

2. According to the palm print and palm vein fusion recognition method in missing mode based on cross-modal generation according to claim 1, it is characterized in that: In Step 1, 200 < n < 500, and the missing rate of the palmprint image and the palmar vein image in the test set is 0 < β < 1.

3. According to the palm print and palm vein fusion recognition method in missing mode based on cross-modal generation according to claim 1, it is characterized in that: In step 4, the adaptive sparse modal fusion model F is constructed and trained. r First, VGG16 combined with the spatiotemporal attention mechanism is used to extract features from palm print images and palm vein images to obtain the palm print modal feature f p and palmar vein modal characteristics f v , and then the palmprint modal feature f p and palmar vein modal characteristics f v Adaptive sparse fusion obtains the fusion feature f of palm print and palm vein, and finally uses mean square error loss and cross entropy loss to train the adaptive sparse modal fusion model F r To ensure that the maximum probability of the classifier is equal to the probability of its corresponding label, through this training process, the trained adaptive sparse modal fusion module model F is obtained. r , as follows: Step 4.1: Remove the last layer classifier of the convolutional neural network VGG16 to obtain a new convolutional neural network; The new convolutional neural network is connected with the spatiotemporal attention mechanism CSAFM to construct the palmprint feature extraction network F p and palm vein feature extraction network F v , corresponding to the extracted palmprint features f p and palmar vein characteristics v : f p =F p (x p ) (17) f v =F v (x v ) (18) Among them, x p represents the palm print image, x v Indicates palm vein image; Step 4.2: Optimize the parameters of the feature extraction network by single-modal classification learning based on palm print modality and palm vein modality; transform the feature vector f of the palm print image into p and the feature vector f of the palm vein image v , respectively input into the palmprint unimodal classifier C p and palm vein unimodal classifier C v Then, the Sigmoid activation function and the Max function are used to obtain the maximum predicted probability P of the palmprint modality corresponding to the category of the i-th image. 1,i and the maximum prediction probability of palmar vein modality P 2,i : P p,i =Sigmoid(C p (f p )) (19) P v,i =Sigmoid(C v (f v )) (20) P 1,i =Max(P p,i ) (21) P 2,i =Max(P v,i ) (22) Among them, Sigmoid(·) represents the sigmoid activation function, Max(·) represents the maximum value, P p,i and P v,i All are intermediate variables; Step 4.3: Take the maximum predicted probability P of the palmprint modality corresponding to the category of the i-th image 1,i and the maximum prediction probability of palmar vein modality P 2,i , get the screening probability P of the palm print modality and palm vein modality of the category corresponding to the i-th image i =[P i p ,P i v ]: Among them, P i p is the screening probability of the palmprint modality of the i-th image, P i v is the screening probability of the palm vein modality of the i-th image; Step 4.4: Screening probability P based on palm print modality and palm vein modality i Perform random sampling to obtain the random variable r j =[r j p ,r j v ] T , j = 1, 2, ..., 512: r j ~Multinomial(1,p) (25) Among them, j is the number of screening features; r j The value of represents a certain category in the input, and only one modal feature is selected each time, which satisfies r j Only one value is equal to 1, and the rest are equal to 0. T means transpose; Step 4.5: Random variable r of palmprint p =[r1 p ,r2 p ,...,r 512 p ] T and the palmprint feature vector f p Perform element-by-element multiplication to filter out the palmprint feature vector f′ p At the same time, the random variable r of the palm vein v =[r1 v ,r2 v ,...,r 512 v ] T and the palm vein feature vector f v Perform element-by-element multiplication to filter out the palm vein feature vector f′ v , and filter out the palmprint feature vector f′ p And filter out the palm vein feature vector f′ v Add them together to get the fusion feature vector f of palm print modality and palm vein modality: f′ p =r p ·f p (26) f′ v =r v ·f v (27) f=f′ p +f′ v (28) Step 4.6: Palm print modality and palm vein modality fusion feature vector f, palm print feature vector f p and the palm vein feature vector f v They are sent to the fully connected layer respectively, and then pass through the SoftMax layer to obtain the predicted probability distribution. The cross entropy loss L makes the predicted category of the sample equal to the true category of the sample: y i =SoftMax(FC(f)) (29) y i p =SoftMax(FC(f p )) (30) y i v =SoftMax(FC(f v )) (31) L2 = L2_1 + L2_2 (35) L = L1 + α·L2 (36) Among them, SoftMax (FC(·)) means that it will pass through the fully connected layer first and then the SoftMax layer; B represents the batch size; α is the balance parameter of the loss function; y i ′ represents the true category of the i-th image; L1, L2_1, L2_2 and L2 are all intermediate variables.

4. According to the palm print and palm vein fusion recognition method in missing mode based on cross-modal generation according to claim 3, it is characterized in that: In Step 6, input the palmprint image and the palmar vein image in the test set into the trained overall palmprint and palmar vein fusion recognition model F, use CRR to verify the accuracy of the network, and use PSNR and SSIM to verify the quality of the missing modality images generated across modalities, specifically as follows: Step 6.1: During the test, if both the palm print image and the palm vein image are not missing, they are directly sent to the adaptive sparse modal fusion model F r , for recognition; if the palm print image or palm vein image is missing, the known modality image is used to generate the palm print image through the palm vein image generator G vp Or a generator G that generates a palm vein image from a palm print image pv , generate the missing modality image, and then send the generated missing modality image and the known modality image into the adaptive sparse modality fusion model F r , for identification: For cases where neither the palmar vein image nor the palmar vein image is missing: y i =SoftMax{FC[connect(r p ·F P (x p )+r v ·F v (x v ))]} (37) Where, connect(·) represents the modality fusion network; where, y i Represents the predicted category of the i-th image; For cases where the palmprint modality is missing: y i =SoftMax{FC[connect(r p ·F P (G vp (x v ))+r v ·F v (x v ))]} (38) For cases where the palmar vein modality is missing: y i =SoftMax{FC[connect(r p ·F P (x p )+r v ·F v (G pv (x p )))]} (39) Step 6.2: During the generation process, use PSNR and SSIM to evaluate the generator G that generates palm print images from palm vein images. vp and a generator G that generates a palm vein image from a palm print image pv performance; In the recognition process, CRR is used to evaluate the adaptive sparse modal fusion model F r Performance: Among them, MAX I Indicates the maximum value of the color of the image point; MSE represents the mean square error between the real image and the generated image; μ x represents the mean of the real image; μ y represents the mean of the generated image; σ x represents the variance of the real image; σ y represents the variance of the generated image; σ xy represents the covariance between the real image and the generated image; c1 and c2 represent two constants to avoid the denominator being 0.

Images