A lightweight dual-modal biometric recognition method based on joint progressive sparsification

Abstract

The application discloses a kind of lightweight bimodal biological feature recognition methods based on joint progressive sparse, mainly includes the following steps: selecting palm print and palm vein image constructs bimodal dataset, using MobileNetV1 as double-branch network feature extraction, and through joint progressive sparse strategy reduces model parameter redundancy, improves computational efficiency;Using double channel attention feature fusion module effectively integrates feature information, forms unified feature representation, constructs lightweight double-branch model;Introduce biological feature structure consistency loss function and detail texture loss function to fine-tune the model, ensure efficient, accurate identity recognition, and through validation set evaluation saves the best model parameters.The application effectively improves the efficiency, accuracy and reliability of identification, while realizing the lightweight design of model, make it more suitable for resource-constrained application scenarios, meet the dual requirements of performance and resources for modern biological feature recognition technology.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of multi-modal biometric recognition, and specifically relates to a lightweight dual-modal biometric recognition method based on joint progressive sparsity. BACKGROUND

[0002] Biometric recognition technology has been widely applied in finance, public security and information security fields by analyzing the physiological or behavioral characteristics of human body to identify individual identity. Common biometric recognition methods include fingerprint, face, iris, palmprint and palm vein recognition, etc. However, single-modal biometric recognition has some limitations in practical application. When the single-modal biometric feature is affected by environmental factors (such as light changes, occlusion, noise, etc.), the recognition performance will be significantly reduced. In addition, a single biometric feature cannot fully reflect the individual identity information, which further affects the accuracy of recognition. Therefore, multi-modal biometric recognition technology has gradually become a research hotspot, which combines multiple biometric feature information (such as palmprint and palm vein) to improve the accuracy and robustness of recognition. Related research shows that the use of multi-modal fusion technology can significantly improve the performance of the recognition system in complex environments. For example, Zhang et al. pointed out in “Multimodal Biometric Recognition Using Palmprint and Palm Vein Features” that the combination of palmprint and palm vein features can improve the performance of the recognition system in complex environments, and the information amount of the fused multi-modal features is much larger than that of the single-modal. In addition, Kumar and Zhang verified in “Hand Geometry, Palm Vein, and Fingerprint Multimodal Biometric Fusion for Identity Recognition” that the fusion of hand geometry features, palm vein and fingerprint can further improve the accuracy and robustness of recognition.

[0003] Although multi-modal biometric recognition technology can greatly improve the recognition accuracy, it is accompanied by an increase in computational complexity and model parameter amount, especially in resource-constrained devices (such as embedded devices, mobile terminals), the high computational load and storage requirement of traditional multi-modal recognition models become a bottleneck. Generally, multi-modal fusion technology relies on complex deep neural network models, which increase parameter redundancy and computational overhead while improving recognition accuracy. Related research shows that with the improvement of model complexity, especially in embedded and mobile devices, traditional multi-modal recognition methods are difficult to balance recognition accuracy and computational resource constraints. For example, Chen et al. pointed out that the power consumption and delay of existing multi-modal recognition systems on mobile devices have become major bottlenecks, and how to improve recognition efficiency is a problem to be solved.

[0004] Based on the above challenges, the present application proposes a lightweight dual-modal biometric recognition method based on joint progressive sparsity. This method not only gradually reduces the redundant parameters of the model through the progressive sparsity strategy, but also jointly sparsifies the dual-modal information (such as palmprint and palm vein), ensuring that the complementary information between the two modalities is fully utilized during the sparsification process, thereby achieving a balance between recognition accuracy and computational efficiency. In addition, the progressive sparsity strategy allows the model to gradually adapt during the sparsification process, ensuring that sparsification does not cause a sharp decline in recognition performance. This method is suitable for resource-constrained devices (such as embedded devices and mobile terminals), reducing power consumption and computational demand while still achieving high-precision and high-robust biometric recognition. As biometric recognition technology is increasingly popular in portable devices, it is necessary to develop lightweight and efficient recognition methods. SUMMARY

[0005] The present application proposes a lightweight dual-modal biometric recognition method based on joint progressive sparsity, which addresses the high computational complexity and large parameter redundancy in resource-constrained devices, improving recognition efficiency and accuracy.

[0006] The technical solution of the present application is a lightweight dual-modal biometric recognition method based on joint progressive sparsity, comprising the following steps:

[0007] Step 1, select palmprint and palm vein images to construct a dual-modal dataset, divide the dual-modal dataset into a training set and a validation set, the training set and the validation set both contain palmprint images and palm vein images, and go to step 2.

[0008] Step 2, use MobileNetV1 as a dual-branch network, i.e. a first branch network and a second branch network, to extract palmprint features and palm vein features, respectively, to capture dual-modal identity feature information, i.e. palmprint feature information and palm vein feature information, and go to step 3.

[0009] Step 3, perform effective sparsification processing on the dual-branch network through a joint progressive sparsity strategy to obtain sparse palmprint feature maps and palm vein feature maps, and go to step 4.

[0010] Step 4, input the sparse palmprint feature maps and palm vein feature maps into a dual-channel attention feature fusion module, integrate the sparse palmprint feature information and palm vein feature information, and form a unified feature representation, thereby constructing a lightweight dual-modal biometric recognition network, and going to step 5.

[0011] Step 5, the lightweight dual-modal biometric feature recognition network is trained by using the training set, and identity recognition cross-entropy loss and modal feature structure consistency loss function and detail texture loss function are introduced to make up for the weak edge texture detail information of the model, optimize the recognition performance of the lightweight dual-modal biometric feature recognition network, and obtain a lightweight dual-modal biometric feature recognition model, and step 6 is entered.

[0012] Step 6, identity verification is performed on the verification set using the lightweight dual-modal biometric feature recognition model, and the final performance of the lightweight dual-modal biometric feature recognition model is evaluated.

[0013] Compared with the prior art, the present application has the following advantages:

[0014] (1) The joint progressive sparsity strategy proposed in the present application is aimed at the problem of efficient operation of the model in resource-limited environments such as embedded devices and mobile terminals, so that the model gradually adapts to sparsity during the training process, fully utilizes the complementary information between the two modalities, and balances the recognition accuracy and computational efficiency.

[0015] (2) The dual attention feature fusion module proposed in the present application effectively fuses palm print and palm vein modal information, fully utilizes the complementarity, improves the accuracy and robustness of recognition in complex environments, introduces the biometric feature structure consistency loss function and the detail texture loss function, ensures the structure consistency and detail fidelity of the fused image, and effectively improves the accuracy of recognition in the detail-sensitive biometric feature recognition task. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of a lightweight dual-modal biometric feature recognition method based on joint progressive sparsity of the present application.

[0017] Figure 2 A general model diagram of a lightweight dual-modal biometric feature recognition method based on joint progressive sparsity of the present application.

[0018] Figure 3 A joint progressive sparsity module diagram of a lightweight dual-modal biometric feature recognition method based on joint progressive sparsity of the present application.

[0019] Figure 4 A fusion module diagram of a lightweight dual-modal biometric feature recognition method based on joint progressive sparsity of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the various embodiments of the present application can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0021] The technical solutions among various embodiments of the present application can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0022] The technical solutions in the various embodiments of the present application can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0023] The technical solutions in the various embodiments of the present application can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application. Figure 1 Figure 2 The technical solutions in the various embodiments of the present application can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0024] Step 1, select palmprint and palm vein image to construct a dual-modal data set, and divide the data set into a training set and a verification set.

[0025] Go to step 2.

[0026] Step 2, use MobileNetV1 as a dual-branch network, i.e. a first branch network and a second branch network, to extract palmprint features and palm vein features respectively, and capture dual-modal identity feature information, i.e. palmprint feature information and palm vein feature information.

[0027] The first branch network and the second branch network are both constructed by using a MobileNetV1 baseline network, as shown below:

[0028] X l1 =X1*F l1 , l = 1,..., L (1)

[0029] X l2 =X2*F l2 , l = 1,..., L (2)

[0030] Wherein, X1 represents a palmprint image input into the first branch network, X2 represents a palm vein image input into the second branch network, F l1 represents the convolution kernel weight of the lth layer of the first branch network, F l2 represents the convolution kernel weight of the lth layer of the second branch network, and l represents the index layer number of the dual-branch network, which is from 1 to L, X l1 ​X l2 Y l denotes the palm vein image feature map of the l-th layer extracted by the second branch network, and denotes the two-dimensional convolution operation.

[0031] Go to step 3.

[0032] Step 3, the double branch network is effectively sparsified by joint progressive sparsification strategy, aiming to reduce model parameter redundancy, improve model calculation efficiency, retain key features, enhance information interaction and complementarity between double modalities, and avoid performance degradation, so as to obtain sparse palm print feature map and palm vein feature map, as follows:

[0033] Step 3.1, the progressive sparsification strategy is constructed for the double branch network to realize dynamic sparsification adjustment mechanism changing with network depth, as shown in formula (3):

[0034]

[0035] Wherein, S l denotes the sparsity of the l-th layer, S1 denotes the initial sparsity of the network, S max denotes the maximum sparsity of the network, and l denotes the index value of the network layer, which is from 1 to L, and L denotes the total number of network sparsification layers;

[0036] Step 3.2, the dense weight matrix of the double branch network is efficiently and accurately sparsified by combining gradient information and weight information, as shown in formula (4) and formula (5):

[0037] IMP l1 = |G l1 |⊙|W l1 | (4)

[0038] IMP l2 = |G l2 |⊙|W l2 | (5)

[0039] Wherein: IMP l1 and IMP l2 respectively denote the importance measure of the dense weight matrix of the l-th layer of the two branch networks, G l1 and G l2 respectively denote the gradient matrix of the dense weight of the l-th layer in the two branch networks, W l1 and W l2 are the weight matrix of the dense weight of the l-th layer in the two branch networks, and is the element-wise multiplication operation, and is the absolute value operation.

[0040] Step 3.3, calculate the joint importance of the dense weight matrix of the double-branch baseline network, generate a joint sparse mask, and accurately sparse the dense weight matrix, specifically as shown in equation (6) and equation (7):

[0041] IMP lu = max(IMP l1 , IMP l2 ) (6)

[0042]

[0043] where max( ) is a maximum value operation on the importance measure of the dense weight matrix of the double-branch network at the same position, IMP lu represents the joint importance measure of the dense weight matrix of the two branch networks, M l represents the joint sparse mask of the l-th layer, Bottom(IMP lu ) represents the minimum IMP lu value about the l-th layer;

[0044] Step 3.4, apply the joint sparse mask to the dense weight matrix of the two branch networks respectively, to obtain the sparse weight matrix of the two branch networks, specifically as shown in equation (8) and equation (9):

[0045] W l1s = W l1 ⊙M l (8)

[0046] W l2s = W l2 ⊙M l (9)

[0047] where W l1 represents the dense weight matrix of the l-th layer of the first branch network, W l1s represents the sparse weight matrix of the l-th layer of the first branch network, W l2 represents the dense weight matrix of the l-th layer of the second branch network, and W l2s represents the sparse weight matrix of the l-th layer of the second branch network.

[0048] Step 3.5, multiply the palmprint and palm vein feature maps output by each layer of the double-branch network with the sparse weight matrix of each layer of the double-branch network, to obtain the feature maps output by each layer of the sparse double-branch network, i.e. the sparse palmprint feature map and palm vein feature map:

[0049]

[0050]

[0051] where W l1sdenotes the sparse weight matrix of the l-th layer of the first branch network, X l1 denotes the palmprint feature map of the l-th layer of the first branch network, X l1s denotes the palmprint feature map output by the l-th layer of the first branch network after sparsification, W l2s denotes the sparse weight matrix of the l-th layer of the second branch network, X l2 denotes the palm vein feature map of the l-th layer of the second branch network, X l2s denotes the palm vein feature map output by the l-th layer of the second branch network after sparsification, is a matrix multiplication operation.

[0052] Go to step 4.

[0053] Step 4, input the sparse palmprint feature map and the palm vein feature map into the dual-channel attention feature fusion module, integrate the sparse palmprint feature information and the palm vein feature information, form a unified feature representation, and thus construct a lightweight dual-modal biometric feature recognition network, as follows:

[0054] Step 4.1, after global average pooling and maximum pooling operations on the sparse palmprint feature map and the palm vein feature map, perform channel dimension splicing operation to obtain the dual-branch spliced feature map, as follows:

[0055] C1 = Concat c (AvgPool(X L1s ), Maxpool(X L1s )) (12)

[0056] C2 = Concat c (AvgPool(X L2s ), Maxpool(X L2s )) (13)

[0057] wherein X L1s denotes the final output palmprint feature map of the sparse first branch network, AvgPool is a global average pooling operation, MaxPool is a maximum pooling operation, X L2s denotes the final output palm vein feature map of the sparse second branch network, Concat c denotes the splicing operation in the channel dimension, C1 denotes the first branch network feature map obtained after splicing, and C2 denotes the second branch network feature map obtained after splicing.

[0058] Step 4.2, perform two-layer deep convolution operation on the dual-branch spliced feature map respectively, add the obtained feature maps to obtain a dual-modal feature map, and normalize it through a sigmoid function to obtain a dual-modal attention map, as follows:

[0059]

[0060]

[0061] F = σ (D1 + D2) (16)

[0062] wherein, represents a two-layer deep convolution operation with a size of 3x3, D1 represents a feature map obtained by the first branch network through the deep convolution operation, D2 represents a feature map obtained by the second branch network through the deep convolution operation, σ represents a sigmoid function normalization operation, and F represents a bimodal attention map obtained by performing the sigmoid function normalization operation on the feature maps of the two branches;

[0063] Step 4.3, element-wise multiplication of the bimodal attention map and the simple fusion map of the two branches to obtain a bimodal attention fusion map, thereby constructing a lightweight bimodal biometric feature recognition network, as shown in formula (17):

[0064] Y = F O (X L1s + X L2s ) (17)

[0065] wherein, X L1s represents the final output palmprint feature map of the first branch network after sparsification, X L2s represents the final output palm vein feature map of the second branch network after sparsification, O represents element-wise multiplication operation, and Y represents the final bimodal attention fusion map.

[0066] Go to step 5.

[0067] Step 5, training the lightweight bimodal biometric feature recognition network using the training set, and introducing an identity recognition cross-entropy loss and a modal feature structure consistency loss function and a detail texture loss function to make up for the weak edge texture detail information of the model, optimizing the recognition performance of the lightweight bimodal biometric feature recognition network, and obtaining a lightweight bimodal biometric feature recognition model, as follows:

[0068]

[0069]

[0070]

[0071] L total = L CE + α1L 2fusion + α2L texture (21)

[0072] wherein, L CEis denoted as identity cross-entropy loss, N represents the number of samples, T represents the number of categories, is the hot encoding of the true label of the nth sample on the category t, is the predicted probability of the model that the nth sample belongs to the category t; L 2fusion is denoted as modal feature structure consistency loss, H represents the height of the feature map, W represents the width of the feature map, represents the pixel value of the feature extraction output feature map of the first branch network belonging to element (i, j), represents the pixel value of the feature extraction output feature map of the second branch network belonging to element (i, j), Y i,j represents the pixel value of the bimodal attention fusion graph belonging to element (i, j); L texture represents the detail texture loss, represents the gradient value of the feature extraction output feature map of the first branch network belonging to element (i, j), represents the gradient value of the feature extraction output feature map of the second branch network belonging to element (i, j), is the gradient value of the modal attention fusion graph belonging to element (i, j); L total represents the total constraint loss function of the model, and represents hyperparameters.

[0073] Go to step 6.

[0074] Step 6, use the optimized model to perform identity verification on the validation set, and evaluate the final performance of the model.

[0075] Embodiment 1

[0076] Combined Figure 1 , Figure 2 The specific steps of the lightweight bimodal biometric feature recognition method according to the application are as follows:

[0077] Step 1, select the CUMT dataset in the field of biometric feature recognition, and construct the dataset of the application. The dataset is composed of palm images of 290 volunteers, each of which provides 20 images, including 10 palmprint images and 10 palm vein images. The entire database contains 290 categories, a total of 5800 images. We pre-process the palmprint and palm vein images, and unify the size of each image to 224x224 pixels. In the experiment, we randomly divide the dataset into training set and test set according to the ratio of 1:1, each part contains 2900 images, to ensure the consistency of data distribution.

[0078] Step 2, MobileNetV1 is used as a double-branch network, i.e., a first branch network and a second branch network, for extracting palmprint features and palm vein features, respectively, to capture dual-modal identity feature information, i.e., palmprint feature information and palm vein feature information. Both the first branch network and the second branch network are constructed using the MobileNetV1 baseline network, as shown below:

[0079] X l1 = X1*F l1 , l = 1, …, L (1)

[0080] X l2 = X2*F l2 , l = 1, …, L (2)

[0081] wherein X1 represents a palmprint image input to the first branch network, X2 represents a palm vein image input to the second branch network, F l1 represents a convolution kernel weight of the lth layer of the first branch network, F l2 represents a convolution kernel weight of the lth layer of the second branch network, l represents an index layer number of the double-branch network, ranging from 1 to L, X l1 represents a palmprint image feature map of the lth layer extracted by the first branch network, X l2 represents a palm vein image feature map of the lth layer extracted by the second branch network, and * represents a two-dimensional convolution operation.

[0082] Step 3, the double-branch network is effectively sparse processed by a joint progressive sparse strategy, aiming to reduce model parameter redundancy, improve model calculation efficiency, retain key features, enhance information interaction and complementarity between dual modalities, and avoid performance degradation, so as to obtain sparse palmprint feature maps and palm vein feature maps, as follows:

[0083] Step 3.1, a progressive sparse strategy is constructed for the double-branch network to realize a dynamic sparse adjustment mechanism varying with network depth, as shown in equation (3):

[0084]

[0085] wherein S l represents a sparsity of the lth layer, S1 represents an initial sparsity of the network, S max represents a maximum sparsity of the network, l represents an index value of the network layer, ranging from 1 to L, and L represents a total number of network sparse layers.

[0086] Step 3.2, the dense weight matrix of the double-branch network is efficiently and accurately sparse in combination with gradient information and weight information, as shown in equations (4) and (5):

[0087] IMP l1= | G l1 | ⊙ | W l1 | (4)

[0088] IMP l2 = | G l2 | ⊙ | W l2 | (5)

[0089] where IMP l1 and IMP l2 denote the importance measure of the l-th layer of the dense weight matrix of the two branch networks, G l1 and G l2 denote the gradient matrix of the l-th layer of the dense weight of the two branch networks, W l1 and W l2 are the weight matrix of the l-th layer of the dense weight of the two branch networks, and is the element-wise multiplication operation, and | | is the absolute value operation.

[0090] Step 3.3, calculate the joint importance of the dense weight matrix of the double-branch baseline network, generate a joint sparse mask, and accurately sparse the dense weight matrix, as shown in formula (6) and formula (7):

[0091] IMP lu = max(IMP l1 , IMP l2 ) (6)

[0092]

[0093] where max() is an operation of taking the maximum value at the same position of the importance measure of the dense weight matrix of the double-branch network, IMP lu denotes the joint importance measure of the dense weight matrix of the two branch networks, M l denotes the joint sparse mask of the l-th layer, and Bottom(IMP lu ) denotes the minimum IMP lu value about the l-th layer.

[0094] Step 3.4, apply the joint sparse mask to the dense weight matrix of the two branch networks respectively to obtain the sparse weight matrix of the two branch networks, as shown in formula (8) and formula (9):

[0095] W l1s = W l1 ⊙ M l (8)

[0096] W l2s = W l2 ⊙ M l (9)

[0097] where Wl1 denotes the dense weight matrix of the lth layer of the first branch network, W l1s denotes the sparse weight matrix of the lth layer of the first branch network, W l2 denotes the dense weight matrix of the lth layer of the second branch network, W l2s denotes the sparse weight matrix of the lth layer of the second branch network.

[0098] Step 3.5, multiply the palm vein feature maps output by each layer of the double-branch network with the sparse weight matrix of each layer of the double-branch network to obtain the feature maps output by each layer of the double-branch network after sparsification, i.e. the sparse palmprint feature maps and palm vein feature maps:

[0099]

[0100]

[0101] wherein W l1s denotes the sparse weight matrix of the lth layer of the first branch network, X l1 denotes the palmprint feature map of the lth layer of the first branch network, X l1s denotes the palmprint feature map output by the lth layer of the first branch network after sparsification, W l2s denotes the sparse weight matrix of the lth layer of the second branch network, X l2 denotes the palm vein feature map of the lth layer of the second branch network, X l2s denotes the palm vein feature map output by the lth layer of the second branch network after sparsification, is a matrix multiplication operation.

[0102] Step 4, input the sparse double-branch network into the dual-channel attention feature fusion module, integrate the sparse palmprint feature information and palm vein feature information, form a unified feature representation, and thus construct a lightweight dual-modal biometric feature recognition network, as follows:

[0103] Step 4.1, perform global average pooling and maximum pooling operations on the sparse palmprint feature maps and palm vein feature maps, and then perform channel dimension concatenation operation to obtain the double-branch concatenated feature maps, as follows:

[0104] C1 = Concat c (AvgPool(X L1s ), Maxpool(X L1s )) (12)

[0105] C2 = Concat c (AvgPool(X L2s ), Maxpool(X L2s )) (13)

[0106] wherein X L1s represents the final output palmprint feature map of the first branch network after sparsification, AvgPool is a global average pooling operation, MaxPool is a max pooling operation, X L2s represents the final output palm vein feature map of the second branch network after sparsification, Concat c represents a concatenation operation in the channel dimension, C1 represents the first branch network feature map obtained after concatenation, and C2 represents the second branch network feature map obtained after concatenation.

[0107] Step 4.2, two layers of deep convolution operations are respectively performed on the feature maps obtained after concatenation of the two branches, the obtained feature maps are added to obtain a dual-modal feature map, and a sigmoid function normalization is performed to obtain a dual-modal attention map, and the specific formula is as follows:

[0108]

[0109]

[0110] F = σ (D1 + D2) (16)

[0111] wherein represents two layers of deep convolution operations with a size of 3x3, D1 represents the feature map obtained by performing deep convolution operation on the first branch network, D2 represents the feature map obtained by performing deep convolution operation on the second branch network, σ represents a sigmoid function normalization operation, and F represents a dual-modal attention map obtained by performing a sigmoid function normalization operation on the feature maps of the two branches.

[0112] Step 4.3, the dual-modal attention map is multiplied element by element with the simple fusion map of the two branches to obtain a dual-modal attention fusion map, thereby constructing a lightweight dual-modal biometric feature recognition network, and the specific formula is as shown in formula (17):

[0113] Y = F ⊙ (X L1s + X L2s ) (17)

[0114] wherein X L1s represents the final output palmprint feature map of the first branch network after sparsification, X L2s represents the final output palm vein feature map of the second branch network after sparsification, and represents an element-by-element multiplication operation. Y represents the final dual-modal attention fusion map obtained.

[0115] Step 5, train the lightweight dual-modal biometric feature recognition network with the training set, and introduce identity recognition cross-entropy loss and modal feature structure consistency loss function and detail texture loss function to make up for the weak edge texture detail information of the model, optimize the recognition performance of the lightweight dual-modal biometric feature recognition network, and obtain a lightweight dual-modal biometric feature recognition model, as follows:

[0116]

[0117]

[0118]

[0119] L total =L CE +α1L 2fusion +α2L texture (21)

[0120] wherein, L CE is identity recognition cross-entropy loss, N represents the number of samples, T represents the number of categories, is the hot encoding of the true label of the nth sample on the category t, is the predicted probability of the model that the nth sample belongs to category t; L 2fusion is modal feature structure consistency loss, H represents the height of the feature map, and W represents the width of the feature map, represents the pixel value of the feature extraction output feature map of the first branch network belonging to element (i,j), represents the pixel value of the feature extraction output feature map of the second branch network belonging to element (i,j), Y i,j represents the pixel value of the modal attention fusion map belonging to element (i,j); L texture represents the detail texture loss, represents the gradient value of the feature extraction output feature map of the first branch network belonging to element (i,j), represents the gradient value of the feature extraction output feature map of the second branch network belonging to element (i,j), is the gradient value of the modal attention fusion map belonging to element (i,j); L total represents the total constraint loss function of the model, and α1 and α2 represent hyperparameters.

[0121] Step 6, use the optimized model to perform identity verification on the verification set to evaluate the final performance of the model.

[0122] In the present application, we use Nvidia 1650Ti GPU platform, combined with Python programming language and PyTorch framework for experiment. The batchsize of network training is 8, using stochastic gradient descent (SGD) algorithm for optimization, momentum (Momentum) is set to 0.9, weight decay is 1e-5. In order to prevent overfitting, the dropout layer is introduced in the model, and the dropout rate is set to 0.5.

[0123] In the experiment, the palm print and palm vein of the same palm are regarded as the same category, and the data is not expanded, so as to truly evaluate the performance of the palm print and the palm vein.

[0124] In the experiment, the model is trained in the training set, and the performance of the model is verified in the verification set, and the experimental results are shown in Table 1.

[0125] Table 1 Experimental results

[0126]

[0127] According to the experimental results shown in Table 1, the present application based on the joint progressive sparse lightweight dual-modal biometric recognition method has achieved excellent performance on the CUMT database. The experiment shows that the recognition accuracy of the method reaches 99.59%, the parameter amount is only 3.66M, and the running time is only 0.009 seconds, which fully demonstrates the significant advantages of the present application in efficiency and lightweight design. Although the model parameter amount and computational complexity are greatly reduced, the present application can still maintain high-precision recognition performance, making it particularly suitable for resource-constrained embedded devices and mobile terminal application scenarios, meeting the task requirements of high computational efficiency and real-time performance.

Claims

1. A lightweight dual-modal biometric recognition method based on joint progressive sparsification, characterized in that, The method comprises the following steps: Step 1, select palm print and palm vein images to construct a dual-mode data set, and divide the dual-mode data set into a training set and a verification set, wherein the training set and the verification set both contain palm print images and palm vein images, and proceed to step 2; Step 2, MobileNetV1 is used as a dual-branch network, i.e. a first branch network and a second branch network, for extracting palm print features and palm vein features respectively, and capturing dual-mode identity feature information, i.e. palm print feature information and palm vein feature information, and proceeding to step 3; Step 3, the dual-branch network is effectively sparse processed by a joint progressive sparsity strategy to obtain sparse palm print feature maps and palm vein feature maps, and the specific process is as follows: Step 3.1, a progressive sparsity strategy is constructed for the dual-branch network to realize a dynamic sparse adjustment mechanism varying with network depth, and the specific process is as shown in formula (3): wherein S l represents the sparsity of the lth layer, S1represents the initial sparsity of the network, S max represents the maximum sparsity of the network, l represents the index value of the network layer, the size is from 1 to L, and L represents the total number of network sparsity layers; Step 3.2, the dense weight matrix of the dual-branch network is efficiently and accurately sparse in combination with gradient information and weight information, and the specific process is as shown in formula (4) and formula (5): IMP l1 = | G l1 | ⊙ | W l1 | (4) IMP l2 = |G l2 |⊙|W l2 | (5) where: IMP l1 and IMP l2 denote the importance measure of the l-th layer of the dense weight matrix of the two branch networks, G l1 and G l2 denote the gradient matrix of the l-th layer of the dense weight of the two branch networks, W l1 and W l2 are the weight matrix of the l-th layer of the dense weight of the two branch networks, is the element-wise multiplication operation, and || is the absolute value operation; Step 3.3, the joint importance of the dense weight matrix of the dual-branch baseline network is calculated to generate a joint sparse mask, and the dense weight matrix is accurately sparse, and the specific process is as shown in formula (6) and formula (7): IMP lu = max(IMP l1 , IMP l2 ) (6) where max() is a max operation taking the importance measures of the dense weight matrices of the two branch networks at the same position, IMP lu denotes the joint importance measure of the dense weight matrices of the two branch networks, M l denotes the joint sparse mask of the l-th layer, Bottom(IMP lu ) denotes the IMP lu value with respect to the l-th layer that is the smallest. Step 3.4, the joint sparse mask is applied to the dense weight matrix of the two branch networks respectively to obtain the sparse weight matrix of the two branch networks, and the specific process is as shown in formula (8) and formula (9): W l1s = W l1 M l (8) W l2s = W l2 ⊙ M l (9) where W l1 denotes the dense weight matrix of the first branch network at layer l, W l1s denotes the sparse weight matrix of the first branch network at layer l, W l2 denotes the dense weight matrix of the second branch network at layer l, W l2s denotes the sparse weight matrix of the second branch network at layer l; Step 3.5, the palm print and palm vein feature maps output by each layer of the dual-branch network are multiplied by the sparse weight matrix of each layer of the dual-branch network to obtain the feature maps output by each layer of the sparse dual-branch network, i.e. the sparse palm print feature maps and palm vein feature maps: wherein, W l1s denotes the sparse weight matrix of the lth layer of the first branch network, X l1 denotes the palmprint feature map of the lth layer of the first branch network, X l1s denotes the palmprint feature map output by the lth layer of the first branch network after sparsification, W l2s denotes the sparse weight matrix of the lth layer of the second branch network, X l2 denotes the palm vein feature map of the lth layer of the second branch network, X l2s denotes the palm vein feature map output by the lth layer of the second branch network after sparsification, is a matrix multiplication operation; proceed to step 4; Step 4, the sparse palm print feature maps and palm vein feature maps are input into a dual-channel attention feature fusion module to integrate the sparse palm print feature information and palm vein feature information to form a unified feature representation, thereby constructing a lightweight dual-mode biological feature recognition network, and proceeding to step 5; Step 5, the training set is used to train the lightweight dual-mode biological feature recognition network, and an identity recognition cross-entropy loss and a modal feature structure consistency loss function and a detail texture loss function are introduced to make up for the weak edge texture detail information of the model, optimize the recognition performance of the lightweight dual-mode biological feature recognition network, and obtain a lightweight dual-mode biological feature recognition model, and proceed to step 6; Step 6, the lightweight dual-mode biological feature recognition model is used to verify the identity of the verification set to evaluate the final performance of the lightweight dual-mode biological feature recognition model.

2. The lightweight dual-modal biometric recognition method based on joint progressive sparsification according to claim 1, characterized in that, In step 2, MobileNetV1 is used as a dual-branch network, i.e. a first branch network and a second branch network, for extracting palm print features and palm vein features respectively, and capturing dual-mode identity feature information, i.e. palm print feature information and palm vein feature information, and the specific process is as follows: Both the first branch network and the second branch network are constructed by using a MobileNetV1 baseline network, and the specific process is as follows: X l1 = X1*F l1 , l = 1,..., L (1) X l2 = X2*F l2 , l = 1,..., L (2) wherein X1 represents the palmprint image input to the first branch network, X2 represents the palm vein image input to the second branch network, F l1 represents the convolution kernel weight of the first branch network, F l2 represents the convolution kernel weight of the second branch network, l represents the index layer number of the double branch network, X l1 represents the palmprint image feature map of the first branch network extracted at the lth layer, X l2 represents the palm vein image feature map of the second branch network extracted at the lth layer, * represents a two-dimensional convolution operation.

3. The lightweight dual-modal biometric recognition method based on joint progressive sparsification according to claim 2, characterized in that, In step 4, the sparse palmprint feature map and the palm vein feature map are input into a dual-channel attention feature fusion module, the sparse palmprint feature information and the palm vein feature information are integrated to form a unified feature representation, and a lightweight dual-modal biometric feature recognition network is constructed, as follows: Step 4.1, after global average pooling and maximum pooling operations are performed on the sparse palmprint feature map and the palm vein feature map, a channel dimension splicing operation is performed to obtain a dual-branch spliced feature map, as follows: C1 = Concat c (AvgPool(X L1s ), Maxpool(X L1s )) (12) C2 = Concat c (AvgPool(X L2s ), Maxpool(X L2s )) (13) wherein X L1s represents the final output palmprint feature map of the first branch network after sparsification, AvgPool is a global average pooling operation, MaxPool is a max pooling operation, X L2s represents the final output palm vein feature map of the second branch network after sparsification, Concat c represents a concatenation operation in the channel dimension, C1 represents the first branch network feature map obtained after concatenation, and C2 represents the second branch network feature map obtained after concatenation. Step 4.2, two layers of deep convolution operations are respectively performed on the dual-branch spliced feature map, the obtained feature maps are added to obtain a dual-modal feature map, and a sigmoid function is used for normalization to obtain a dual-modal attention map, as follows: F = σ (D1 + D2) (16) wherein, represents a two-layer deep convolution operation with a size of 3x3, D1 represents a feature map obtained by performing a deep convolution operation on the first branch network, D2 represents a feature map obtained by performing a deep convolution operation on the second branch network, σ represents a sigmoid function normalization operation, and F represents a bimodal attention map obtained by performing a sigmoid function normalization operation on the feature maps of the two branches. Step 4.3, the dual-modal attention map and the simple fusion map of the two branches are multiplied element by element to obtain a dual-modal attention fusion map, thereby constructing a lightweight dual-modal biometric feature recognition network, as shown in equation (17) Step 5, the training set is used to train the lightweight dual-modal biometric feature recognition network, and identity recognition cross-entropy loss, modal feature structure consistency loss function and detail texture loss function are introduced to make up for the weak edge texture detail information of the model, optimize the recognition performance of the lightweight dual-modal biometric feature recognition network, and obtain a lightweight dual-modal biometric feature recognition model, as follows: Y = F O (X L1s + X L2s ) (17) wherein X L1s represents the final output palmprint feature map of the first branch network after sparsification, X L2s represents the final output palm vein feature map of the second branch network after sparsification, represents the element-wise multiplication operation, and Y represents the final bimodal attention fusion map.

4. The lightweight dual-modal biometric recognition method based on joint progressive sparsification according to claim 3, characterized in that, ​ L total = L CE + α1L 2fusion + α2L texture (21) wherein, L CE represents the identity cross-entropy loss, N represents the number of samples, T represents the number of categories, is the hot encoding of the real label of the nth sample on the category t, is the predicted probability of the model that the nth sample belongs to the category t; L 2fusion represents the modal feature structure consistency loss, H represents the height of the feature map, W represents the width of the feature map, represents the pixel value of the feature extraction output feature map of the first branch network belonging to element (i, j), represents the pixel value of the feature extraction output feature map of the second branch network belonging to element (i, j), Y i,j represents the pixel value of the bimodal attention fusion graph belonging to element (i, j); L texture represents the detail texture loss, represents the gradient value of the feature extraction output feature map of the first branch network belonging to element (i, j), represents the gradient value of the feature extraction output feature map of the second branch network belonging to element (i, j), is the gradient value of the modal attention fusion graph belonging to element (i, j); L total represents the model total constraint loss function, and a1 and a2 represent hyperparameters.

Images