Lightweight bimodal biometric feature recognition method based on joint progressive sparsity

Abstract

The invention discloses a lightweight bimodal biological feature recognition method based on joint progressive sparse, which mainly comprises the following steps: selecting palm print and palm vein images to construct a bimodal data set, adopting MobileNetV1 as a double-branch network to extract features, and reducing model parameter redundancy and improving calculation efficiency through a joint progressive sparse strategy; a dual-channel attention feature fusion module is used for effectively integrating feature information, uniform feature representation is formed, and a lightweight dual-branch model is constructed; a biological feature structure consistency loss function and a detail texture loss function are introduced to carry out fine optimization on the model, efficient and accurate identity recognition is ensured, and optimal model parameters are evaluated and stored through a verification set. The method effectively improves the recognition efficiency, accuracy and reliability, achieves the lightweight design of the model, enables the model to be more suitable for application scenes with limited resources, and meets the dual requirements of the modern biological feature recognition technology for performance and resources.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal biometric recognition, and in particular relates to a lightweight dual-modal biometric recognition method based on joint progressive sparsity. Background Art

[0002] Biometric recognition technology identifies individuals by analyzing the physiological or behavioral characteristics of the human body. It has been widely used in finance, public security, information security and other fields. Common biometric recognition methods include fingerprint, face, iris, palm print and palm vein recognition. However, single-modal biometric recognition has some limitations in practical applications. When a single-modal biometric feature is affected by environmental factors (such as light changes, occlusion, noise, etc.), the recognition performance will be significantly reduced. In addition, a single biometric feature cannot fully reflect the identity information of an individual, which in turn affects the accuracy of recognition. Therefore, multimodal biometric recognition technology has gradually become a research hotspot, which improves the accuracy and robustness of recognition by combining multiple biometric information (such as palm prints and palm veins). Related studies have shown that the use of multimodal fusion technology can significantly improve the performance of the recognition system in complex environments. For example, Zhang et al. pointed out in "Multimodal Biometric Recognition Using Palmprint and Palm Vein Features" that the combination of palm print and palm vein features can improve the performance of the recognition system in complex environments, and the amount of information fused by multimodal features is much greater than that of single modality. In addition, Kumar and Zhang verified in “HandGeometry, Palm Vein, and Fingerprint Multimodal Biometric Fusion for IdentityRecognition” that the accuracy and robustness of recognition can be further improved by fusing palm geometry features, palm veins and fingerprints.

[0003] Although multimodal biometric recognition technology can significantly improve recognition accuracy, it is also accompanied by an increase in computational complexity and model parameters. In particular, in resource-constrained devices (such as embedded devices and mobile terminals), the high computational complexity and storage requirements of traditional multimodal recognition models become bottlenecks. Generally, multimodal fusion technology relies on complex deep neural network models, which increase parameter redundancy and computational overhead while improving recognition accuracy. Related studies have shown that with the increase in model complexity, especially in embedded and mobile devices, traditional multimodal recognition methods are difficult to balance recognition accuracy and computational resource limitations. For example, Chen et al. pointed out that power consumption and latency of existing multimodal recognition systems on mobile devices have become major bottlenecks, and how to improve recognition efficiency is an urgent problem to be solved.

[0004] Based on the above challenges, the present invention proposes a lightweight dual-modal biometric recognition method based on joint progressive sparsification. This method not only gradually reduces the redundant parameters of the model through a progressive sparsification strategy, but also performs joint sparsification processing on the dual-modal information (such as palm prints and palm veins), ensuring that the complementary information between the two modalities is fully utilized during the sparsification process, thereby achieving a balance between recognition accuracy and computational efficiency. In addition, the progressive sparsification strategy enables the model to gradually adapt during the sparsification process, thereby ensuring that the sparsification process does not cause a sharp drop in recognition performance. This method is suitable for resource-constrained devices (such as embedded devices and mobile terminals), and can achieve high-precision and high-robustness biometric recognition while reducing power consumption and computing requirements. With the increasing popularity of biometric recognition technology in portable devices, it has become necessary to develop lightweight and efficient recognition methods. Summary of the invention

[0005] The present invention proposes a lightweight dual-modal biometric recognition method based on joint progressive sparsity, which improves the recognition efficiency and accuracy by addressing the problems of high computational complexity and large parameter redundancy in resource-constrained devices.

[0006] The technical solution to realize the present invention is: a lightweight dual-modal biometric feature recognition method based on joint progressive sparsity, comprising the following steps:

[0007] Step 1: Select palm prints and palm vein images to construct a bimodal dataset, and divide the bimodal dataset into a training set and a validation set. Both the training set and the validation set contain palm print images and palm vein images, and then proceed to step 2.

[0008] Step 2: Use MobileNetV1 as a dual-branch network, i.e., the first branch network and the second branch network, to extract palm print features and palm vein features respectively, to capture dual-modal identity feature information, i.e., palm print feature information and palm vein feature information, and proceed to step 3.

[0009] Step 3: Use a joint progressive sparse strategy to effectively thin out the dual-branch network, obtain the sparse palm print feature map and palm vein feature map, and then go to step 4.

[0010] Step 4: Input the sparse palm print feature map and palm vein feature map into the dual-channel attention feature fusion module, integrate the sparse palm print feature information and palm vein feature information to form a unified feature representation, thereby constructing a lightweight dual-modal biometric recognition network and proceed to step 5.

[0011] Step 5. Use the training set to train the lightweight bimodal biometric recognition network, and introduce the identity recognition cross entropy loss, modal feature structure consistency loss function and detail texture loss function to compensate for the weakened edge texture detail information of the model, optimize the recognition performance of the lightweight bimodal biometric recognition network, obtain the lightweight bimodal biometric recognition model, and proceed to step 6.

[0012] Step 6: Use the lightweight bimodal biometric recognition model to authenticate the validation set and evaluate the final performance of the lightweight bimodal biometric recognition model.

[0013] Compared with the prior art, the present invention has the following significant advantages:

[0014] (1) The joint progressive sparsity strategy proposed in this paper aims to solve the problem of efficient operation of the model in resource-constrained environments such as embedded devices and mobile terminals. It enables the model to gradually adapt to sparsity during the training process, fully utilizes the complementary information between the two modalities, and achieves a balance between recognition accuracy and computational efficiency.

[0015] (2) The present invention proposes a dual attention feature fusion module, which effectively fuses the palm print and palm vein modal information, fully utilizes their complementarity, and improves the accuracy and robustness of recognition in complex environments. The biometric structural consistency loss function and detail texture loss function are introduced to ensure the structural consistency and detail fidelity of the fused image, effectively improving the recognition accuracy in detail-sensitive biometric recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a flowchart of a lightweight dual-modal biometric feature recognition method based on joint progressive sparsity.

[0017] Figure 2 This is an overall model diagram of a lightweight dual-modal biometric recognition method based on joint progressive sparsity of the present invention.

[0018] Figure 3 This is a joint progressive sparse module diagram of a lightweight dual-modal biometric feature recognition method based on joint progressive sparseness of the present invention.

[0019] Figure 4 This is a fusion module diagram of a lightweight dual-modal biometric feature recognition method based on joint progressive sparsity of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0022] The following will further introduce the specific implementation method, as well as the technical difficulties and inventive points of this invention in combination with this design example.

[0023] Combination Figure 1 and Figure 2 The lightweight dual-modal biometric feature recognition method based on joint progressive sparsity of the present invention comprises the following steps:

[0024] Step 1: Select palm prints and palm vein images to construct a dual-modal dataset, and divide the dataset into a training set and a validation set.

[0025] Go to step 2.

[0026] Step 2: Use MobileNetV1 as a dual-branch network, i.e., the first branch network and the second branch network, to extract palm print features and palm vein features respectively, and capture dual-modal identity feature information, i.e., palm print feature information and palm vein feature information.

[0027] The first branch network and the second branch network are both constructed using the MobileNetV1 baseline network, as shown below:

[0028] X l1 =X 1 *F l1 ,l=1,...,L (1)

[0029] X l2 =X 2 *F l2 ,l=1,...,L (2)

[0030] Among them, X 1 represents the palmprint image input by the first branch network, X 2 represents the palm vein image input by the second branch network, F l1 represents the convolution kernel weight of the first branch network layer l, F l2represents the convolution kernel weight of the lth layer of the second branch network, l represents the index layer number of the dual-branch network, ranging from 1 to L, X l1 represents the palmprint image feature map of the first layer extracted by the first branch network, X l2 represents the palm vein image feature map of the lth layer extracted by the second branch network, and * represents a two-dimensional convolution operation.

[0031] Go to step 3.

[0032] Step 3: Effectively sparse the dual-branch network through a joint progressive sparse strategy, aiming to reduce model parameter redundancy, improve the model's computational efficiency, retain key features, enhance information interaction and complementarity between the two modalities, and avoid performance degradation, thereby obtaining the sparse palm print feature map and palm vein feature map, as follows:

[0033] Step 3.1: Construct a progressive sparse strategy for the dual-branch network to implement a dynamic sparse adjustment mechanism that changes with the network depth, as shown in formula (3):

[0034]

[0035] Among them, S l represents the sparsity rate of the lth layer, S 1 represents the initial sparseness rate of the network, S max Indicates the maximum sparseness rate of the network, l represents the index value of the network layer, ranging from 1 to L, and L represents the total number of sparse layers of the network;

[0036] Step 3.2: Combine the gradient information and weight information to efficiently and accurately sparse the dense weight matrix of the dual-branch network, as shown in formula (4) and formula (5):

[0037] IMP l1 =|G l1 |⊙|W l1 | (4)

[0038] IMP l2 =|G l2 |⊙|W l2 | (5)

[0039] Among them: IMP l1 and IMP l2 Represents the importance measure of the lth layer of the dense weight matrix of the two branch networks, G l1 and G l2 Respectively represent the gradient matrices of the dense weights of the lth layer in the two branch networks, W l1 and W l2 is the weight matrix of the dense weights of the lth layer in the two branch networks, ⊙ is the element-by-element product operation, | | is the absolute value operation;

[0040] Step 3.3, calculate the joint importance of the dense weight matrix of the dual-branch baseline network, generate a joint sparse mask, and refine the sparse dense weight matrix, as shown in formula (6) and formula (7):

[0041] IMP lu =max(IMP l1 ,IMP l2 ) (6)

[0042]

[0043] Among them, max() is the importance measure of the dense weight matrix of the two-branch network, taking the maximum value operation at the same position, IMP lu represents the joint importance measure of the dense weight matrices of the two branch networks, M l Represents the joint sparse mask of layer l, Bottom(IMP lu ) represents the smallest IMP on layer l lu value;

[0044] Step 3.4: Apply the joint sparse mask to the dense weight matrices of the two branch networks respectively to obtain the sparse weight matrices of the two branch networks, as shown in equations (8) and (9):

[0045] W l1s =W l1 ⊙M l (8)

[0046] W l2s =W l2 ⊙M l (9)

[0047] Where W l1 represents the dense weight matrix of the first branch network layer l, W l1s represents the sparse weight matrix of the first branch network layer l, W l2 represents the dense weight matrix of the second branch network layer l, W l2s Represents the sparse weight matrix of the lth layer of the second branch network;

[0048] Step 3.5, multiply the palm print and palm vein feature maps output by each layer of the dual-branch network by the sparse weight matrix of each layer of the dual-branch network to obtain the sparse feature maps output by each layer of the dual-branch network, that is, the sparse palm print feature map and palm vein feature map:

[0049]

[0050]

[0051] Among them, W l1s represents the sparse weight matrix of the first branch network layer l, X l1 represents the palmprint feature map of the first branch network layer l, X l1s represents the palmprint feature map output by the lth layer of the first branch network after sparseness, W l2s represents the sparse weight matrix of the second branch network layer l, X l2 represents the palm vein feature map of the lth layer of the second branch network, X l2s represents the palm vein feature map output by the lth layer of the second branch network after sparseness, is a matrix multiplication operation.

[0052] Go to step 4.

[0053] Step 4: Input the sparse palm print feature map and palm vein feature map into the dual-channel attention feature fusion module, integrate the sparse palm print feature information and palm vein feature information to form a unified feature representation, thereby constructing a lightweight dual-modal biometric recognition network, as follows:

[0054] Step 4.1: Perform global average pooling and maximum pooling operations on the sparse palm print feature map and palm vein feature map, and then perform channel dimension splicing operations to obtain a dual-branch spliced ​​feature map, as follows:

[0055] C 1 =Concat c (AvgPool(X L1s ),Maxpool(X L1s )) (12)

[0056] C 2 =Concat c (AvgPool(X L2s ),Maxpool(X L2s )) (13)

[0057] Among them, X L1s represents the final output palmprint feature map of the first branch network after sparseness, AvgPool is the global average pooling operation, MaxPool is the maximum pooling operation, X L2s Represents the final output palm vein feature map of the second branch network after sparseness, Concat c Indicates the concatenation operation in the channel dimension, C 1 represents the first branch network feature map obtained after splicing, C 2 Represents the second branch network feature map obtained after splicing;

[0058] Step 4.2: Perform two layers of deep convolution operations on the feature maps after the two branches are concatenated, add the obtained feature maps to obtain the bimodal feature map, and normalize it with the sigmoid function to obtain the bimodal attention map. The specific formula is as follows:

[0059]

[0060]

[0061] F=σ(D 1 +D 2 ) (16)

[0062] in, Denotes two layers of depthwise convolution operation with a size of 3×3, D 1 Denotes the feature map obtained by the first branch network after deep convolution operation, D 2 represents the feature map obtained by the deep convolution operation of the second branch network, σ represents the sigmoid function normalization operation, and F represents the bimodal attention map obtained after the sigmoid function normalization operation on the feature maps of the two branches;

[0063] Step 4.3: Multiply the bimodal attention map and the simple fusion map of the two branches element by element to obtain the bimodal attention fusion map, thereby constructing a lightweight bimodal biometric recognition network, as shown in formula (17):

[0064] Y=F⊙(X L1s +X L2s ) (17)

[0065] Among them, X L1s represents the final output palmprint feature map of the first branch network after sparseness, X L2s represents the final output palm vein feature map of the second branch network after sparseness, ⊙ represents the element-by-element multiplication operation, and Y represents the final bimodal attention fusion map.

[0066] Go to step 5.

[0067] Step 5: Use the training set to train the lightweight bimodal biometric recognition network, and introduce the identity recognition cross entropy loss, modal feature structure consistency loss function, and detail texture loss function to compensate for the weakened edge texture detail information of the model, optimize the recognition performance of the lightweight bimodal biometric recognition network, and obtain a lightweight bimodal biometric recognition model, as follows:

[0068]

[0069]

[0070]

[0071] L total =L CE +α 1 L 2fusion +α 2 L texture (twenty one)

[0072] Among them, L CE It is expressed as identity recognition cross entropy loss, N represents the number of samples, T represents the number of categories, is the hot encoding of the true label of the nth sample in category t, is the model's predicted probability that the nth sample belongs to category t; L 2fusion It is expressed as the modal feature structure consistency loss, H represents the height of the feature map, W represents the width of the feature map, Represents the pixel value of the feature extraction output feature map of the first branch network belonging to element (i, j), The feature extraction output feature map of the second branch network belongs to the pixel value of element (i, j), Y i,j Indicates the pixel value of the bimodal attention fusion map belonging to element (i, j); L texture Indicates detail texture loss, Indicates the gradient value of the feature extraction output feature map of the first branch network belonging to element (i, j), Indicates the gradient value of the feature extraction output feature map of the second branch network belonging to element (i, j), is the gradient value of element (i, j) in the modal attention fusion graph; L total Represents the total constraint loss function of the model, α 1 and α 2 Represents a hyperparameter.

[0073] Go to step 6.

[0074] Step 6: Use the optimized model to authenticate the validation set and evaluate the final performance of the model.

[0075] Example 1

[0076] Combination Figure 1 , Figure 2 The lightweight dual-modal biometric recognition method with joint progressive sparseness described in the present invention comprises the following specific steps:

[0077] Step 1. Select the CUMT dataset in the field of biometric recognition to construct the dataset of the present invention. The dataset consists of palm images of 290 volunteers. Each volunteer provides 20 images, including 10 palm print images and 10 palm vein images. The entire database contains 290 categories and a total of 5,800 images. We preprocess the palm print and palm vein images and normalize the size of each image to 224×224 pixels. In the experiment, we randomly divide the dataset into a training set and a test set in a 1:1 ratio, and each part contains 2,900 images to ensure the consistency of data distribution.

[0078] Step 2: Use MobileNetV1 as a dual-branch network, namely the first branch network and the second branch network, to extract palm print features and palm vein features respectively, and capture dual-modal identity feature information, namely palm print feature information and palm vein feature information. Both the first branch network and the second branch network are constructed using the MobileNetV1 baseline network, as shown below:

[0079] X l1 =X 1 *F l1 ,l=1,...,L (1)

[0080] X l2 =X 2 *F l2 ,l=1,...,L (2)

[0081] Among them, X 1 represents the palmprint image input by the first branch network, X 2 represents the palm vein image input by the second branch network, F l1 represents the convolution kernel weight of the first branch network layer l, F l2 represents the convolution kernel weight of the lth layer of the second branch network, l represents the index layer number of the dual-branch network, ranging from 1 to L, X l1 represents the palmprint image feature map of the first layer extracted by the first branch network, X l2 represents the palm vein image feature map of the lth layer extracted by the second branch network, and * represents a two-dimensional convolution operation.

[0082] Step 3: Effectively sparse the dual-branch network through a joint progressive sparse strategy, aiming to reduce model parameter redundancy, improve the model's computational efficiency, retain key features, enhance information interaction and complementarity between the two modalities, and avoid performance degradation, thereby obtaining the sparse palm print feature map and palm vein feature map, as follows:

[0083] Step 3.1: Construct a progressive sparse strategy for the dual-branch network to implement a dynamic sparse adjustment mechanism that changes with the network depth, as shown in formula (3):

[0084]

[0085] Among them, S l represents the sparsity rate of the lth layer, S 1 represents the initial sparseness rate of the network, S max Indicates the maximum sparsity rate of the network, l represents the index value of the network layer, ranging from 1 to L, and L represents the total number of sparse layers of the network.

[0086] Step 3.2: Combine the gradient information and weight information to efficiently and accurately sparse the dense weight matrix of the dual-branch network, as shown in formula (4) and formula (5):

[0087] IMP l1 =|G l1 |⊙|W l1 | (4)

[0088] IMP l2 =|G l2 |⊙|W l2 | (5)

[0089] Among them: IMP l1 and IMP l2 Represents the importance measure of the lth layer of the dense weight matrix of the two branch networks, G l1 and G l2 Respectively represent the gradient matrices of the dense weights of the lth layer in the two branch networks, W l1 and W l2 is the weight matrix of the dense weights of the lth layer in the two branch networks, ⊙ is the element-by-element product operation, and | | is the absolute value operation.

[0090] Step 3.3, calculate the joint importance of the dense weight matrix of the dual-branch baseline network, generate a joint sparse mask, and refine the sparse dense weight matrix, as shown in formula (6) and formula (7):

[0091] IMP lu =max(IMP l1 ,IMP l2 ) (6)

[0092]

[0093] Among them, max() is the importance measure of the dense weight matrix of the two-branch network, taking the maximum value operation at the same position, IMP lu represents the joint importance measure of the dense weight matrices of the two branch networks, M l Represents the joint sparse mask of layer l, Bottom(IMP lu) represents the smallest IMP on layer l lu value.

[0094] Step 3.4: Apply the joint sparse mask to the dense weight matrices of the two branch networks respectively to obtain the sparse weight matrices of the two branch networks, as shown in equations (8) and (9):

[0095] W l1s =W l1 ⊙M l (8)

[0096] W l2s =W l2 ⊙M l (9)

[0097] Where W l1 represents the dense weight matrix of the first branch network layer l, W l1s represents the sparse weight matrix of the first branch network layer l, W l2 represents the dense weight matrix of the second branch network layer l, W l2s Represents the sparse weight matrix of the lth layer of the second branch network.

[0098] Step 3.5, multiply the palm print and palm vein feature maps output by each layer of the dual-branch network by the sparse weight matrix of each layer of the dual-branch network to obtain the sparse feature maps output by each layer of the dual-branch network, that is, the sparse palm print feature map and palm vein feature map:

[0099]

[0100]

[0101] Among them, W l1s represents the sparse weight matrix of the first branch network layer l, X l1 represents the palmprint feature map of the first branch network layer l, X l1s represents the palmprint feature map output by the lth layer of the first branch network after sparseness, W l2s represents the sparse weight matrix of the second branch network layer l, X l2 represents the palm vein feature map of the lth layer of the second branch network, X l2s represents the palm vein feature map output by the lth layer of the second branch network after sparseness, is a matrix multiplication operation.

[0102] Step 4: Input the sparse dual-branch network into the dual-channel attention feature fusion module to integrate the sparse palm print feature information and palm vein feature information to form a unified feature representation, thereby constructing a lightweight dual-modal biometric recognition network, as follows:

[0103] Step 4.1: Perform global average pooling and maximum pooling operations on the sparse palm print feature map and palm vein feature map, and then perform channel dimension splicing operations to obtain a dual-branch spliced ​​feature map, as follows:

[0104] C 1 =Concat c (AvgPool(X L1s ),Maxpool(X L1s )) (12)

[0105] C 2 =Concat c (AvgPool(X L2s ),Maxpool(X L2s )) (13)

[0106] Among them, X L1s represents the final output palmprint feature map of the first branch network after sparseness, AvgPool is the global average pooling operation, MaxPool is the maximum pooling operation, X L2s Represents the final output palm vein feature map of the second branch network after sparseness, Concat c Indicates the concatenation operation in the channel dimension, C 1 represents the first branch network feature map obtained after splicing, C 2 Represents the second branch network feature map obtained after splicing.

[0107] Step 4.2: Perform two layers of deep convolution operations on the feature maps after the two branches are concatenated, add the obtained feature maps to obtain the bimodal feature map, and normalize it with the sigmoid function to obtain the bimodal attention map. The specific formula is as follows:

[0108]

[0109]

[0110] F=σ(D 1 +D 2 ) (16)

[0111] in, Denotes two layers of depthwise convolution operation with a size of 3×3, D 1 Denotes the feature map obtained by the first branch network after deep convolution operation, D 2 It represents the feature map obtained by the deep convolution operation of the second branch network, σ represents the sigmoid function normalization operation, and F represents the bimodal attention map obtained after the feature maps of the two branches are normalized by the sigmoid function.

[0112] Step 4.3: Multiply the bimodal attention map and the simple fusion map of the two branches element by element to obtain the bimodal attention fusion map, thereby constructing a lightweight bimodal biometric recognition network, as shown in formula (17):

[0113] Y=F⊙(X L1s +X L2s ) (17)

[0114] Among them, X L1s represents the final output palmprint feature map of the first branch network after sparseness, X L2s represents the final output palm vein feature map of the second branch network after sparseness, ⊙ represents the element-by-element multiplication operation, and Y represents the final bimodal attention fusion map.

[0115] Step 5: Use the training set to train the lightweight bimodal biometric recognition network, and introduce the identity recognition cross entropy loss, modal feature structure consistency loss function, and detail texture loss function to compensate for the weakened edge texture detail information of the model, optimize the recognition performance of the lightweight bimodal biometric recognition network, and obtain a lightweight bimodal biometric recognition model, as follows:

[0116]

[0117]

[0118]

[0119] L total =L CE +α 1 L 2fusion +α 2 L texture (twenty one)

[0120] Among them, L CE It is expressed as identity recognition cross entropy loss, N represents the number of samples, T represents the number of categories, is the hot encoding of the true label of the nth sample in category t, is the model's predicted probability that the nth sample belongs to category t; L 2fusion It is expressed as the modal feature structure consistency loss, H represents the height of the feature map, W represents the width of the feature map, Represents the pixel value of the feature extraction output feature map of the first branch network belonging to element (i, j), The feature extraction output feature map of the second branch network belongs to the pixel value of element (i, j), Y i,j Indicates the pixel value of the bimodal attention fusion map belonging to element (i, j); L textureIndicates detail texture loss, Indicates the gradient value of the feature extraction output feature map of the first branch network belonging to element (i, j), Indicates the gradient value of the feature extraction output feature map of the second branch network belonging to element (i, j), is the gradient value of element (i, j) in the modal attention fusion graph; L total Represents the total constraint loss function of the model, α 1 and α 2 Represents a hyperparameter.

[0121] Step 6: Use the optimized model to authenticate the validation set and evaluate the final performance of the model.

[0122] In this paper, we use Nvidia 1650 Ti GPU platform, combined with Python programming language and PyTorch framework for experiments. The batch size of network training is 8, and the stochastic gradient descent (SGD) algorithm is used for optimization, with momentum set to 0.9 and weight decay to 1e-5. To prevent overfitting, a dropout layer is introduced into the model, and the dropout rate is set to 0.5.

[0123] In the experiment, the palm prints and palm veins of the same palm were considered as the same category, and the data was not expanded to truly evaluate the performance of palm prints and palm veins.

[0124] In this experiment, the model is trained in the training set and the model performance is verified in the validation set. The experimental results are shown in Table 1.

[0125] Table 1 Experimental results

[0126]

[0127] According to the experimental results in Table 1, the lightweight dual-modal biometric recognition method based on joint progressive sparsity of the present invention has achieved excellent performance on the CUMT database. Experiments show that the recognition accuracy of this method reaches 99.59%, the number of parameters is only 3.66M, and the running time is only 0.009 seconds, which fully demonstrates the significant advantages of the present invention in terms of high efficiency and lightweight design. Although the number of model parameters and computational complexity are greatly reduced, the present invention can still maintain high-precision recognition performance, making it particularly suitable for resource-constrained embedded devices and mobile terminal application scenarios, and meeting the task requirements with high computing efficiency and real-time requirements.

Claims

1. A lightweight dual-modal biometric recognition method based on joint progressive sparsity, characterized in that: The following steps are involved: Step 1: Select palm prints and palm vein images to construct a dual-modal dataset, and divide the dual-modal dataset into a training set and a validation set. Both the training set and the validation set contain palm print images and palm vein images, and then proceed to step 2. Step 2: Use MobileNetV1 as a dual-branch network, i.e., the first branch network and the second branch network, to extract palm print features and palm vein features respectively, capture dual-modal identity feature information, i.e., palm print feature information and palm vein feature information, and proceed to step 3; Step 3: effectively thin out the dual-branch network by using a joint progressive thinning strategy to obtain a thinned palm print feature map and a palm vein feature map, and then proceed to step 4; Step 4: Input the sparse palm print feature map and palm vein feature map into the dual-channel attention feature fusion module, integrate the sparse palm print feature information and palm vein feature information to form a unified feature representation, thereby constructing a lightweight dual-modal biometric feature recognition network, and proceed to step 5; Step 5: Use the training set to train the lightweight bimodal biometric recognition network, and introduce the identity recognition cross entropy loss, modal feature structure consistency loss function, and detail texture loss function to compensate for the weakened edge texture detail information of the model, optimize the recognition performance of the lightweight bimodal biometric recognition network, obtain the lightweight bimodal biometric recognition model, and proceed to step 6; Step 6: Use the lightweight bimodal biometric recognition model to authenticate the validation set and evaluate the final performance of the lightweight bimodal biometric recognition model.

2. The lightweight dual-modal biometric feature recognition method based on joint progressive sparsity according to claim 1, characterized in that: In step 2, MobileNetV1 is used as a dual-branch network, namely the first branch network and the second branch network, which are used to extract palm print features and palm vein features respectively, and capture dual-modal identity feature information, namely palm print feature information and palm vein feature information, as follows: The first branch network and the second branch network are both constructed using the MobileNetV1 baseline network, as shown below: X l1 =X1*F l1 ,l=1,...,L (1) X l2 =X2*F l2 ,l=1,...,L (2) Where X1 represents the palm print image input by the first branch network, X2 represents the palm vein image input by the second branch network, and F l1 represents the convolution kernel weight of the first branch network layer l, F l2 represents the convolution kernel weight of the lth layer of the second branch network, l represents the index layer number of the dual-branch network, X l1 represents the palmprint image feature map of the first layer extracted by the first branch network, X l2 represents the palm vein image feature map of the lth layer extracted by the second branch network, and * represents a two-dimensional convolution operation.

3. The lightweight dual-modal biometric feature recognition method based on joint progressive sparsity according to claim 2, characterized in that: In step 3, the dual-branch network is effectively thinned by a joint progressive thinning strategy to obtain the thinned palm print feature map and palm vein feature map, as follows: Step 3.1: Construct a progressive sparse strategy for the dual-branch network to implement a dynamic sparse adjustment mechanism that changes with the network depth, as shown in formula (3): Among them, S l represents the sparseness rate of the lth layer, S1 represents the initial sparseness rate of the network, S max Indicates the maximum sparseness rate of the network, l represents the index value of the network layer, ranging from 1 to L, and L represents the total number of sparse layers of the network; Step 3.2: Combine the gradient information and weight information to efficiently and accurately sparse the dense weight matrix of the dual-branch network, as shown in formula (4) and formula (5): IMP l1 =|G l1 |⊙|W l1 | (4) IMP l2 =|G l2 |⊙|W l2 | (5) Among them: IMP l1 and IMP l2 Represents the importance measure of the lth layer of the dense weight matrix of the two branch networks, G l1 and G l2 Respectively represent the gradient matrices of the dense weights of the lth layer in the two branch networks, W l1 and W l2 is the weight matrix of the dense weights of the lth layer in the two branch networks, ⊙ is the element-by-element product operation, | | is the absolute value operation; Step 3.3, calculate the joint importance of the dense weight matrix of the dual-branch baseline network, generate a joint sparse mask, and refine the sparse dense weight matrix, as shown in formula (6) and formula (7): IMP lu =max(IMP l1 ,IMP l2 ) (6) Among them, max() is the importance measure of the dense weight matrix of the two-branch network, taking the maximum value operation at the same position, IMP lu represents the joint importance measure of the dense weight matrices of the two branch networks, M l Represents the joint sparse mask of layer l, Bottom(IMP lu ) represents the smallest IMP on layer l lu value; Step 3.4: Apply the joint sparse mask to the dense weight matrices of the two branch networks respectively to obtain the sparse weight matrices of the two branch networks, as shown in equations (8) and (9): IN l1s =In l1 ⊙M l (8) IN l2s =In l2 ⊙M l (9) Where W l1 represents the dense weight matrix of the first branch network layer l, W l1s represents the sparse weight matrix of the first branch network layer l, W l2 represents the dense weight matrix of the second branch network layer l, W l2s Represents the sparse weight matrix of the lth layer of the second branch network; Step 3.5, multiply the palm print and palm vein feature maps output by each layer of the dual-branch network by the sparse weight matrix of each layer of the dual-branch network to obtain the sparse feature maps output by each layer of the dual-branch network, that is, the sparse palm print feature map and palm vein feature map: Among them, W l1s represents the sparse weight matrix of the first branch network layer l, X l1 represents the palmprint feature map of the first branch network layer l, X l1s represents the palmprint feature map output by the lth layer of the first branch network after sparseness, W l2s represents the sparse weight matrix of the second branch network layer l, X l2 represents the palm vein feature map of the lth layer of the second branch network, X l2s represents the palm vein feature map output by the lth layer of the second branch network after sparseness, is a matrix multiplication operation.

4. The lightweight dual-modal biometric feature recognition method based on joint progressive sparsity according to claim 3, characterized in that: In step 4, the sparse palm print feature map and the palm vein feature map are input into the dual channel attention feature fusion module to integrate the sparse palm print feature information and the palm vein feature information to form a unified feature representation, thereby constructing a lightweight dual-modal biometric recognition network, as follows: Step 4.1: Perform global average pooling and maximum pooling operations on the sparse palm print feature map and palm vein feature map, and then perform channel dimension splicing operations to obtain a dual-branch spliced ​​feature map, as follows: C1=Concat c (AvgPool(X L1s ),Maxpool(X L1s )) (12) C2=Concat c (AvgPool(X L2s ),Maxpool(X L2s )) (13) Among them, X L1s represents the final output palmprint feature map of the first branch network after sparseness, AvgPool is the global average pooling operation, MaxPool is the maximum pooling operation, X L2s Represents the final output palm vein feature map of the second branch network after sparseness, Concat c Indicates a splicing operation in the channel dimension, C1 represents the first branch network feature map obtained after splicing, and C2 represents the second branch network feature map obtained after splicing; Step 4.2: Perform two layers of deep convolution operations on the feature maps after the two branches are concatenated, add the obtained feature maps to obtain the bimodal feature map, and normalize it with the sigmoid function to obtain the bimodal attention map. The specific formula is as follows: F=σ(D1+D2) (16) in, represents a two-layer deep convolution operation of size 3×3, D1 represents the feature map obtained by the first branch network after the deep convolution operation, D2 represents the feature map obtained by the second branch network after the deep convolution operation, σ represents the sigmoid function normalization operation, and F represents the bimodal attention map obtained after the feature maps of the two branches are normalized by the sigmoid function; Step 4.3: Multiply the bimodal attention map and the simple fusion map of the two branches element by element to obtain the bimodal attention fusion map, thereby constructing a lightweight bimodal biometric recognition network, as shown in formula (17): As shown: Y=F⊙(X L1s +X L2s ) (17) Among them, X L1s represents the final output palmprint feature map of the first branch network after sparseness, X L2s represents the final output palm vein feature map of the second branch network after sparseness, ⊙ represents the element-by-element multiplication operation, and Y represents the final bimodal attention fusion map.

5. The lightweight dual-modal biometric feature recognition method based on joint progressive sparsity according to claim 4, characterized in that: In step 5, the lightweight bimodal biometric recognition network is trained using the training set, and the identity recognition cross entropy loss, modal feature structure consistency loss function, and detail texture loss function are introduced to compensate for the weakened edge texture detail information of the model, optimize the recognition performance of the lightweight bimodal biometric recognition network, and obtain a lightweight bimodal biometric recognition model, as follows: L total =L CE +α1L 2fusion +α2L texture (21) Among them, L CE It is expressed as identity recognition cross entropy loss, N represents the number of samples, T represents the number of categories, is the hot encoding of the true label of the nth sample in category t, is the model's predicted probability that the nth sample belongs to category t; L 2fusion It is expressed as the modal feature structure consistency loss, H represents the height of the feature map, W represents the width of the feature map, Represents the pixel value of the feature extraction output feature map of the first branch network belonging to element (i, j), The feature extraction output feature map of the second branch network belongs to the pixel value of element (i, j), Y i,j Indicates the pixel value of the bimodal attention fusion map belonging to element (i, j); L texture Indicates detail texture loss, Indicates the gradient value of the feature extraction output feature map of the first branch network belonging to element (i, j), Indicates the gradient value of the feature extraction output feature map of the second branch network belonging to element (i, j), is the gradient value of element (i, j) in the modal attention fusion graph; L total represents the total constraint loss function of the model, and α1 and α2 represent hyperparameters.

Images