A multi-modal biometric recognition method based on an attention module

Abstract

The application relates to a multi-modal biological recognition method based on an attention module and belongs to the field of computer biological recognition. The method comprises a data acquisition and preprocessing stage, a neural network model training and verification stage, an attention module feature fusion stage and a deployment application stage. The method has the advantages that multi-modal biological features are used for identity recognition, the problem of insufficient biological features on a single mode and the problem of insecure single-modal biological features are solved, mutual influence is carried out between the multi-modal modes, more comprehensive and effective biological features are helped to be extracted, the arcloss function is used as a loss function in the extraction method, compared with the softmax loss loss function, the model recognition rate is improved, the attention mechanism is used in the feature layer fusion process, better feature fusion performance is achieved, and compared with the traditional method of learning a weight for the feature of each mode and then performing weighted concatenation on the features, the features of each mode can better play a role.

Description

Technical Field

[0001] This invention belongs to the field of computer biometrics, and particularly relates to a multimodal biometric method based on an attention module. Background Technology

[0002] Currently, biometric technology primarily relies on a single modality for system identification. However, due to external factors and individual differences, using single-modal biometrics for identification inevitably has limitations. For example, factors such as lighting can affect the accuracy of facial recognition; fingerprints can be worn down by prolonged work or affected by sweaty fingers. Furthermore, the leakage of single-modal biometric data can permanently damage user privacy, while multimodal biometric technology can significantly enhance identification security. Therefore, multimodal biometric recognition technology has become a new research direction and is developing rapidly. Multimodal recognition schemes fuse information based on the characteristics of different features to achieve complementary advantages. Multimodal systems offer superior robustness and security, providing a better option for identity authentication. As early as 1995, Brunelli et al. proposed using facial and voice feature information fusion for identity recognition, performing feature fusion at the decision-making level to demonstrate the feasibility and effectiveness of multimodal recognition systems. In 2013, Shekhar et al. used a sparse linear combination method to represent face, iris, and fingerprint features, discussing the correlation and coupling between different features during the fusion process. In 2016, Wild et al. investigated the dynamic detection and suppression of outliers in the liveness detection of spoofed samples under a multimodal face and fingerprint fusion structure, and studied its inherent normalization.

[0003] Existing multimodal biometrics mainly focus on faces, fingerprints, and voiceprints, with limited research on finger multimodal biometrics. Fingers possess rich feature information, such as fingerprints, knuckle patterns, and finger veins, and are easy to collect, making them suitable as a research subject for multimodal feature fusion. Fingerprint recognition technology is the earliest developed, has the most mature algorithms, and is the most widely used in the market. Fingerprint recognition technology utilizes the unique combination of ridges and valleys on the skin at the fingertip for identification. Traditional fingerprint recognition methods use the Iterative Closest Point (ICP) algorithm to achieve fingerprint image registration and stitching. Sequence image stitching methods can be used to calculate the offset of the sliding direction even when the image slides at non-uniform speeds. In learning-based methods, the variability between global orientation patterns is learned through a training set, and the most suitable pattern from the training set is used to replace the fingerprint region, effectively addressing the influence of noise.

[0004] Finger veins, being subcutaneous tissue, offer advantages over other biometric identification methods, including high anti-spoofing capabilities, high recognition rates, and user-friendliness. Current finger vein enhancement algorithms are based on Gabor filters, scattering models, and adaptive curve transforms, combining Gabor filters with Weber's theorem to address uneven illumination in finger vein images. Feature descriptors are used to extract palm prints and finger veins, with SIFT and SURF algorithms respectively used to extract features from finger vein segments. Finally, a support vector machine is used to classify the feature vectors formed by the extracted features.

[0005] Knuckle prints refer to the flexural lines or textured areas located on the knuckles of the fingers, possessing rich textural and linear features. Knuckle print images can be acquired using low-resolution cameras, making acquisition convenient and inexpensive. These advantages make knuckle prints a desirable biometric feature, and knuckle print-based recognition technologies have developed rapidly in recent years. A knuckle print recognition algorithm based on a multi-channel Gabor filter achieves good recognition results by using principal component analysis (PCA) and linear discriminant analysis (LDA) to reduce the dimensionality of the extracted features.

[0006] Biometric data acquisition is a crucial step in multimodal biometric identification. Biometric data acquisition is a pattern recognition system. First, the system acquires an individual's biometric data, then extracts feature sets from the data. By comparing two different sets of biometric features, the system can determine whether they belong to the same person. Therefore, an automated biometric identification system can typically be considered to consist of the following modules: 1) an acquisition module; 2) a feature extraction module; and 3) a matching module. Additionally, the system must include a database for storing and managing the feature sets.

[0007] The acquisition of multimodal biometric features has been a subject of academic research in recent years. In 2016, Yang Jinfeng et al. designed a multimodal biometric acquisition system encompassing finger veins, knuckle prints, and fingerprints. In 2019, they further designed a novel dual-modal finger image acquisition system, integrating ROI localization algorithms for finger veins and knuckle prints. The currently researched multimodal biometric acquisition system can be applied to smart locks.

[0008] Existing multimodal biometric methods mainly use decision-level fusion methods. Feature-level fusion methods have been less studied due to the large amount of data and training required, as well as the significant differences between multimodal features. Furthermore, finger veins are an emerging finger feature with limited feature extraction methods. How to extract effective features and perform multimodal feature fusion is a challenge.

[0009] Existing multimodal fusion methods include direct feature splicing recognition and correlation analysis. However, these methods have significant limitations when faced with more complex feature relationships. Finding a more effective feature fusion method is a challenge for multimodal biometrics. Summary of the Invention

[0010] This invention provides a multimodal biometric identification method based on an attention module to address the problems of insufficient biometric features in a single modality and insecurity of single-modal biometric features in identity recognition.

[0011] The technical solution adopted by this invention includes the following steps:

[0012] (1) Data collection: Using a biometric collector, collect biometric features of three modalities: fingerprint, finger vein, and knuckle print.

[0013] (2) Data preprocessing: The collected biometric data of the three modalities are preprocessed to meet the input requirements of the corresponding models of the three modalities.

[0014] (3) In the neural network model training and verification stage, the data of multiple modalities after preprocessing in step (2) are used to extract features through ResNet and DenseNet. The parameters of ResNet and DenseNet are obtained through training and verification.

[0015] (4) Feature fusion of the attention module: The feature vectors of the three modalities are extracted and concatenated to form a total feature vector, which is then sent to the classifier for feature classification. The attention mechanism is adopted. First, the Q, K, and V matrices are initialized according to the feature vector formed by the concatenation of the three modalities. K is the key value in the calculation of the attention mechanism, Q represents the query, and K is the key value. Then, according to the initialized Q, K, and V, the optimal weight is calculated by training and learning, and the joint feature vector is obtained and sent to the classifier for classification, thereby achieving a better recognition effect.

[0016] (5) Deployment and application: The trained algorithm is deployed and applied to the smart lock. Visitors provide the required biometric features, and the smart lock identifies whether they are valid users based on the data in the database.

[0017] The data acquisition in step (1) of the present invention includes: using a biometric collector to collect the user's fingerprint image dataset A, finger vein image dataset B, and knuckle print image dataset C;

[0018] The biometric collector is the biometric collector module of the smart lock. The smart lock includes a motherboard, clutch, biometric collector, memory, and microprocessor.

[0019] The data preprocessing in step (2) of this invention involves preprocessing the collected biological image datasets of three modalities. The preprocessing steps are as follows:

[0020] 1) Automatically label the collected biological images to obtain the volunteer number and finger number corresponding to each biological image, and then divide the biological images into training set and test set according to the ratio;

[0021] 2) Perform image preprocessing on the acquired biological images by subtracting the average value of each image pixel from the biological image, using standard color enhancement, and then batch normalizing;

[0022] 3) Image enhancement: The Sobel operator is used to enhance the input fingerprint image, finger vein image, and knuckle print image respectively. Then, ROI localization and cropping are used to segment the effective fingerprint, finger vein, and knuckle print datasets and training sets. The effective fingerprint training set is A1, the effective fingerprint test set is A2, the effective finger vein training set is B1, the effective finger vein test set is B1, the effective knuckle print training set is C1, and the effective knuckle print test set is C2.

[0023] The specific steps in step (3) of this invention are as follows:

[0024] 1) During the model training phase, features of the multiple modalities preprocessed in step (2) are extracted using ResNet and DenseNet. The fingerprint training set A1 and the finger vein training set B1 are input into the ResNet34 and ResNet50 networks for independent training, respectively.

[0025] ① Use dataloader to load fingerprint training set A1 and finger vein training set B1 respectively, and store them in the fingerprint and finger vein training databases as tensors;

[0026] ② Construct ResNet34 and ResNet50 networks. Both ResNet34 and ResNet50 contain 4 residual blocks. Initialize the weights of the networks respectively, select the arcloss function as the loss function, define the optimizer as the Adam optimizer, and set the learning rate of the optimizer.

[0027] ③ Start training ResNet34 and ResNet50 networks using the tensors of the fingerprint and finger vein training databases loaded in the dataloader, respectively. Continuously perform gradient descent and iterate the training multiple times. Calculate the error and loss between positive and negative examples using the loss function, and backpropagate the gradients of the parameters in the two networks respectively. Repeat this process and dynamically update the loss.

[0028] ④ The fingerprint and finger vein images to be matched are mapped to the probability of each category using the arcloss function to complete the final single-modal feature matching;

[0029] 2) Input the knuckle print training set images into the DenseNet121 network for independent training:

[0030] ① Use dataloader to load the fingerprint training set and finger vein training set respectively, and store them in the fingerprint and finger vein training database as tensors;

[0031] ② Construct the DenseNet121 network, which contains three Dense blocks and one Transition Block. Initialize the network weights, select the arcloss function as the loss function, define the optimizer as the Adam optimizer, and set the learner rate of the optimizer.

[0032] ③ Start training DenseNet121 with the tensors of the fingerprint and finger vein training database loaded in the dataloader, continuously perform gradient descent, perform multiple iterations of training, use the loss function to calculate the error and loss between positive and negative examples, and perform backpropagation of gradients on the parameters in the two networks respectively. Repeat this process and dynamically update the loss.

[0033] ④ The fingerprint and finger vein images to be matched are mapped to the probability of each category using the arcloss function to complete the final single-modal feature matching;

[0034] 3) Model Validation Phase: Validate the trained ResNet34, ResNet50, and DenseNet121 networks respectively.

[0035] ① Use dataloader to load fingerprint test set A2, finger vein test set B2, and knuckle pattern test set C2 respectively, and store the fingerprint, finger vein, and knuckle pattern test datasets as tensors;

[0036] ② The tensors of the fingerprint, finger vein, and knuckle print verification databases loaded in the dataloader are used to perform network verification on ResNet34, ResNet50, and DenseNet121 respectively. Finally, the label with the highest probability in the matching results is output, which is the identity information corresponding to the image. The parameters are continuously adjusted through previous training until the model converges.

[0037] The attention feature fusion in step (4) of this invention includes:

[0038] 1) The training steps for the attention module are as follows:

[0039] First, based on the single-modal feature extraction model adjusted in step (3), extract the single-modal feature vectors of fingerprint training set A1, fingerprint training set B1, and fingerprint test set C1 respectively. Initialize the extracted multiple modal data features and assign a weight to each to form an initial modal feature weight matrix.

[0040] ① Perform dot product self-attention calculation on the extracted feature vectors to obtain a dynamically weighted vector. This can be obtained according to the self-attention calculation formula. In particular, the activation function chosen here is ReLU. 2 As an activation function;

[0041] ② The attention model introduces the concept of multi-head to allow the model to learn H different representation subspaces at each position while maintaining the same computational efficiency, which is usually represented as a parameterized input;

[0042] ③ The model is trained using a batch size of 32. To ensure consistency within the same batch, all features are first padded into the longest sequence in the batch during data processing. Since the modal features have different dimensions, they need to be mapped into the model's internal space, which has a dimension of 1024. The model is based on Transformer, with N = 2 layers and H = 4 heads. The final generator's output dimension is consistent with the descriptive word vector size, at 10172 dimensions. In the final loss calculation, label smoothing uses γ = 0.7, the loss probability p = 0.1, and the learning rate is 10. -5 The Adam optimizer is used to train the description generator;

[0043] Through model training, the optimal weights of the data features of each modality are obtained. The data features of all modalities are then weighted and fused: first, the weight of each modality is multiplied by the data features of its corresponding modality, and then the feature vectors obtained from all modalities are concatenated. The feature vectors obtained from all modalities refer to the fingerprint feature vector, finger vein feature vector obtained by the ResNet model and the finger joint pattern feature vector obtained by the DenseNet model, which form a joint feature vector. Finally, the vector is sent to the classifier for classification.

[0044] 2) The verification steps for the attention module are as follows:

[0045] ① Use dataloader to load the fingerprint training set and finger vein verification set respectively, and store them in the fingerprint and finger vein verification databases in the form of tensors;

[0046] ② The fingerprint and finger vein verification database tensors loaded in the dataloader are first used for feature extraction through ResNet34 and ResNet50, then weights are configured through the attention module, and then feature fusion is performed. Finally, the label with the highest probability in the matching result is output, which is the identity information corresponding to the image. The parameters are continuously adjusted through previous training until the model converges.

[0047] In step (5) of this invention, the fusion algorithm based on the attention module obtained through training and verification is stored in the memory of the smart lock. When a visitor performs verification, the smart lock can obtain the recognition result based on the visitor's biometric information and the trained algorithm.

[0048] The present invention has the following advantages:

[0049] 1. This invention uses multimodal biometrics for identity recognition, which makes up for the problems of insufficient biometrics in a single modality and the insecurity of single-modal biometrics. The interaction between multiple modalities helps to extract more comprehensive and effective biometrics.

[0050] 2. In the single-modal feature extraction method based on fingerprint, finger vein, and knuckle pattern described in this invention, the arcloss function is used as the loss function, which improves the model recognition rate compared to the softmax loss function.

[0051] 3. In the feature layer fusion process described in this invention, the attention mechanism is adopted, which has better feature fusion performance compared to simple feature concatenation. Furthermore, the traditional softmax activation function is replaced with ReLU for the traditional attention mechanism. 2The activation function, compared to the traditional approach of learning a weight for each modality's features and then weighting and concatenating the features, allows each modality's features to play a better role. Attached Figure Description

[0052] Figure 1 This is a flowchart of the present invention;

[0053] Figure 2 This is a block diagram illustrating the principle of the smart combination lock of this invention;

[0054] Figure 3 This is a flowchart of step (3) of the present invention;

[0055] Figure 4 This is a flowchart of step (4) of the present invention. Detailed Implementation

[0056] See Figure 1 The steps include the following:

[0057] 1. Data acquisition: Using a biometric scanner, collect biometric features from three modalities: fingerprints, finger veins, and knuckle prints;

[0058] 2. Data preprocessing: The images of the three biometric modalities were preprocessed to meet the input requirements of the corresponding models for the three modalities.

[0059] 3. Neural network model training and validation: Feature extraction is performed on the multi-modal data after preprocessing in step 2 using ResNet and DenseNet. The parameters of ResNet and DenseNet are obtained through training and validation.

[0060] 4. The attention module performs feature fusion by concatenating the feature vectors extracted from the three modalities to form a single overall feature vector. This vector is then fed into the classifier for feature classification. Employing an attention mechanism, the Q, K, and V matrices are first initialized based on the concatenated feature vector. Here, K is the key value in the attention mechanism calculation, Q represents the query, and K is the key value. Then, based on the initialized Q, K, and V, the optimal weights are calculated through training and summation to obtain the joint feature vector, which is then fed into the classifier for classification, achieving better recognition results.

[0061] 5. Deploy the application: Deploy the trained algorithm onto the smart lock. Visitors provide the required biometric features, and the smart lock identifies whether they are valid users based on the data in the database.

[0062] Step 1, the data acquisition phase for fingerprint, finger vein, and knuckle print data, includes:

[0063] Using a biometric collector, fingerprint image dataset A, finger vein image dataset B, and knuckle print image dataset C are collected from the user. This biometric collector is the biometric collector module of the smart lock. The structure of the smart lock is as follows: Figure 2 As shown, it includes: motherboard, clutch, biometric data acquisition unit, memory, and microprocessor.

[0064] In step 2, the data preprocessing stage involves preprocessing the collected biometric data from the three modalities to ensure they meet the input requirements of the corresponding models. The preprocessing steps are as follows:

[0065] (1) Automatically label the collected biological images to obtain the volunteer number and finger number corresponding to each biological image, and then divide the biological images into training set and test set in a 4:1 ratio;

[0066] (2) The collected biological images are preprocessed by subtracting the average value of each image pixel from the biological image, then using standard color enhancement, and then batch normalization.

[0067] (3) Image Enhancement: The Sobel operator is used to enhance the input fingerprint image, finger vein image, and knuckle print image respectively. Then, ROI localization and cropping are used to segment the valid fingerprint, finger vein, and knuckle print images. The formula for the Sobel operator is as follows:

[0068]

[0069] Where A represents the original image before processing, and Gx and Gy represent the images obtained after horizontal and vertical edge detection, respectively; the approximate horizontal and vertical gradient values ​​G of each pixel in the original image can be expressed as:

[0070]

[0071] The gradient direction θ is:

[0072]

[0073] Among them, the effective fingerprint training set is A1, the effective fingerprint test set is A2, the effective finger vein training set is B1, the effective finger vein test set is B1, the effective knuckle print training set is C1, and the effective knuckle print test set is C2.

[0074] In step 3, features are extracted from the preprocessed data of multiple modalities from step 2 using ResNet and DenseNet. The feature extraction steps are as follows: Figure 3 As shown. Specifically, the training and validation phases of the neural network model include:

[0075] 3.1 ResNet Network Feature Extraction and Training: The fingerprint training set A1 and the finger vein training set B1 were respectively input into the ResNet34 and ResNet50 networks for feature extraction and training to determine the network model parameters;

[0076] (1) Use dataloader to load the fingerprint training set and finger vein training set respectively, and store them in the fingerprint and finger vein training database in the form of tensors;

[0077] (2) Construct ResNet34 and ResNet50 networks. Both ResNet34 and ResNet50 contain 4 residual blocks, and initialize the weights of the networks respectively. Select the arcloss function as the loss function. The formula for the arcloss function is:

[0078]

[0079] Where N is the number of samples, i represents the i-th sample, s is the radius, m is the angular distance between the two classes, and θ is the angle between the parameter w and the feature vector x;

[0080] Define the optimizer as the Adam optimizer and set its learning rate; here, the learning rate is set to 10. -4 ;

[0081] (3) Start training ResNet34 and ResNet50 networks using the tensors of the fingerprint and finger vein training databases loaded in the dataloader. Perform gradient descent according to the Adam optimizer and train for a total of 5 iterations. When all data in the training set has been trained once, it means that one iteration of training is completed. Calculate the error and loss between positive and negative examples using the loss function, and backpropagate the gradients of the parameters in the two networks respectively. Repeat this process and dynamically update the loss.

[0082] (4) The fingerprint and finger vein images to be matched are mapped to the probability of each category through the arcloss function to complete the final single-modal feature matching;

[0083] 3.2 DenseNet Network Feature Extraction and Training: The knuckleprint training database was input into the DenseNet121 network for independent training.

[0084] (1) Load the knuckleprint training set C1 using dataloader and store it in the fingerprint and finger vein training database as a tensor.

[0085] (2) Construct the DenseNet121 network. The DenseNet121 network contains three Dense blocks and one Transition Block. Initialize the weights of the network, select the arcloss function as the loss function, define the optimizer as the Adam optimizer, and set the learning rate of the optimizer.

[0086] (3) Start training DenseNet121 with the tensor of the fingerprint and finger vein training database loaded in the dataloader. Perform gradient descent according to the Adam optimizer. Perform a total of 5 iterations of training. When all the data in the training set has been trained once, it means that one iteration of training is completed. Calculate the error and loss between positive and negative examples using the loss function, and backpropagate the gradients of the parameters in the two networks respectively. Repeat this process and dynamically update the loss.

[0087] (4) The fingerprint and finger vein images to be matched are mapped to the probability of each category through the arcloss function to complete the final single-modal feature matching;

[0088] 3.3 Validation of ResNet and DenseNet networks: Validation was performed on the trained ResNet34, ResNet50, and DenseNet121 networks respectively.

[0089] (1) Use dataloader to load fingerprint test set A2, finger vein test set B2, and finger joint pattern test set C2 respectively, and store them in the fingerprint and finger vein verification database in the form of tensors.

[0090] (2) The tensors of the fingerprint and finger vein verification databases loaded in the dataloader are used to perform network verification on ResNet34 and ResNet50 respectively. Finally, the label with the highest probability in the matching results is output, which is the identity information corresponding to the image. The parameters are continuously adjusted through previous training until the model converges. The convergence condition is that the loss is less than the target value of 10. -5 ;

[0091] The method for fusing multimodal feature vectors in step 4 is as follows: Figure 4 As shown, the feature fusion stage of the attention module includes training and validation processes:

[0092] 4.1 Perform feature fusion on the data from multiple modalities extracted in step 3, such as... Figure 4 As shown, the training steps are as follows:

[0093] (1) Based on the network model trained in step 3, biometric features are extracted from fingerprint training set A1, finger vein training set B1, and finger joint pattern training set C1 respectively. The extracted data features of multiple modalities are initialized and assigned a weight to form an initial modal feature weight matrix. Here, the weights are equal and are all 1.

[0094] (2) Here, a multi-head attention mechanism is used for calculation. First, the extracted feature vectors are subjected to dot product self-attention calculation to obtain a dynamically weighted vector. The formula for the self-attention vector is:

[0095]

[0096] In this formula, each feature vector is represented by K, where K is the key in the self-attention calculation, V = Q = K, Q represents the query, and V represents the value. T This represents the dot product operation, used to obtain the similarity between all vectors. The scaling factor is represented by N, which represents the length of the vector. head(Q,K,V) is the calculation of a single head in the multi-head attention mechanism.

[0097] Then via ReLU 2 The function is activated, where the ReLU function can be written as:

[0098]

[0099] f represents the output of the activation function;

[0100] (3) Then multi-head attention is defined as mapping back to the query subspace D. q The concatenation of K attention heads, where H is the number of distinct representation subspaces.

[0101] Attention(q,k,v)=[head1(q,k,v),head2(q,k,v),…,head H (q,k,v)]W out

[0102] (4) The model is trained using a batch size of 32. To ensure consistency within the same batch, all features are first filled into the longest sequence in the batch during data processing. Since the modal features have different dimensions, they need to be mapped into the model's internal space, which has a dimension of 1024. The model is based on Transformer, with N = 2 layers and H = 4 heads. The final generator output dimension is consistent with the descriptive word vector size, which is 10172 dimensions. In the final loss calculation, label smoothing uses γ = 0.7, the loss probability p = 0.1, and the learning rate is 10. -5 The Adam optimizer is used to train the description generator;

[0103] Through model training, the optimal weights of the data features of each modality are obtained. The data features of all modalities are then weighted and fused: first, the weight of each modality is multiplied by its corresponding data feature, and then the feature vectors obtained from all modalities are concatenated. The feature vectors obtained from all modalities refer to the fingerprint feature vector obtained by the ResNet34 model, the finger vein feature vector obtained by the ResNet50 model, and the knuckle feature vector obtained by the DenseNet121 model, forming a joint feature vector, which is finally sent to the classifier for classification.

[0104] 4.2 Validation of the attention fusion module:

[0105] (1) Use dataloader to load fingerprint test set A2, finger vein test set B2, and finger joint pattern test set C2 respectively, and store them in the fingerprint and finger vein verification database in the form of tensors.

[0106] (2) The fingerprint and finger vein verification database tensors loaded in the dataloader are first processed by ResNet34 and ResNet50 for feature extraction, then weights are configured through the attention module, and then feature fusion is performed. Finally, the label with the highest probability in the matching results is output, which is the identity information corresponding to the image. The parameters are continuously adjusted through previous training until the attention module converges. The convergence condition is that the error value is less than the predetermined expectation of 10. -5 .

[0107] In step 5, the application deployment stage, the fusion algorithm based on the attention module obtained through training and verification in steps 1-4 is stored in the memory of the smart lock.

[0108] Specifically, when a visitor attempts to verify their identity, the smart lock first collects the visitor's biometric information. Based on a pre-trained multimodal biometric recognition method, it extracts features and compares the resulting fused features with pre-trained fused features in the database. It selects the set of features with the highest similarity. If they match, the feature match is considered valid; otherwise, the visitor is considered illegitimate and access is denied.

Claims

1. A multimodal biometric identification method based on an attention module, characterized in that, Includes the following steps: (1) Data acquisition: Using a biometric collector, biometric features of three modalities, namely fingerprints, finger veins and knuckle prints, are collected; (2) Data preprocessing: The collected biometric data of the three modalities are preprocessed to meet the input requirements of the corresponding models of the three modalities; (3) In the neural network model training and validation stage, the data of multiple modalities after preprocessing in step (2) are used to extract features through ResNet and DenseNet. The parameters of ResNet and DenseNet are obtained through training and validation. (4) Feature fusion of the attention module: The feature vectors of the three modalities are extracted and concatenated to form a total feature vector, which is then sent to the classifier for feature classification. The attention mechanism is adopted. First, the Q, K, and V matrices are initialized based on the feature vector formed by the concatenation of the three modalities. Here, K is the key value in the calculation of the attention mechanism, Q represents the query, and K is the key value. Then, based on the initialized Q, K, and V, the optimal weights are calculated by training and learning, and the joint feature vector is obtained and sent to the classifier for classification, thereby achieving a better recognition effect; where: 1) The training steps for the attention module are as follows: First, based on the single-modal feature extraction model adjusted in step (3), extract the single-modal feature vectors of fingerprint training set A1, fingerprint training set B1, and fingerprint test set C1 respectively. Initialize the extracted multiple modal data features and assign a weight to each to form an initial modal feature weight matrix. ① Perform dot product self-attention calculation on the extracted feature vectors to obtain a dynamically weighted vector. This can be calculated using the self-attention formula. The activation function chosen here is ReLU. 2 As an activation function; ② The attention model introduces the concept of multi-head to allow the model to learn H different representation subspaces at each position while maintaining the same computational efficiency, represented as parameterized input; ③ The model is trained using a batch size of 32. To ensure consistency within the same batch, all features are first padded into the longest sequence in the batch during data processing. Since the modal features have different dimensions, they need to be mapped into the model's internal space, which has a dimension of 1024. The model is based on Transformer, with N = 2 layers and H = 4 heads. The final generator's output dimension matches the descriptive word vector size, being 10172 dimensions. In the final loss calculation, label smoothing uses γ = 0.7, a loss probability p = 0.1, and a learning rate of 10. -5 The Adam optimizer is used to train the description generator; Through model training, the optimal weights of the data features of each modality are obtained. The data features of all modalities are then weighted and fused: first, the weight of each modality is multiplied by the data features of its corresponding modality, and then the feature vectors obtained from all modalities are concatenated. The feature vectors obtained from all modalities refer to the fingerprint feature vector, finger vein feature vector obtained by the ResNet model and the finger joint pattern feature vector obtained by the DenseNet model, which form a joint feature vector. Finally, the vector is sent to the classifier for classification. 2) The verification steps for the attention module are as follows: ① Use dataloader to load the fingerprint training set and finger vein verification set respectively, and store them in the fingerprint and finger vein verification databases in the form of tensors; ② The fingerprint and finger vein verification database tensors loaded in the dataloader are first used for feature extraction through ResNet34 and ResNet50, then weights are configured through the attention module, then feature fusion is performed, and finally the label with the highest probability in the matching result is output, which is the identity information corresponding to the image. The parameters are continuously adjusted through previous training until the model converges. (5) Deployment and application: The trained algorithm is deployed and applied to the smart lock. Visitors provide the required biometric features, and the smart lock identifies whether they are valid users based on the data in the database.

2. The multimodal biometric identification method based on an attention module according to claim 1, characterized in that, The data acquisition in step (1) includes: using a biometric collector to collect the user's fingerprint image dataset A, finger vein image dataset B, and knuckle print image dataset C; The biometric collector is the biometric collector module of the smart lock. The smart lock includes a motherboard, clutch, biometric collector, memory, and microprocessor.

3. The multimodal biometric identification method based on an attention module according to claim 1, characterized in that, The data preprocessing in step (2) involves preprocessing the collected biological image datasets of three modalities. The preprocessing steps are as follows: 1) Automatically label the collected biological images to obtain the volunteer number and finger number corresponding to each biological image, and then divide the biological images into training set and test set according to the ratio; 2) Perform image preprocessing on the acquired biological images by subtracting the average value of each image pixel from the biological image, using standard color enhancement, and then batch normalizing; 3) Image enhancement: The Sobel operator is used to enhance the input fingerprint image, finger vein image, and knuckle print image respectively. Then, ROI localization and cropping are used to segment the effective fingerprint, finger vein, and knuckle print datasets and training sets. The effective fingerprint training set is A1, the effective fingerprint test set is A2, the effective finger vein training set is B1, the effective finger vein test set is B1, the effective knuckle print training set is C1, and the effective knuckle print test set is C2.

4. The multimodal biometric identification method based on an attention module according to claim 1, characterized in that, The specific steps in step (3) are as follows: 1) During the model training phase, features of the multiple modalities preprocessed in step (2) are extracted using ResNet and DenseNet. The fingerprint training set A1 and the finger vein training set B1 are input into the ResNet34 and ResNet50 networks for independent training, respectively. ① Use dataloader to load fingerprint training set A1 and finger vein training set B1 respectively, and store them in the fingerprint and finger vein training databases as tensors; ② Construct ResNet34 and ResNet50 networks. Both ResNet34 and ResNet50 contain 4 residual blocks. Initialize the weights of the networks respectively, select the arcloss function as the loss function, define the optimizer as the Adam optimizer, and set the learning rate of the optimizer. ③ Start training ResNet34 and ResNet50 networks using the tensors of the fingerprint and finger vein training databases loaded in the dataloader, respectively. Continuously perform gradient descent and iterate the training multiple times. Calculate the error and loss between positive and negative examples using the loss function, and backpropagate the gradients of the parameters in the two networks respectively. Repeat this process and dynamically update the loss. ④ The fingerprint and finger vein images to be matched are mapped to the probability of each category using the arcloss function to complete the final single-modal feature matching; 2) Input the knuckle print training set images into the DenseNet121 network for independent training: ① Use dataloader to load the fingerprint training set and finger vein training set respectively, and store them in the fingerprint and finger vein training database as tensors; ② Construct the DenseNet121 network. The DenseNet121 network contains three Dense blocks and one TransitionBlock. Initialize the network weights, select the arcloss function as the loss function, define the optimizer as the Adam optimizer, and set the learner rate of the optimizer. ③ Start training DenseNet121 with the tensors of the fingerprint and finger vein training database loaded in the dataloader, continuously perform gradient descent, perform multiple iterations of training, use the loss function to calculate the error and loss between positive and negative examples, and perform backpropagation of gradients on the parameters in the two networks respectively. Repeat this process and dynamically update the loss. ④ The fingerprint and finger vein images to be matched are mapped to the probability of each category using the arcloss function to complete the final single-modal feature matching; 3) Model Validation Phase: Validate the trained ResNet34, ResNet50, and DenseNet121 networks respectively. ① Use dataloader to load fingerprint test set A2, finger vein test set B2, and knuckle pattern test set C2 respectively, and store the fingerprint, finger vein, and knuckle pattern test datasets as tensors; ② The tensors of the fingerprint, finger vein, and knuckle print verification databases loaded in the dataloader are used to perform network verification on ResNet34, ResNet50, and DenseNet121 respectively. Finally, the label with the highest probability in the matching results is output, which is the identity information corresponding to the image. The parameters are continuously adjusted through previous training until the model converges.

5. The multimodal biometric identification method based on an attention module according to claim 1, characterized in that, In step (5), the fusion algorithm based on the attention module obtained through training and verification is stored in the memory of the smart lock. When a visitor performs verification, the smart lock can obtain the recognition result based on the visitor's biometric information and the trained algorithm.

Images