Finger and palm vein authentication method combining CNN and Transformer

Abstract

The present invention relates to the technical field of computer vision, and proposes a finger and palm vein authentication method combining CNN and Transformer, comprising the following steps: obtaining a finger and palm vein image to be authenticated; extracting finger and palm vein features to be authenticated from the finger and palm vein image to be authenticated by using a deep learning model for extracting finger and palm vein features; wherein the deep learning model comprises: a deep learning model composed of CNN and Transformer; performing similarity measurement on the finger and palm vein features to be authenticated and the finger and palm vein features registered in a preset database, if the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, then the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and more specifically, to a finger and palm vein authentication method combining CNN and Transformer. Background Art

[0002] As a technology that can identify and authorize personal identity, identity authentication technology is of great significance to the protection of personal information and is widely used in scenarios such as mobile payment and security systems. Different from traditional authentication methods such as keys and passwords, the method of using biometrics for identity authentication has gradually become the mainstream due to its safer and more convenient characteristics. As characteristic information unique to organisms, biometrics have high uniqueness and identifiability. Biometric recognition technology uses the uniqueness and specificity of individual biometrics to distinguish different individuals. It can easily and effectively identify and verify personal identity information and show high security performance in identity authentication technology. Commonly used biometrics include face, fingerprint, iris, gait, finger and palm vein, etc. Compared with other biometric recognition technologies, finger and palm vein authentication has become a safe, easy-to-use and widely used biometric recognition technology with its high stability, difficulty in counterfeiting, convenience of use, and ability to detect liveness.

[0003] Existing finger and palm vein authentication methods use image segmentation technology to extract vein patterns. However, they cannot accurately identify and separate vein patterns in complex backgrounds or when light conditions vary greatly, resulting in low accuracy of authentication results. Image segmentation technology also requires more computing resources and time to process image data, resulting in slow processing speed and poor real-time performance. Summary of the invention

[0004] In order to overcome the defects of the above-mentioned prior art in terms of low result accuracy and poor real-time performance, the present invention provides a finger and palm vein authentication method combining CNN and Transformer with high result accuracy and strong real-time performance.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] Acquire the finger and palm vein images to be authenticated;

[0007] Extracting the finger and palm vein features to be authenticated from the finger and palm vein images to be authenticated using a deep learning model for extracting finger and palm vein features; wherein the deep learning model includes: a deep learning model composed of CNN and Transformer;

[0008] The similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features in the preset database is measured. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails.

[0009] The present invention also proposes a finger and palm vein authentication system combining CNN and Transformer to implement the above-mentioned finger and palm vein authentication method combining CNN and Transformer, and the system includes:

[0010] The module for acquiring the image to be authenticated is used to acquire the finger and palm vein images to be authenticated;

[0011] The feature extraction module to be authenticated is used to extract the finger and palm vein features to be authenticated from the finger and palm vein image to be authenticated by using a deep learning model for extracting finger and palm vein features; wherein the deep learning model includes: a deep learning model composed of CNN and Transformer;

[0012] The authentication module is used to measure the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features in a preset database. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails.

[0013] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned finger and palm vein authentication method combining CNN and Transformer.

[0014] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0015] A deep learning model is used to extract finger and palm vein features from finger and palm vein images, and the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is measured. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than the preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails. The real-time performance of feature extraction using a deep learning model is high, and the deep learning model composed of CNN and Transformer can not only use CNN to fully extract the local features of the finger and palm vein images, but also use Transformer with excellent global information interaction ability due to the self-attention mechanism to fully extract the global features of the overall vein distribution of the finger and palm vein images. The local and global features of the finger and palm vein images are comprehensively considered, and the result is highly accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the finger and palm vein authentication method combining CNN and Transformer proposed in Example 1;

[0017] Figure 2 This is a schematic diagram of the structure of the deep learning model for extracting finger and palm vein features proposed in Example 2;

[0018] Figure 3 This is a schematic diagram of the structure of MobileNetV2 proposed in Example 2;

[0019] Figure 4 This is a schematic diagram of the multi-scale feature fusion module proposed in Example 2;

[0020] Figure 5 This is a schematic diagram of the structure of the Transformer basic module proposed in Example 2;

[0021] Figure 6 This is the overall framework diagram of the finger and palm vein authentication system combining CNN and Transformer proposed in Example 3. DETAILED DESCRIPTION

[0022] The accompanying drawings are only used for illustrative purposes and should not be construed as limiting the present embodiment;

[0023] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0024] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0025] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0026] Example 1

[0027] This embodiment proposes a finger and palm vein authentication method that combines CNN and Transformer. Figure 1 Schematic diagram of the process of the finger and palm vein authentication method combining CNN and Transformer in this embodiment;

[0028] The finger and palm vein authentication method combining CNN and Transformer proposed in this embodiment includes the following steps:

[0029] S1: Acquire the finger and palm vein images to be authenticated;

[0030] S2: extracting the finger and palm vein features to be authenticated from the finger and palm vein image to be authenticated using a deep learning model for extracting finger and palm vein features; wherein the deep learning model includes: a deep learning model composed of CNN and Transformer;

[0031] S3: The finger and palm vein features to be authenticated are measured for similarity with the registered finger and palm vein features in a preset database. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, the finger and palm vein authentication is successful; otherwise, the finger and palm vein authentication fails.

[0032] In this optional embodiment, the full name of CNN in English is: Convolutional Neural Network, and the full name in Chinese is: Convolutional Neural Network; Transformer is a neural network based on the attention mechanism;

[0033] In this optional embodiment, a deep learning model is used to extract finger and palm vein features from a finger and palm vein image, and the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is measured. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails. The real-time performance of feature extraction using a deep learning model is high, and the deep learning model composed of CNN and Transformer can not only use CNN to fully extract the local features of the finger and palm vein image, but also use Transformer with excellent global information interaction capabilities due to the self-attention mechanism to fully extract the global features of the overall vein distribution of the finger and palm vein image, comprehensively considering the local features and global features of the finger and palm vein image, and the result has high accuracy.

[0034] In an optional embodiment, the step of obtaining the finger and palm vein images to be authenticated includes:

[0035] Use a near-infrared light source to illuminate the hand, and use a near-infrared camera to photograph the fingers and palms to obtain the finger and palm vein images to be authenticated.

[0036] As an exemplary illustration, the finger and palm vein image to be authenticated is a grayscale image;

[0037] In this optional embodiment, the palmar veins are hidden under the skin of the fingers, and ordinary camera imaging cannot capture the information of the palmar veins. By utilizing the characteristic that the living hemoglobin in the palmar veins can absorb near-infrared light, the palmar vein images are illuminated with near-infrared light of a fixed wavelength and collected using a near-infrared camera. As an exemplary illustration, 700-1000 nanometers of near-infrared light is used. Due to the imaging principle of the near-infrared camera, the palmar vein images will not be affected by epidermal damage or changes in the external environment, and the acquired palmar vein images have high imaging quality.

[0038] In an optional embodiment, before extracting the finger and palm vein features from the finger and palm vein image to be authenticated using a deep learning model for extracting finger and palm vein features, the finger and palm vein image to be authenticated is preprocessed, and the preprocessing step includes:

[0039] Perform edge detection on the finger and palm of the finger and palm vein image to be authenticated, segment the finger, palm and background based on the edge detection results, and highlight the finger joints;

[0040] Fitting the finger midline based on the finger portion of the segmented finger and palm vein image to be authenticated, and adjusting the finger and palm vein image to be authenticated so that the angle between the fitted finger midline and the horizontal direction is a preset angle;

[0041] Intercepting the area between the first knuckle and the second knuckle of the highlighted finger in the finger-palm vein image to be authenticated, and obtaining the finger vein region of interest of the finger-palm vein image to be authenticated;

[0042] The maximum inscribed rectangle of the palm part of the finger and palm vein image to be authenticated obtained by segmentation is solved, and the area corresponding to the maximum inscribed rectangle is intercepted to obtain the palm vein region of interest of the finger and palm vein image to be authenticated.

[0043] As an exemplary explanation, the maximum inscribed rectangle of the palm part of the finger and palm vein image to be authenticated is solved based on the solution method of the maximum inscribed rectangle of a polygon, and the maximum inscribed rectangle of a polygon refers to a rectangle with the largest area that can be exactly inscribed with the polygon;

[0044] As an exemplary explanation, the preprocessing step further includes: performing image enhancement on the finger and palm vein images to be authenticated that include the annotated finger vein regions of interest and palm vein regions of interest by using random brightness changes, random radiation changes, random perspective changes, etc.;

[0045] In this optional embodiment, the full English name of the region of interest is: region of interest, abbreviated as: ROI region.

[0046] In an optional embodiment, in a deep learning model composed of CNN and Transformer, the CNN includes: a MobileNetV2 composed of a series of repeatedly stacked bottleneck structures; the MobileNetV2 sequentially includes: a 1×1 convolution layer that expands the number of input channels to a specified value, a 3×3 convolution layer with a depthwise separable convolution, and a 1×1 convolution layer that reduces the number of channels to an original value;

[0047] The bottleneck structure of the MobileNetV2 is connected to the 1×1 convolutional layer that does not belong to MobileNetV2, and the 1×1 convolutional layer that does not belong to MobileNetV2 is connected to the input layer of the Transformer;

[0048] The last convolution layer of the MobileNetV2 is connected to the average pooling layer, and the output of the average pooling layer of the MobileNetV2 is weightedly fused with the output of the Transformer, and the weighted fusion result is the finger and palm vein features extracted by the deep learning model;

[0049] The palmar vein characteristics F out The expression is:

[0050] F out =α*F Transformer +(1-α)*F Pooling

[0051] In the formula, F Transformer represents the output of Transformer, F Pooling represents the output of the average pooling layer of MobileNetV2, α represents an adjustable hyperparameter, and also represents F Transformer The weight of F Pooling The weight of , * indicates a multiplication operation.

[0052] In this optional embodiment, MobileNetV2 is a lightweight convolutional neural network. As an exemplary illustration, the network structure of MobileNetV2 is mainly composed of a series of repeatedly stacked bottleneck structures. The bottleneck structure is the basic building block of MobileNetV2, which consists of a 1×1 convolution layer (which expands the number of input channels to a specified value), a 3×3 convolution layer with depthwise separable convolution (usually using convolution with a step size of 2 to reduce the size of the feature map), and another 1×1 convolution layer (which reduces the number of channels to the original value); the main purpose of this structural design is to improve the network's expressiveness without significantly increasing the computational complexity;

[0053] In this optional embodiment, first, MobileNetV2 is used as a CNN module to extract local features of the finger and palm veins; then, feature maps of different scales in the CNN module are taken out and input into the multi-scale feature fusion module to achieve enhancement and fusion of multi-scale features; then, the fused features are input into the Transformer module, and the global features of the finger and palm veins are fully extracted using the self-attention mechanism; finally, the local features output by the CNN module and the global features output by the Transformer module are weightedly fused to obtain the final output finger and palm vein features.

[0054] In an optional embodiment, using a deep learning model for extracting finger and palm vein features, before extracting the finger and palm vein features to be authenticated from the finger and palm vein image to be authenticated, the deep learning model is trained to obtain a trained deep learning model;

[0055] When training the deep learning model, the training data set used includes: several finger and palm vein images of several individuals with category labels; wherein all finger and palm vein images of the same individual carry the same category label, and the category label carried by any finger and palm vein image of one person is different from the category label carried by any finger and palm vein image of another person.

[0056] In an optional embodiment, the step of training the deep learning model includes:

[0057] Introducing a loss function L of the deep learning model based on the training data set, inputting the training data set into the deep learning model, training the deep learning model using the loss function L, iteratively solving the loss function L during the training process, and ending the training when the number of iterations reaches a preset value or the loss function value reaches a minimum value, thereby obtaining a trained deep learning model;

[0058] The expression of the loss function L is:

[0059] L=aL NLL +bL Center

[0060]

[0061] Where, L NLL represents the negative log-likelihood loss function, L Center represents the center loss function, a represents L NLL The weight of L Center The weight of; N represents the total number of category labels in the training data set, y i represents the category label carried by the i-th finger and palm vein image in the training dataset, p iThe category label carried by the i-th finger and palm vein image in the training dataset predicted by the deep learning model is y i probability;

[0062] When extracting the finger and palm vein features to be authenticated from the finger and palm vein images to be authenticated, the trained deep learning model is used to perform feature extraction.

[0063] As an exemplary explanation, during the training process of the deep learning model, the finger and palm vein features output by the deep learning model are used together with the category center to calculate the center loss, and the category center is optimized.

[0064] In an optional embodiment, the preset database includes: finger and palm vein features corresponding to a number of finger and palm vein images of a number of individuals pre-extracted using a trained deep learning model.

[0065] In an optional embodiment, when measuring the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features, the cosine similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is calculated. If the cosine similarity is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails.

[0066] The expression of the cosine similarity is:

[0067]

[0068] In the formula, F out Indicates the finger and palm vein features to be authenticated, F know Indicates the registered palm vein features, cos(F out ,F know ) indicates F out and F know The cosine distance between .

[0069] In the specific implementation process, as an exemplary explanation, the first way to construct the database is: store the finger and palm vein features of the same person as a subset, and there are M subsets if a total of M people are registered; if it is necessary to verify whether the finger and palm vein features to be authenticated belong to a person registered in the database, then when performing cosine distance calculation, only one feature is randomly selected from each subset to perform similarity measurement with the feature to be authenticated, and the calculation is stopped when a result greater than a preset threshold is calculated, and the authentication is successful; if the similarity between the M features randomly selected is less than the preset threshold, then M features are taken from the sample library until a result greater than the preset threshold is calculated, and the authentication is successful, and if no result greater than the preset threshold is traversed through all features, the authentication fails; if it is necessary to verify whether the finger and palm vein features to be authenticated belong to a specific person registered in the database, then when performing cosine distance calculation, the subset corresponding to the specific person is retrieved, and the features in the subset are used to perform similarity measurement with the features to be authenticated;

[0070] As an exemplary illustration, the second way to construct the database is as follows: each finger and palm vein feature is stored separately; if it is necessary to verify whether the finger and palm vein feature to be authenticated belongs to a person registered in the database, then, during authentication, it is necessary to calculate the cosine distance between each feature in the database and the finger and palm vein feature to be authenticated, and stop calculating if a result greater than a preset threshold is calculated, and the authentication is successful; if no result greater than the preset threshold is found after traversing all features, the authentication fails.

[0071] Example 2

[0072] This embodiment is based on the finger and palm vein authentication method combining CNN and Transformer proposed in Embodiment 1, and proposes the following specific implementation examples:

[0073] Figure 2 This is a schematic diagram of the structure of the deep learning model for extracting finger and palm vein features proposed in this embodiment. Figure 3 This is a schematic diagram of the structure of MobileNetV2 proposed in this embodiment; Figure 4 Schematic diagram of the multi-scale feature fusion module proposed in this embodiment. Figure 5 This is a schematic diagram of the structure of the basic module of the Transformer proposed in this embodiment;

[0074] If the output layer of CNN is directly connected to the input layer of Transformer, only the deepest features in CNN will be transmitted to Transformer for processing. Transformer cannot perceive the feature information of different scales in the palm veins, resulting in the loss of some low-level semantic features.

[0075] In order to fully obtain the multi-scale features of the finger and palm veins, this application designs a multi-scale feature fusion module (MSFF) between CNN and Transformer; as an exemplary illustration, the multi-scale feature fusion module includes: a 1×1 convolutional layer and a Transformer that do not belong to MobileNetV2,

[0076] like Figure 2 As shown, the multi-scale feature fusion module is used to extract features of different scales in MobileNetV2 and input the extracted features into a 1×1 convolutional layer. As an exemplary illustration, the features of different scales include: H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, where H and W represent the height and width of the image data input to MobileNetV2, respectively. Figure 3 The layers corresponding to the feature outputs of MobileNetV2 at different scales are shown. Figure 3 In the equation, t is the dimension-raising factor (determines the dimension in the bottleneck), c is the number of output channels, n is the number of stacks, and s is the step size. When s is 2, the feature map will be downsampled. Figure 3 Four feature maps of different scales, F1, F2, F3 and F4, are extracted, with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original input respectively;

[0077] As an example, Figure 2 The classifier uses linear mapping and nonlinear activation function to map the palm vein features to the classification space and calculates the negative log-likelihood loss (NLL loss).

[0078] As an exemplary illustration, the Transformer includes: T2T (Tokens-to-Token) and a Transformer basic module, wherein the T2T is a T2T module in a Transformer-based visual model (T2T-ViT), and the T2T module includes a Transformer basic module; Figure 4 Schematic diagram of the multi-scale feature fusion module proposed in this embodiment. Figure 4 It shows the feature map F1 as input Figure 2 The MSFF section of the process is within the box. Figure 4 Medium, F I+1 I=1, if the feature map F2 is used as input, then F I+1 I=2, and so on; the MSFF in this embodiment not only connects CNN and Transformer, but also enhances and fuses the features of different scales in CNN so that more feature information can be paid attention to in the subsequent Transformer processing, thereby improving the feature richness of the entire model;

[0079] Regardless of the training process or the authentication process, the finger and palm vein images first pass through the CNN module, which is implemented by MobileNetV2. The local features of the finger and palm veins are extracted through convolution, and the input feature map is gradually downsampled. Then, the feature maps of different scales output by CNN are input into the MSFF module, such as Figure 3 As shown in the figure, the module first performs channel dimension reduction on feature maps of different scales, fixes the number of channels to 256, and uses the T2T module to achieve feature enhancement, and then adds it to the feature map after dimension reduction of the next scale to achieve the fusion of feature maps of different scales; then, the fused features are input into the Transformer module, which contains 5 basic blocks, such as Figure 5 As shown in the figure, each basic block includes a linear transformation layer, multi-head self-attention, post-batch normalization and a feedforward neural network. The input feature map will be divided into several image blocks for parallel input. The linear transformation layer realizes the transformation of the feature space and the change of the feature dimension through the linear mapping function. The multi-head self-attention uses the self-attention mechanism to obtain the correlation between the parallel inputs to extract the global features of the feature map. The feedforward neural network uses a multi-layer perceptron and a nonlinear activation function to enhance the features. Post-batch normalization improves the stability of the model and strengthens the extraction of the overall distribution features of the samples. Finally, the features output by the Transformer module are weighted fused with the features output by the last layer of the CNN module to obtain the finger and palm vein features output by the model. During the training process, the finger and palm vein features output by the model will be input into the classifier. After being mapped to the classification space by the linear layer, the finger and palm vein features are processed by the activation layer for nonlinear mapping, so that the output results are input into the NLL loss and the center loss for supervised training. During the certification process, the classifier and the loss function are removed, and the main part of the model is used to extract the finger and palm vein features.

[0080] In summary, in CNN, MobileNetV2 is used to extract local features of the finger and palm veins; in MSFF, the features of different scales extracted by CNN are taken out and the T2T module is used to realize feature enhancement and downsampling, and multi-scale feature fusion is realized at the same time; in Transformer, the fused feature map is divided into multiple image blocks for parallel input, the self-attention mechanism is used to realize global feature extraction, and the feedforward neural network is used to realize the expression ability of enhanced features. Finally, the features output by CNN and the features output by Transformer are weighted and fused to obtain the finger and palm vein features output by the model.

[0081] Example 3

[0082] This embodiment proposes a finger and palm vein authentication system combining CNN and Transformer, which is used to implement the finger and palm vein authentication method combining CNN and Transformer proposed in Example 1.

[0083] Figure 6 This is the overall framework diagram of the finger and palm vein authentication system combining CNN and Transformer in this embodiment.

[0084] The finger and palm vein authentication system combining CNN and Transformer includes:

[0085] The module for acquiring the image to be authenticated is used to acquire the finger and palm vein images to be authenticated;

[0086] The feature extraction module to be authenticated is used to extract the finger and palm vein features to be authenticated from the finger and palm vein image to be authenticated by using a deep learning model for extracting finger and palm vein features; wherein the deep learning model includes: a deep learning model composed of CNN and Transformer;

[0087] The authentication module is used to measure the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features in a preset database. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails.

[0088] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the finger and palm vein authentication method combining CNN and Transformer described in Example 1.

[0089] It can be understood that the finger and palm vein authentication system and computer device combining CNN and Transformer in this embodiment improves the method of embodiment 1. The options in the above embodiment 1 are also applicable to this embodiment, so they will not be repeated here.

[0090] The same or similar reference numerals correspond to the same or similar components;

[0091] The terms used to describe the positional relationship in the drawings are only used for illustrative purposes and should not be construed as limiting the present embodiment;

[0092] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. The finger and palm vein authentication method combining CNN and Transformer is characterized by: The following steps are involved: Acquire the finger and palm vein images to be authenticated; The finger and palm vein features to be authenticated are extracted from the finger and palm vein images to be authenticated by using a deep learning model for extracting finger and palm vein features; wherein the deep learning model comprises: a deep learning model composed of CNN and Transformer; in the deep learning model composed of CNN and Transformer, the CNN comprises: MobileNetV2 composed of a series of repeatedly stacked bottleneck structures; the MobileNetV2 comprises in sequence: a 1×1 convolution layer for expanding the number of input channels to a specified value, a 3×3 convolution layer with depthwise separable convolution, and a 1×1 convolution layer for reducing the number of channels to the original value; the Transformer comprises: a T2T module and 5 Transformer basic modules; wherein each Transformer basic module comprises a linear transformation layer, a multi-head self-attention, a post-batch normalization, and a feedforward neural network; The bottleneck structure of the MobileNetV2 is connected to a 1×1 convolutional layer that does not belong to MobileNetV2, and the 1×1 convolutional layer that does not belong to MobileNetV2 is connected to the input layer of the Transformer; wherein the MobileNetV2 inputs feature maps of different sizes into the Transformer, the T2T module of the Transformer fuses the feature maps of different sizes, and inputs the fused feature maps into the Transformer basic module, and the fused feature maps are processed by 5 Transformer basic modules and output by the Transformer; The last convolution layer of the MobileNetV2 is connected to the average pooling layer, and the output of the average pooling layer of the MobileNetV2 is weightedly fused with the output of the Transformer, and the weighted fusion result is the finger and palm vein features extracted by the deep learning model; The palmar vein characteristics F out The expression is: F out =α*F Transformer +(1-a)*F Pooling In the formula, F Transformer represents the output of Transformer, F Pooling represents the output of the average pooling layer of MobileNetV2, α represents an adjustable hyperparameter, and also represents F Transformer The weight of F Pooling The weight of , * indicates a multiplication operation; The similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features in the preset database is measured. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails.

2. The finger and palm vein authentication method combining CNN and Transformer according to claim 1 is characterized in that: The steps of obtaining the finger and palm vein images to be authenticated include: Use a near-infrared light source to illuminate the hand, and use a near-infrared camera to photograph the fingers and palms to obtain the finger and palm vein images to be authenticated.

3. The finger and palm vein authentication method combining CNN and Transformer according to claim 1, characterized in that: Before extracting the finger and palm vein features from the finger and palm vein image to be authenticated using the deep learning model for extracting finger and palm vein features, the finger and palm vein image to be authenticated is preprocessed, and the preprocessing steps include: Perform edge detection on the finger and palm of the finger and palm vein image to be authenticated, segment the finger, palm and background based on the edge detection results, and highlight the finger joints; Fitting the finger midline based on the finger portion of the segmented finger and palm vein image to be authenticated, and adjusting the finger and palm vein image to be authenticated so that the angle between the fitted finger midline and the horizontal direction is a preset angle; Intercepting the area between the first knuckle and the second knuckle of the highlighted finger in the finger-palm vein image to be authenticated, and obtaining the finger vein region of interest of the finger-palm vein image to be authenticated; The maximum inscribed rectangle of the palm part of the finger and palm vein image to be authenticated obtained by segmentation is solved, and the area corresponding to the maximum inscribed rectangle is intercepted to obtain the palm vein region of interest of the finger and palm vein image to be authenticated.

4. The finger and palm vein authentication method combining CNN and Transformer according to any one of claims 1 to 3, characterized in that: Using a deep learning model for extracting finger and palm vein features, before extracting the finger and palm vein features to be authenticated from the finger and palm vein image to be authenticated, the deep learning model is trained to obtain a trained deep learning model; When training the deep learning model, the training data set used includes: several finger and palm vein images of several individuals with category labels; wherein all finger and palm vein images of the same individual carry the same category label, and the category label carried by any finger and palm vein image of one person is different from the category label carried by any finger and palm vein image of another person.

5. The finger and palm vein authentication method combining CNN and Transformer according to claim 4 is characterized in that: The step of training the deep learning model includes: Introducing a loss function L of the deep learning model based on the training data set, inputting the training data set into the deep learning model, training the deep learning model using the loss function L, iteratively solving the loss function L during the training process, and ending the training when the number of iterations reaches a preset value or the loss function value reaches a minimum value, thereby obtaining a trained deep learning model; The expression of the loss function L is: <h2 style=";text-align:left;direction:ltr">L=aL<h2 style=";text-align:left;direction:ltr"> NLL <h2 style=";text-align:left;direction:ltr"> +bL<h2 style=";text-align:left;direction:ltr"> Center Where, L NLL represents the negative log-likelihood loss function, L Center represents the center loss function, a represents L NLL The weight of L Center The weight of; N represents the total number of category labels in the training data set, y i represents the category label carried by the i-th finger and palm vein image in the training dataset, p i The category label carried by the i-th finger and palm vein image in the training dataset predicted by the deep learning model is y i probability; When extracting the finger and palm vein features to be authenticated from the finger and palm vein images to be authenticated, the trained deep learning model is used to perform feature extraction.

6. The finger and palm vein authentication method combining CNN and Transformer according to claim 4, characterized in that: The preset database includes: finger and palm vein features corresponding to a plurality of finger and palm vein images of a plurality of individuals pre-extracted using a trained deep learning model.

7. The finger and palm vein authentication method combining CNN and Transformer according to claim 1, characterized in that: When measuring the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features, the cosine similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is calculated. If the cosine similarity is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails; The expression of the cosine similarity is: In the formula, F out Indicates the finger and palm vein features to be authenticated, F know Indicates the registered palm vein features, cos(F out ,F know ) indicates F out and F know The cosine distance between .

8. A finger and palm vein authentication system combining CNN and Transformer, used to implement the finger and palm vein authentication method combining CNN and Transformer as described in any one of claims 1 to 7, characterized in that: include: The module for acquiring the image to be authenticated is used to acquire the finger and palm vein images to be authenticated; The feature extraction module to be authenticated is used to extract the finger and palm vein features to be authenticated from the finger and palm vein image to be authenticated by using a deep learning model for extracting finger and palm vein features; wherein the deep learning model includes: a deep learning model composed of CNN and Transformer; The authentication module is used to measure the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features in a preset database. If the similarity between the finger and palm vein features to be authenticated and the registered finger and palm vein features is greater than a preset threshold, the finger and palm vein authentication is successful, otherwise, the finger and palm vein authentication fails.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor executes the steps of the finger and palm vein authentication method combining CNN and Transformer as described in any one of claims 1 to 7.

Images