Finger vein image quality evaluation method based on two-stage fusion model

Abstract

The invention discloses a finger vein image quality evaluation method based on a two-stage fusion model, and the basic principle of the method comprises the steps: firstly carrying out the image preprocessing of a finger vein gray-scale image, and obtaining a vein binary image through binarization; then taking the grey-scale image and the binary image as an input training model of a neural network model to extract quality features; and finally, judging whether the vein image is a high-quality vein image or a low-quality vein image through a quality category output by the model. According to the method, firstly, a two-stage fused double-branch finger vein image quality evaluation model is designed, and the model is mainly composed of four parts: a main path used for extracting finger vein grayscale image features; an auxiliary path used for extracting finger vein binary image features; a first-stage single-branch multi-scale fusion module; in the second stage, based on a Vision Transform cross-branch fusion module, the finger vein image quality is accurately judged through image preprocessing and a deep learning technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of biometric feature recognition, and in particular relates to a finger vein image quality assessment method based on a two-stage fusion model. Background Art

[0002] As an efficient, safe and convenient means of identity authentication, biometric recognition technology has been widely used in various fields of our lives, such as access control systems, payment verification, security protection, etc. Among the many biometric recognition technologies, finger vein recognition technology has become a hot topic in recent years due to its uniqueness, liveness and anti-counterfeiting characteristics.

[0003] In actual application, finger vein recognition technology faces many challenges, among which the quality of vein images is a key factor restricting recognition performance. Finger vein images are easily affected by factors such as the performance of the acquisition equipment, ambient lighting, or the position of the user's fingers during the acquisition process. The quality of the acquired images varies. Low-quality vein images are not conducive to subsequent feature extraction and are prone to recognition errors.

[0004] Although the current finger vein image quality assessment model has introduced convolutional neural network (CNN) for feature extraction, most models fail to fully explore and utilize the multi-level and multi-scale information between feature maps during the feature extraction process within a single branch (such as grayscale image branch or binary image branch), resulting in the limited expression ability of single branch features and the inability to fully reflect the quality characteristics of finger vein images.

[0005] The existing dual-branch quality assessment model mainly uses a simple fully connected layer feature fusion method to fuse grayscale and binary image features, ignoring the potential connection between the internal feature map of a single branch during the extraction process and the deep correlation and complementary information between the two branches. This fusion strategy limits the model's ability to accurately assess the quality of vein images in complex environments. Summary of the invention

[0006] In order to solve the technical problems existing in the background technology, the present invention aims to provide a finger vein image quality assessment method based on a two-stage fusion model, which can accurately judge the quality of finger vein images through image preprocessing and deep learning technology.

[0007] In order to solve the technical problem, the technical solution of the present invention is:

[0008] A finger vein image quality assessment method based on a two-stage fusion model, the method comprising:

[0009] Obtain a finger vein grayscale image and a binary image, input them into a constructed two-stage fusion dual-branch finger vein image quality assessment model, and judge whether it is a high-quality vein image or a low-quality one by the quality category output by the model; the two-stage fusion dual-branch finger vein image quality assessment model comprises: a main network for extracting finger vein grayscale image features, an auxiliary network for extracting finger vein binary image features, a multi-scale fusion module of a single branch in the first stage, and a cross-branch fusion module based on ViT in the second stage;

[0010] Among them, in the first stage, the internal features of a single branch are optimized, and a multi-scale feature fusion strategy is adopted for both the main and auxiliary paths, and the feature maps are finely processed in combination with the channel attention mechanism to enhance the feature expression ability; in the second stage, deep fusion of cross-branch features is performed, and the ViT-based method is used to capture the global dependency between the main and auxiliary path features, and the feature information extracted from the two branches is deeply integrated.

[0011] Furthermore, by performing image preprocessing on the collected grayscale image of the finger veins, the Sobel edge detection algorithm is applied to identify the top and bottom boundaries in the image, thereby defining a region of interest (ROI), and then the blood vessels of the obtained ROI are extracted to obtain a binary image of the finger veins.

[0012] Furthermore, in the main network: the finger vein grayscale image is used as the main input, and the main network is composed of five layers of convolution modules. In the fourth and fifth convolution layers, a channel attention module is embedded. By assigning different weights to feature channels, important features are emphasized, the feature expression ability of the model is enhanced, and the key information in the vein image is captured.

[0013] Furthermore, in the auxiliary network: the finger vein binary image is used as the auxiliary input, and a black-and-white image containing the vascular venous structure is obtained through the maximum entropy threshold segmentation method. The auxiliary network uses five deeply separable convolution blocks for stacking, and introduces a channel attention module in the fourth and fifth convolution layers to strengthen the extraction of key vascular venous features and supplement and enhance the features extracted by the main network.

[0014] Furthermore, the multi-scale fusion module of the single branch in the first stage includes two parts: channel attention weighting and feature map reorganization fusion;

[0015] Determine the channel attention module input feature map The size of is C×H×W, and global average pooling and maximum average pooling are performed on each channel to obtain two feature vectors of shape C×1×1; the two vectors are then passed through a multi-layer perceptron MLP to generate the attention weight of each channel; finally, the calculated channel weight is multiplied by each channel of the input feature map to obtain the enhanced feature map; here, the output feature map size of the fourth layer convolution block of the two branches is 256×32×32, which is reorganized into 4 feature sub-maps of size 256×4×4, and then fused with the fifth layer output to output a 1536×8×8 feature map. The channel attention weight calculation formula and the weighted feature map are shown in formulas (1) and (2):

[0016] M C =σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (6)

[0017]

[0018] Among them, M C represents the channel attention weight, σ represents the sigmoid activation operation, represents the weighted feature map and represents the dot product.

[0019] Furthermore, in the second stage of ViT cross-branch fusion, the feature map after the two branches are concatenated is used as the module input. The feature map size is 8×8, the patch size is 4×4, and it is divided into 4 patches. After each patch is flattened into a one-dimensional vector, the one-dimensional vector is mapped to a fixed dimension through a linear layer to form an embedded vector, as shown in formula (3):

[0020]

[0021] in, is the one-dimensional vector after the patch is flattened, E is the linear projection matrix, and E pos is the position encoding, Z0 is the initial embedding vector, and the normalized embedding vector is sent to the multi-head self-attention layer to calculate the self-attention of the input sequence and capture the dependencies between different patches. Then, the MSA output is directly added to the input using a skip connection to form a residual connection to alleviate the gradient disappearance, as shown in formula (4):

[0022]

[0023] In the formula, is the self-attention output, LN is the normalization operation, No. The input embedding vector of the layer, After normalization, feature transformation and extraction are performed through MLP, and then a skip connection is added to add the input to the output of MLP to complete the processing of the Transformer layer, as shown in formula (5):

[0024]

[0025] In the formula, is the output embedding vector of the current layer, The index of the current layer is processed by the feedforward neural network and layer normalization, and the output of the feedforward neural network is added to the self-attention output. The whole process includes 8 Transformer layers, and finally a 1×1 convolution is performed to reduce the feature dimension. The output feature map retains the global and local key feature information of the input image, which promotes the improvement of image quality assessment effect.

[0026] Compared with the prior art, the advantages of the present invention are:

[0027] 1. Two-stage fusion strategy: The entire method adopts a two-stage fusion strategy for the model. The first stage focuses on optimizing the internal features of a single branch, and the second stage performs deep fusion of cross-branch features. This staged fusion method helps to gradually improve the feature expression ability of the model and ultimately improve the overall performance of the model.

[0028] 2. Multi-scale feature fusion and channel attention mechanism: In a single branch (grayscale image branch and binary image branch), a multi-scale feature fusion strategy is adopted, combined with a channel attention mechanism, to perform fine processing on the feature map. This key technical point aims to enhance the feature expression capability within a single branch, so that the model can more comprehensively capture the multi-level and multi-scale information in the vein image.

[0029] 3. ViT-based cross-branch feature fusion: A ViT-based cross-branch fusion method is introduced between the two branches (grayscale image branch and binary image branch) to deeply integrate the feature information extracted from the two branches. The ViT model can capture the global dependency between features through the self-attention mechanism, thereby effectively integrating the deep correlation and complementary information in the grayscale image and the binary image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 , Finger vein grayscale Figure 2 The process of valuing;

[0031] Figure 2 , dual-branch finger vein image quality assessment model structure based on two-stage fusion;

[0032] Figure 3 , main road network structure;

[0033] Figure 4 , auxiliary road network structure;

[0034] Figure 5 ,Multi-scale fusion of a single branch;

[0035] Figure 6 ,ViT feature enhancement module. DETAILED DESCRIPTION

[0036] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0037] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0038] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0039] Embodiment 1:

[0040] This paper proposes a finger vein image quality assessment method based on a two-stage fusion model. The basic principle of this method is: firstly, the finger vein grayscale image is preprocessed to obtain a vein binary image through binarization; then the grayscale image is combined with the binary image. Figure 1 The training model is used as the input of the neural network model to extract quality features; finally, the quality category of the model output is used to determine whether it is a high-quality vein image or a low-quality one. This method first designs a two-stage fusion dual-branch finger vein image quality assessment model, which mainly consists of four parts: the main path for extracting the grayscale features of the finger vein; the auxiliary path for extracting the binary features of the finger vein; the multi-scale fusion module of the single branch in the first stage; and the cross-branch fusion module based on ViT in the second stage.

[0041] On the one hand, the finger vein grayscale image is used as the main input. Its unique light and dark contrast shows the difference between veins and surrounding tissues, which is the key input information of vein recognition technology. In order to extract the features in the grayscale image, the main network structure is composed of five layers of convolution modules. The channel attention module is embedded in the fourth and fifth convolution layers to enhance the feature expression ability of the model.

[0042] On the other hand, the input of the auxiliary path is a binary image of finger veins, and a black-and-white image containing only the vascular venous structure is obtained through maximum entropy threshold segmentation. Since the binary image only contains the venous veins, a feature extraction network structure with lower complexity can be selected. The present invention uses five depth-separable convolutional blocks to stack and construct an auxiliary network structure, which reduces parameters and calculations while maintaining a certain feature extraction capability. At the same time, channel attention is also introduced in the fourth and fifth convolution layers, aiming to effectively extract key vascular venous features and supplement and enhance the features extracted by the main path.

[0043] In order to effectively fuse the features of the main road and the auxiliary road, the present invention first adopts the first-stage fusion strategy to enhance the expression of each branch feature. In view of the fact that the last convolution layer may ignore the details of small targets due to its large receptive field, the feature map of the previous layer rich in details is combined with the feature map of the last layer with a large receptive field, and the channel attention mechanism is introduced. Channel attention strengthens important information and weakens secondary features by adjusting the weights of each channel. Fusion after channel attention can improve the information richness of the feature map.

[0044] This module consists of two parts: channel attention weighting and feature map reorganization fusion. If the channel attention module inputs feature map The size of is C×H×W, and global average pooling and maximum average pooling are performed on each channel to obtain two feature vectors of shape C×1×1; the two vectors are then passed through a multilayer perceptron (MLP) to generate the attention weight of each channel; finally, the calculated channel weight is multiplied by each channel of the input feature map to obtain the enhanced feature map. Here, the output feature map size of the fourth layer convolution block of the two branches is 256×32×32, which is reorganized into 4 feature sub-maps of size 256×4×4, and then fused with the fifth layer output through channels to output a 1536×8×8 feature map. The weight calculation formula of the channel attention and the weighted feature map are shown in formulas (1) and (2).

[0045] M C =σ(σ(AvgPool(F))+MLP(MaxPool(F))) (11)

[0046]

[0047] Among them, M C represents the channel attention weight, σ represents the sigmoid activation operation, represents the weighted feature map and represents the dot product.

[0048] As the depth of the network increases, the features become more abstract, which can enhance the model's ability to learn discriminative features, but may also increase the difficulty of model evaluation due to the prominence of common features between different categories.

[0049] To solve this problem, the present invention adopts a second-stage fusion strategy, that is, introducing a ViT module to enhance features. This module utilizes the global feature extraction capability of the visual converter to further enhance the subtle differences between different categories.

[0050] The concatenated feature map of the two branches is used as the module input. The feature map size is 8×8, the patch size is 4×4, and it is divided into 4 patches. After each patch is flattened into a one-dimensional vector, the one-dimensional vector is mapped to a fixed dimension through a linear layer to form an embedded vector, as shown in formula (3).

[0051]

[0052] in, is the one-dimensional vector after the patch is flattened, E is the linear projection matrix, and E pos is the positional encoding, and Z0 is the initial embedding vector. The normalized embedding vector is fed into the multi-head self-attention layer to calculate the self-attention of the input sequence and capture the dependencies between different patches. Then, the MSA output is directly added to the input using a skip connection to form a residual connection to alleviate the gradient vanishing problem, as shown in formula (4).

[0053]

[0054] In the formula, is the self-attention output, LN is the normalization operation, No. The input embedding vector of the layer. After normalization, feature transformation and extraction are performed through MLP, and then a skip connection is added to add the input to the output of MLP to complete the processing of the Transformer layer, as shown in formula (5).

[0055]

[0056] In the formula, is the output embedding vector of the current layer, The index of the current layer is processed by the feedforward neural network and layer normalization, and the output of the feedforward neural network is added to the self-attention output. The whole process includes 8 Transformer layers, and finally a 1×1 convolution is performed to reduce the feature dimension. The output feature map retains the global and local key feature information of the input image, which promotes the improvement of image quality assessment.

[0057] Embodiment 2:

[0058] The beneficial effects of the present invention are illustrated by Example 2:

[0059] 1. Effectiveness of dual-branch feature fusion. The present invention conducts a performance comparison experiment on a model containing only a single branch and a model containing two branches. For different branches and different inputs, the experiment first evaluates the performance of each branch. Finally, the classification accuracy of the model containing only a single branch and the model containing two branches after fusion is compared to verify the effectiveness of the model proposed in the present invention. The comparison results are shown in Table 3, where "SD, MM, UM, HK" are the abbreviations of the data sets SDUMLA, MMCBNU_6000, USM, and HKpolyU, respectively.

[0060] Table 1 - Comparison of classification accuracy between single-branch and dual-branch models

[0061]

[0062] From the data in the table, we can see that although the classification accuracy of the single branch is similar when processing different inputs, the grayscale image input shows slightly better performance than the binary image input, mainly because the grayscale image contains more comprehensive image details and background feature information, while the finger vein binary image only highlights the vascular venous characteristics. The dual-branch structure is significantly better than the single branch because it contains both the global features of the grayscale image extracted by the main branch and the edge features of the binary image. Through complementarity, the model's ability to extract vein image quality features is enhanced, thereby improving the overall classification performance.

[0063] 2. Effectiveness of the two-stage feature fusion method. The present invention compares the different effects of the fusion modules in each stage and the attention mechanism on the classification performance. The experimental results are shown in Table 4. In the table, "MSF" means multi-scale fusion, and all "-" means that the two branches are fused by simple pixel addition in the final feature output stage.

[0064] Table 2-Effectiveness evaluation of fusion modules at each stage in the proposed model

[0065]

[0066] As can be seen from the table, the fusion strategies of the two stages have effectively improved the quality classification performance of the model. The multi-scale fusion in the first stage effectively compensates for the detail features that are ignored due to the increase in the receptive field of the model, and the introduction of channel attention further enhances the expression ability of key features. The cross-branch ViT fusion strategy in the second stage deeply integrates the rich global features in the finger vein grayscale image with the edge features in the finger vein binary image through its powerful self-attention mechanism, realizing the complementarity and enhancement of the two-way information and improving the quality classification performance of the model.

[0067] The finger vein image quality assessment method proposed in the present invention is compared with the existing typical methods. The selected comparison methods include: a quality assessment method using multiple features combined with support vectors, represented by "Hand-craft+SVM"; a quality assessment method combining feature embedding and classifiers, represented by "Patch-DNN+P-SVM"; a quality assessment method integrating SE modules with a neural network model, represented by "SE+CNN"; an assessment method integrating feature attention with a two-branch deep neural network, represented by "DSC-AFF"; and an image quality assessment method combining CNN and multi-scale HOG features, represented by "Multi-Scale HOG+light-CNN". The comparison results are shown in Table 5.

[0068] Table 3 - Comparison of performance of different quality assessment methods

[0069]

[0070] As can be seen from the table, "DSC-AFF" is better than "Hand-craft+SVM, Patch-DNN+P-SVM, SE+CNN, Multi-Scale HOG+light-CNN". This is mainly because the quality features that a single image can provide are relatively limited, which limits the performance of the evaluation. By fusing grayscale images with vein pattern images, the key features of blood vessels and veins can be enhanced on the basis of global image quality features, thereby improving the accuracy and reliability of image quality classification. The method proposed in the present invention exhibits the best classification performance. Thanks to the two-stage fusion method, the accuracy of image quality classification is further improved by mining more distinguishing features.

[0071] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A finger vein image quality assessment method based on a two-stage fusion model, characterized in that: The method comprises: Obtain a finger vein grayscale image and a binary image, input them into a constructed two-stage fusion dual-branch finger vein image quality assessment model, and judge whether it is a high-quality vein image or a low-quality one by the quality category output by the model; the two-stage fusion dual-branch finger vein image quality assessment model comprises: a main network for extracting finger vein grayscale image features, an auxiliary network for extracting finger vein binary image features, a multi-scale fusion module of a single branch in the first stage, and a cross-branch fusion module based on ViT in the second stage; Among them, in the first stage, the internal features of a single branch are optimized, and a multi-scale feature fusion strategy is adopted for both the main and auxiliary paths, and the feature maps are finely processed in combination with the channel attention mechanism to enhance the feature expression ability; in the second stage, deep fusion of cross-branch features is performed, and the ViT-based method is used to capture the global dependency between the main and auxiliary path features, and the feature information extracted from the two branches is deeply integrated.

2. According to claim 1, a finger vein image quality assessment method based on a two-stage fusion model is characterized in that: By preprocessing the grayscale image of the collected finger veins, the Sobel edge detection algorithm is used to identify the top and bottom boundaries in the image, thereby defining a region of interest (ROI). Then, the blood vessels of the acquired ROI are extracted to obtain a binary image of the finger veins.

3. According to the method for evaluating the quality of finger vein images based on a two-stage fusion model in claim 1, it is characterized in that: In the main network: the finger vein grayscale image is used as the main input, and the main network is composed of five layers of convolution modules. In the fourth and fifth convolution layers, a channel attention module is embedded. By assigning different weights to feature channels, important features are emphasized, the feature expression ability of the model is enhanced, and the key information in the vein image is captured.

4. According to the method for evaluating the quality of finger vein images based on a two-stage fusion model in claim 1, it is characterized in that: In the auxiliary network: the finger vein binary image is used as the auxiliary input, and the black-and-white image containing the vascular venous structure is obtained through the maximum entropy threshold segmentation method. The auxiliary network uses five depth-separable convolution blocks for stacking, and introduces a channel attention module in the fourth and fifth convolution layers to strengthen the extraction of key vascular venous features and supplement and enhance the features extracted by the main network.

5. According to the method for evaluating the quality of finger vein images based on a two-stage fusion model as described in claim 1, it is characterized in that: The first-stage single-branch multi-scale fusion module consists of two parts: channel attention weighting and feature map reorganization fusion; The size of the input feature map F of the channel attention module is determined to be C×H×W. Global average pooling and maximum average pooling are performed on each channel to obtain two feature vectors of shape C×1×1. The two vectors are then passed through a multi-layer perceptron MLP to generate the attention weight of each channel. Finally, the calculated channel weight is multiplied by each channel of the input feature map to obtain the enhanced feature map. Here, the output feature map size of the fourth layer convolution block of the two branches is 256×32×32, which is reorganized into 4 feature sub-maps of size 256×4×4, and then fused with the fifth layer output to output a 1536×8×8 feature map. The weight calculation formula of the channel attention and the weighted feature map are shown in formulas (1) and (2): M C =σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (1) Among them, M C represents the channel attention weight, σ represents the sigmoid activation operation, represents the weighted feature map and represents the dot product.

6. According to the method for evaluating the quality of finger vein images based on a two-stage fusion model in claim 1, it is characterized in that: In the second stage, based on ViT cross-branch fusion, the feature map after the two branches are concatenated is used as the module input. The feature map size is 8×8, the patch size is 4×4, and it is divided into 4 patches. After each patch is flattened into a one-dimensional vector, the one-dimensional vector is mapped to a fixed dimension through a linear layer to form an embedded vector, as shown in formula (3): in, is the one-dimensional vector after the patch is flattened, E is the linear projection matrix, and E pos is the position encoding, Z0 is the initial embedding vector, and the normalized embedding vector is sent to the multi-head self-attention layer to calculate the self-attention of the input sequence and capture the dependencies between different patches. Then, the MSA output is directly added to the input using a skip connection to form a residual connection to alleviate the gradient disappearance, as shown in formula (4): Z′ l =MSA(LN(Z l-1 ))+Z l-1 (4) In the formula, Z′ l is the self-attention output, LN is the normalization operation, Z l-1 The input embedding vector of the l-1th layer, Z′ l After normalization, feature transformation and extraction are performed through MLP, and then a skip connection is added to add the input to the output of MLP to complete the processing of the Transformer layer, as shown in formula (5): WITH l =MLP(LN(Z′ l ))+Z′ l (5) In the formula, Z l is the output embedding vector of the current layer, l is the index of the current layer. After the feedforward neural network and layer normalization, the output of the feedforward neural network is added to the self-attention output. The whole process includes 8 Transformer layers. Finally, a 1×1 convolution is performed to reduce the feature dimension. The output feature map retains the global and local key feature information of the input image, which promotes the improvement of image quality assessment effect.

Images