Non-contact palmprint recognition method based on hand shape semantic prior and ViT

Abstract

The application provides a non-contact palmprint recognition method based on hand shape semantic prior and ViT, which aims at the problem of poor user friendliness in the traditional ROI extraction method, adopts a new ROI region division method, and positions the ROI region by two reference points located at the edges of the palm; meanwhile, a network key point positioning network (Key-Point Location Regression Network, KPLRN) is proposed to accurately position the key points, and the ROI can be accurately extracted from the palm image in any direction, and the requirement for the palm posture is also reduced. In view of the technical problem of low recognition accuracy in the traditional recognition method, a new recognition network structure is built for the palmprint recognition task, that is, an inverted-bottleneck block recognition network (Inverted-Bottleneck Block based Recognition Network, IBRN), and by adopting a suitable training strategy, the recognition accuracy is broken through, and comparison and verification are carried out on multiple databases. The application can be widely applied in identity authentication.

Description

Technical Field

[0001] This invention relates to an identity authentication method, and more particularly to a non-contact palmprint recognition method based on hand shape semantic prior and ViT. Background Technology

[0002] Compared to traditional knowledge- or identifier-based authentication methods, including password authentication (where users authenticate by entering a correct password), smart card authentication (where users use a smart card as an identifier and authenticate through a security chip within the card), and hardware token authentication (where users use hardware tokens such as USB dongles or OTP tokens for authentication), biometric identification technology offers higher security, confidentiality, and ease of use. Furthermore, it is not forgotten or lost, making it widely used in various scenarios such as administrative offices, financial payments, social security, and IoT device authentication systems.

[0003] As a member of the biometric family, palm prints have advantages such as complexity, uniqueness, and stability. Palm print recognition technology is being used in scenarios such as security monitoring, identity authentication, and mobile payment, and can provide more reliable evidence, showing broad development prospects.

[0004] Currently, contact palmprint recognition technology, which uses specialized equipment to collect palmprints for identification, is mature and widely used. However, because this technology requires direct contact between the user and the device, it may be affected by factors such as dirt, oil, and sweat, impacting image quality and recognition effectiveness. Furthermore, since contact palmprint recognition devices are mostly used in public places, they may pose hygiene problems. Moreover, the stringent requirements for specialized equipment directly result in high application costs for this technology.

[0005] To overcome the limitations of contact-based palmprint recognition technology, non-contact palmprint recognition technology, which uses palmprint images captured by digital cameras or smartphone cameras for identification, has been widely studied. This type of method mainly involves two steps:

[0006] 1. Extracting the Region of Interest (ROI) from the palm image. Typically, an ROI is a square region with rich texture features in the center of the palm. The operation of extracting this square region with rich texture features in the center of the palm is referred to as ROI extraction in the following description. A good ROI extraction method is fundamental to accurate recognition, requiring that the same palm region be extracted from different images of the same palm taken under different backgrounds and poses.

[0007] 2. Select an image feature extraction method to extract features from the ROI image, and compare the extracted features to obtain the recognition result. In the following description, this step is referred to as ROI recognition.

[0008] Research on non-contact palmprint recognition has also progressed gradually. Initially, the source images used for recognition were palm images taken under constrained scenarios (using a uniform solid color background and a uniform light source, with a fixed shooting angle and shooting device position, requiring the palm to be flat, fingers spread, and pointing directly upwards during shooting). Researchers proposed a series of ROI extraction methods based on traditional image processing techniques, such as Harris corner detection and Canny edge detection algorithms, to extract the ROIs required for subsequent steps.

[0009] In recent years, in order to increase the user-friendliness of non-contact palmprint recognition technology and make it applicable to more scenarios, researchers have begun to study non-contact palmprint recognition methods that use palm images taken in unconstrained scenarios (captured by the camera of mobile devices such as mobile phones, with no background restrictions, natural light or mobile phone flash as the light source, relatively relaxed requirements for shooting angle and shooting device position, and the palm naturally unfolded to ensure that the complete palm part can be captured) as the source image for recognition.

[0010] In unconstrained scenarios, the original ROI extraction methods are no longer applicable, and there is an urgent need to propose a new ROI extraction method that can extract the accurate ROI region from hand images captured in unconstrained scenarios.

[0011] like Figure 1 The image shown is a comparison of hand images acquired under constrained and unconstrained scenarios. Figure 1 In the image, the two images on the left are typical hand images taken in constrained scenarios, while the two images on the right are typical hand images taken in unconstrained scenarios.

[0012] In the second step of palmprint recognition, the quality of Region of Interest (ROI) extraction directly impacts recognition accuracy. Traditional methods include subspace learning-based methods, orientation-based encoding methods, texture-based methods, and correlation filter-based methods. Due to the superior performance of deep learning technology in a range of computer vision tasks in recent years, much research currently focuses on using deep learning methods for recognition, achieving better results than traditional methods. The success of deep learning methods stems from the design of the network structure and related training methods. By designing and training a Convolutional Neural Network (CNN) structure more suitable for palmprint recognition tasks, the recognition accuracy can be further improved compared to existing deep learning and non-deep learning methods, making non-contact palmprint recognition technology safer and more reliable.

[0013] The shortcomings of existing methods are mainly as follows:

[0014] 1. Existing ROI extraction methods have certain shortcomings. When processing images taken in unconstrained scenes, due to the complex background, traditional ROI extraction methods designed for images taken in constrained scenes cannot accurately determine the outline of the hand, and therefore cannot extract accurate ROI regions.

[0015] 2. Moreover, previous ROI extraction methods often relied on the finger valley point as a reference point during the extraction process, and the related algorithms for hand shape processing during extraction were very sensitive to the spatial position of the palm. They had strict requirements on the posture and palm direction when the image was taken (the five fingers must be spread and together, and the palm must be facing directly upwards), resulting in low user-friendliness and low universality.

[0016] 3. Traditional recognition methods are poorly robust to changes in displacement and lighting, resulting in low recognition accuracy and requiring manual parameter adjustments based on experience. Current deep learning-based recognition methods often directly use existing network structures for transfer learning, without designing unique network structures specifically for palmprint recognition. Designing new network structures for this specific task could improve recognition accuracy while reducing unnecessary network parameters, thus reducing computational overhead and increasing efficiency. Summary of the Invention

[0017] The technical problem solved by this invention is the inability to accurately extract the Region of Interest (ROI) from hand images captured in unconstrained scenarios. It proposes to train a Palm Region Segmentation Network (PSRN) using prior information on hand shape semantics to achieve accurate segmentation of the hand region from the background in the image, thus improving the accurate localization of the ROI.

[0018] Therefore, the technical solution of this invention is a non-contact palmprint recognition method based on hand shape semantic prior and ViT. This method uses two deep learning models, the palm segmentation network PSRN and the key point location regression network KPLRN, to extract the ROI region, including the following steps:

[0019] Step 1. Define the ROI region and train the deep learning model involved in adaptive ROI extraction;

[0020] Step 1-1. Define the ROI region of the palm, set two key reference points, and determine the relationship between the reference point positions and the ROI region calculation;

[0021] Step 1-2. Build the PRSN. The PRSN consists of all the downsampling layers of the FCN-32S network connected to 4 VisionTransformer Blocks, and finally connected to the upsampling layer of the FCN-32S.

[0022] Then, using the existing database of hand images and background images captured under constrained scenarios, training samples and labels are automatically generated. The binary cross-entropy loss is used as the loss function to train the PRSN, enabling it to segment hand regions from the original image.

[0023] Steps 1-3. Build KPLRN. KPLRN uses the first three blocks of the VGG16 network as the backbone of the network. In order to realize the regression of key point positions, an auxiliary layer consisting of a fully connected layer, a ReLU activation function layer, a normalization layer, and a fully connected layer is designed. The backbone is connected to the auxiliary layer to form KPLRN.

[0024] Then, a palm region image with a black background was selected as the training sample, and the key point positions were manually labeled as labels. The mean squared error loss was used as the loss function to train KPLRN so that it could output the coordinates of the two key reference points defined in step 1-1 for any input palm region image with a black background.

[0025] Step 2. Build and train the IBRN recognition network:

[0026] Step 2-1. Constructing the IVBN model: The IVBN model is built based on the Inverted Bottleneck Block structure proposed in the ConvNeXt network;

[0027] IBRN uses a total of 12 Inverted Bottleneck Blocks and four ordinary convolutional layers to form the backbone network;

[0028] Step 2-2. Train the IBRN model:

[0029] After the model was built, palmprint ROI images were used as training samples, and the palm number to which the ROI belonged was used as the label to train the model, so that it has good palmprint feature extraction ability.

[0030] Step 3. Apply the trained network to palmprint recognition:

[0031] Step 3-1. Input the original image P of the hand with a natural background into PRSN. PRSN outputs a binary mask Mask. In the Mask, white pixels with a value of 1 represent the hand area, and black pixels with a value of 0 represent the background area. Perform a pixel-by-pixel AND operation between the Mask and the original image to obtain an image P1 with only a black background and the hand area.

[0032] Step 3-2. Input P1 into the keypoint location regression network KPLRN, and the network outputs the position coordinates of two keypoints;

[0033] Step 3-3. Based on the key point positions obtained in the previous step, the coordinates of the four vertices of the ROI region and the angle θ that the image needs to be rotated are calculated. Then, the image P1 is rotated according to θ and the coordinates of the four vertices of the rotated ROI region are calculated. Finally, the ROI region is directly cropped from the rotated P1 based on the rotated coordinates. The cropped ROI region can be directly used for subsequent recognition.

[0034] Steps 3-4. Use the trained IBRN for palmprint recognition: For the input palmprint ROI image, the trained IBRN will output a 512-dimensional feature vector. This feature vector is then compared with the feature vectors in the previously extracted palmprint feature database using a specified standard to obtain the final recognition result.

[0035] Preferably, in step 1-1, the detailed steps for defining the ROI region and selecting key points are as follows:

[0036] Select the two outermost points on the horizontal creases at the base of the index and little fingers of the palm as keypoints D1 and D2. After obtaining the coordinates of D1 and D2, denote the line connecting D1 and D2 as L, with length l. Select the point on this line that is 0.15l from D1 as D3, and the point that is 0.85l from D1 as D4. Using D3 and D4 as two vertices, construct a square with a side length of 0.7L downwards, which will be the final ROI. Figure 3 The area within the Chinese box. Taking the direction from P1 to P2 as the positive direction, the angle between L and the horizontal axis can be denoted as θ, which is used for image rotation.

[0037] Preferably, in steps 1-2, the detailed information of the PRSN network is as follows:

[0038] 1. The input layer takes a hand image with dimensions of 224×224×3 as input, where the three parameters correspond to the height, width, and number of pixels and channels of the image, respectively. Since an RGB image is used as input, the number of input channels is 3. After passing through the feature extraction layer of the fully convolutional network FCN-32S segmentation network, a feature map of 7×7×512 is obtained.

[0039] 2. The feature map is then fed into four consecutive ViT blocks for semantic feature learning, and finally outputs a 49×512 dimensional feature vector.

[0040] 3. After transforming the feature vector into a feature map of size 7×7×512, it is fed into the upsampling layer of the FCN-32S network. The upsampling layer is implemented by transposed convolution. In this embodiment, a direct connection is added between the feature extraction layer and the upsampling layer to increase information flow. Finally, the upsampling layer outputs a feature map of size 224×224×21.

[0041] 4. The final feature map is processed using a 2D convolution with a kernel size of 1×1, 21 input channels, and 2 output channels, resulting in a final segmentation result of 224×224×2. The three numbers correspond to the number of pixels in the height, the number of pixels in the width, and the number of channels of the image, respectively. In the output of the two channels, the first channel corresponds to the confidence score of the pixel belonging to the background class, and the second channel corresponds to the confidence score of the pixel belonging to the palm region class.

[0042] 5. For each pixel position in the segmentation result, take the label corresponding to the larger value in the two channels to obtain the final black and white binary mask.

[0043] Preferably, the automatic generation of PRSN training samples in steps 1-2 is described in detail below:

[0044] 1. Use the original palm images from the palmprint database collected under multiple constrained scenarios as the palm image library; use the images from the object detection database as the background image library;

[0045] 2. Randomly select an image I from the palm image source library, first convert I to a Cr channel, and denote it as I0. cr , to obtain I cr The corresponding image histogram records the number of pixels for each value in the image;

[0046] 3. The histogram uses the Otsu's method to calculate the threshold for image binarization; this threshold is then used to binarize I... cr Binarization is performed, where pixel values ​​less than a threshold are changed to 0, and otherwise to 1; this yields a binary mask M1 that distinguishes the background region from the palm region; since the background color in the original image is uniform, the values ​​obtained using the OTSU algorithm can effectively distinguish the palm region pixels from the background pixels.

[0047] 4. Obtain an image I2 containing only the palm portion by performing a pixel-by-pixel AND operation between the label and the original image;

[0048] 5. Generate an affine transformation parameter matrix K, randomly selecting a rotation angle θ∈(-π,π), a scaling factor α∈(0.7,1.3), a horizontal offset x∈(-W / 4,W / 4) for the transformed image, and a vertical offset y∈(-H / 4,H / 4) for the transformed image; W and H are the width and height of the original image, and the generated matrix K;

[0049]

[0050] 6. Data augmentation: M2 and I3 are obtained by performing an affine transformation with parameter K on M1 and I2. The position and orientation of the hand can be randomly changed, which helps to improve the generalization ability and robustness of the recognition model.

[0051] 7. Randomize the brightness of I3 to obtain I4. Randomly select an image B from the background image library and cover the corresponding position of the palm area of ​​I4 in B to obtain the final training sample image I5, whose corresponding training label is M2.

[0052] Preferably, in steps 1-3, the detailed information of the KPLRN network is as follows:

[0053] 1. Input layer: The input is a hand image with only a black background, with a size of 56×56×3. After passing through the backbone layer of the network, a feature map of 7×7×256 is obtained, where 7×7 represents the height and width of the feature map, and 256 is the number of channels of the feature map.

[0054] 2. The feature map is then processed by a global average pooling layer, becoming a 1×256 one-dimensional feature vector, which is then fed into fully connected layer 1, outputting a 1×64-dimensional feature vector. This vector is then processed by the ReLU activation function, normalized, and fed into FC2, finally yielding a 1×4 coordinate vector, representing the x and y coordinates of the two selected key points; the values ​​of the four output coordinates are all normalized to the range (-1, 1).

[0055] 3. In KPLRN, the dropout rates of Dropout1 and Dropout2 are set to 0.4 and 0.2, respectively, to prevent overfitting of the model.

[0056] Preferably, in step 2-1, the detailed information of the Inverted Bottleneck Block is as follows:

[0057] 1. Input a feature map of size H×W×C, first pass it through a depthwise separable convolution with a kernel size of 7. This depthwise separable convolution layer uses Layer Normalization for normalization and does not set an activation function. Output a feature map of size H×W×C.

[0058] 2. Feature map output from depthwise separable convolution: Input a regular convolutional layer with a kernel size of 1, use GELU as the activation function, and output an H×W×4C feature map;

[0059] 3. The H×W×4C feature map is passed through a convolutional layer with a kernel size of 1. No activation function is set. The output feature map is of size H×W×C and added to the initial input feature map to form the final output.

[0060] The beneficial effect of this invention is that, in response to the technical problem that the Region of Interest (ROI) cannot be accurately extracted when shooting palm images in unconstrained scenarios, it proposes to train a Palm Region Segmentation Network (PSRN) using hand shape semantic prior information to achieve the effect of accurately segmenting the palm region and the background in the image, thus advancing the accurate localization of ROI regions.

[0061] Meanwhile, to address the issue of poor user-friendliness in traditional ROI extraction methods, a new ROI region segmentation method is adopted, using two reference points located at the edge of the palm to locate the ROI region; at the same time, a Key-Point Location Regression Network (KPLRN) is proposed to accurately locate key points, which can accurately extract ROIs for palm images of any orientation and also reduces the requirements for palm pose.

[0062] To address the technical problem of low recognition accuracy in traditional recognition methods, a new recognition network structure, the Inverted-Bottlenck Block based Recognition Network (IBRN), was developed for palmprint recognition tasks. By employing an appropriate training strategy, a breakthrough in recognition accuracy was achieved, and the results were verified on multiple databases. Attached Figure Description

[0063] Figure 1 This is a schematic diagram comparing hand images acquired under constrained and unconstrained scenarios, as mentioned in the background description of this invention.

[0064] Figure 2 This is a schematic diagram of the ROI extraction process of the present invention;

[0065] Figure 3 This is a schematic diagram illustrating the delineation of the ROI region and the selection of key points in this invention;

[0066] Figure 4 This is a schematic diagram of the PRSN training sample generation process of the present invention;

[0067] Figure 5This is a schematic diagram of the Inverted Bottleneck Block structure of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to embodiments.

[0069] This invention provides a non-contact palmprint recognition method based on hand shape semantic prior and ViT. VisionTransformer is a model that applies Transformer to computer vision tasks and incorporates a self-attention mechanism. This method employs two deep learning models: the palm segmentation network PSRN and the keypoint location regression network KPLRN, for region of interest (ROI) extraction. To verify the quality of the ROI images extracted using the proposed method, the deep learning network IBRN is built and trained for final ROI recognition. The non-contact palmprint recognition method based on hand shape semantic prior and ViT includes the following steps:

[0070] Step 1. Define the ROI region and train the deep learning model involved in adaptive ROI extraction;

[0071] Step 1-1. Delineate the ROI region of the palm, set two key reference points, and determine the relationship between the reference point positions and the ROI region calculation; the specific implementation method is as follows:

[0072] In step 1-1, the detailed steps for defining the ROI region and selecting key points are as follows:

[0073] like Figure 3 The diagram shown is a schematic diagram of the delineation of the ROI region and the selection of key points in an embodiment of the present invention.

[0074] Two points on the outermost horizontal creases at the base of the index and little fingers of the palm are selected as key points D1 and D2. After obtaining the coordinates of D1 and D2, the line connecting D1 and D2 is denoted as L, with a length of l. In this embodiment, the point on the line 0.15l away from D1 is selected as D3, and the point 0.85l away is selected as D4. Using D3 and D4 as two vertices, a square with a side length of 0.7L is constructed downwards, which serves as the final ROI; that is... Figure 3 The area within the square frame. Taking the direction from P1 to P2 as the positive direction, the angle between L and the horizontal axis (taking the direction from left to right as the positive direction) can be denoted as θ, which is used for image rotation.

[0075] Steps 1-2. Construct the PRSN. The PRSN consists of all the downsampling layers (five convolutional layers in total) of the FCN-32S network connected to four Vision Transformer Blocks, and finally connected to the upsampling layer (transposed convolutional layer) of the FCN-32S network. Detailed information about the PSRN is as follows:

[0076] In steps 1-2, the detailed information of the PRSN network is as follows:

[0077] 1. The input layer takes a hand image with dimensions of 224×224×3 as input, where the three parameters correspond to the height, width, and number of pixels and channels of the image, respectively. Since an RGB image is used as input, the number of input channels is 3. After passing through the feature extraction layer of the Fully Convolutional Networks Small (FCN-32S) segmentation network, a 7×7×512 feature map is obtained.

[0078] 2. The feature map is then fed into four consecutive ViT (Vision Transformer Blocks) for semantic feature learning, and finally outputs a 49×512 dimensional feature vector.

[0079] 3. After transforming the feature vector into a feature map of size 7×7×512, it is fed into the upsampling layer of the FCN-32S network. The upsampling layer is implemented by transposed convolution. In this embodiment, a skip connection is added between the feature extraction layer and the upsampling layer to increase information flow. Finally, the upsampling layer outputs a feature map of size 224×224×21.

[0080] 4. The final feature map is processed using a 2D convolution with a kernel size of 1×1, 21 input channels, and 2 output channels, resulting in a final segmentation result of 224×224×2. The three numbers correspond to the number of pixels in the height, the number of pixels in the width, and the number of channels of the image, respectively. In the output of the two channels, the first channel corresponds to the confidence score of the pixel belonging to the background class, and the second channel corresponds to the confidence score of the pixel belonging to the palm region class.

[0081] 5. For each pixel position in the segmentation result, take the label corresponding to the larger value in the two channels to obtain the final black and white binary mask;

[0082] Then, using existing databases of hand images captured under constrained scenarios and background data, training samples and labels are automatically generated. Binary Cross Entropy Loss is used as the loss function to train the PRSN, enabling it to segment hand regions from the original images. The automatic generation method for training samples and labels is as follows:

[0083] The automatic generation of PRSN training samples in steps 1-2 is explained in detail below:

[0084] like Figure 4The diagram shown illustrates the steps and process of generating PRSN training samples, specifically described as follows:

[0085] 1. The original palm images collected under four constrained scenarios—the palm print databases of the Hong Kong Polytechnic University (PolyU), Beijing Jiaotong University (BJTU), the Centre of Excellence in Pattern Recognition (COEP) at Pune University, India, and the Real and Simulated Touch (REST) ​​palm print database at the University of California, San Diego—were used as the palm image library; images from the Microsoft CommonObjects in Context 2017 (COCO2017) database were used as the background image library.

[0086] 2. Randomly select an image I from the palm image source library, first convert I to a Cr channel, and denote it as I0. cr , to obtain I cr The corresponding image histogram records the number of pixels for each value in the image;

[0087] 3. The histogram uses Otsu's method to calculate the threshold for image binarization; this threshold is then used to adjust I... cr Binarization is performed, where pixel values ​​less than a threshold are changed to 0, and otherwise to 1; this yields a binary mask M1 that distinguishes the background region from the palm region; since the background color in the original image is uniform, the values ​​obtained using the OTSU algorithm can effectively distinguish the palm region pixels from the background pixels.

[0088] 4. Obtain an image I2 containing only the palm portion by performing a pixel-by-pixel AND operation between the label and the original image;

[0089] 5. Generate an affine transformation parameter matrix K, randomly selecting a rotation angle θ∈(-π,π), a scaling factor α∈(0.7,1.3), a horizontal offset x∈(-W / 4,W / 4) for the transformed image, and a vertical offset y∈(-H / 4,H / 4) for the transformed image; W and H are the width and height of the original image, and the generated matrix K;

[0090]

[0091] 6. Data augmentation: M2 and I3 are obtained by performing an affine transformation with parameter K on M1 and I2. The position and orientation of the hand can be randomly changed, which helps to improve the generalization ability and robustness of the recognition model.

[0092] 7. Randomize the brightness of I3 to obtain I4. Randomly select an image B from the background image library and cover the palm area of ​​I4 at the corresponding position of B to obtain the final training sample image I5, whose corresponding training label is M2.

[0093] Steps 1-3. Build KPLRN. KPLRN uses the first three blocks of the VGG16 network as the backbone. To achieve keypoint regression, an auxiliary layer consisting of a fully connected layer (FC1), a ReLU activation function layer (ReLU), a normalization layer (Layer Norm), and a fully connected layer (FC2) is designed. The backbone is connected to the auxiliary layer to form KPLRN. A detailed description of KPLRN is as follows:

[0094] The detailed information about the KPLRN network in steps 1-3 is as follows:

[0095] 1. Input layer: The input is a hand image with only black background and size of 56×56×3. After passing through the backbone layer of the network (the first three convolutional blocks of VGG16), a feature map of 7×7×256 is obtained, where 7×7 represents the height and width of the feature map and 256 is the number of channels of the feature map.

[0096] 2. The feature map is then processed by a global average pooling layer, becoming a 1×256 one-dimensional feature vector, which is then fed into fully connected layer 1 (FC1), outputting a 1×64-dimensional feature vector. This vector is then processed by the ReLU activation function, normalized, and fed into FC2, finally yielding a 1×4 coordinate vector, representing the x and y coordinates of the two selected key points; the values ​​of the four output coordinates are all normalized to the range (-1, 1).

[0097] 3. In KPLRN, the dropout rates of Dropout1 and Dropout2 are set to 0.4 and 0.2, respectively, to prevent overfitting of the model;

[0098] Then, a palm region image with a black background (PRSN output) was selected as the training sample, and the key point positions were manually labeled as labels. The mean square error loss was used as the loss function to train KPLRN so that it could output the coordinates of the two key reference points defined in step 1-1 for any input palm region image with a black background.

[0099] Step 2. Build and train the IBRN recognition network:

[0100] Step 2-1. Constructing the IVBN model: The IVBN model is built based on the Inverted Bottleneck Block structure proposed in the ConvNeXt network. The specific description of this structure is as follows:

[0101] In step 2-1, the detailed information for the Inverted Bottleneck Block is as follows:

[0102] 1. Input a feature map of size H×W×C, first pass it through a depthwise separable convolution with a kernel size of 7. The normalization operation of this depthwise separable convolution layer adopts layer normalization and no activation function is set; output a feature map of size H×W×C.

[0103] 2. Feature map output from depthwise separable convolution: Input a regular convolutional layer with a kernel size of 1, use GELU as the activation function, and output an H×W×4C feature map;

[0104] 3. The H×W×4C feature map is passed through a convolutional layer with a kernel size of 1 (without setting an activation function), and the output feature map of size H×W×C is added to the initial input feature map to form the final output;

[0105] IBRN employs a backbone network consisting of a total of 12 Inverted Bottleneck Blocks and four ordinary convolutional layers (Conv). Specific parameters and their arrangement are detailed below:

[0106] like Figure 5 The diagram shown is of the Inverted Bottleneck Block structure:

[0107] In the diagram, DWConv (Depthwise Conv) represents depthwise separable convolution, Conv represents regular convolution operation, and 7×7 represents a kernel size of 7.

[0108] In step 2-1, the detailed structure and parameter settings of the IBRN network are shown in Table 1:

[0109] Table 1. IBRN Network Structure

[0110]

[0111] The backbone network's auxiliary connection block consists of a global average pooling layer and a fully connected layer (FC1) arranged sequentially. During training, an additional fully connected layer (FC2) is added for calculating the loss function, and a Dropout layer is added to suppress overfitting of the model.

[0112] Step 2-2. Train the IBRN model:

[0113] After the model is built, palmprint ROI images are used as training samples, and the palm number to which the ROI belongs is used as the label to train the model, so that it has good palmprint feature extraction ability. The specific training strategy is described in the specific implementation section.

[0114] Step 3. Apply the trained network to palmprint recognition:

[0115] Step 3-1. Input the original image P of the hand with a natural background into PRSN. PRSN outputs a binary mask Mask. In the Mask, white pixels with a value of 1 represent the hand area, and black pixels with a value of 0 represent the background area. Perform a pixel-by-pixel AND operation between the Mask and the original image to obtain an image P1 with only a black background and the hand area.

[0116] Step 3-2. Input P1 into the keypoint location regression network KPLRN, and the network outputs the position coordinates of two keypoints;

[0117] Step 3-3. Based on the keypoint positions obtained in the previous step, the coordinates of the four vertices of the ROI region and the angle θ that the image needs to be rotated (see attached diagram for definition) can be calculated. Then, the image P1 is rotated according to θ, and the coordinates of the four vertices of the rotated ROI region are calculated. Finally, the ROI region is directly cropped from the rotated P1 based on the rotated coordinates. The cropped ROI region can be directly used for subsequent recognition. The entire ROI extraction process is as follows: Figure 2 As shown;

[0118] Steps 3-4. Use the trained IBRN for palmprint recognition: For the input palmprint ROI image, the trained IBRN will output a 512-dimensional feature vector. This feature vector is then compared with the feature vectors in the previously extracted palmprint feature database using specified standards, such as Euclidean distance or cosine similarity, to obtain the final recognition result.

[0119] Step 4. Validation:

[0120] To verify the feasibility of the palmprint ROI extraction method proposed in this embodiment, two publicly available unconstrained scene palmprint databases were selected for verification. For each database, the ROI extraction method proposed in this embodiment was first used to extract the ROI, and then the IBRN recognition network proposed in this embodiment was used to identify the ROI and calculate the final recognition accuracy.

[0121] To verify the effectiveness of the IBRN recognition network proposed in this invention, this invention was compared with existing recognition methods on several palmprint databases.

[0122] In step 4, the results of the invention validity verification are explained as follows:

[0123] Palmprint recognition tasks can be further divided into identification and verification. Identification tasks can be divided into two categories: identification and verification. Identification refers to whether a given palmprint ROI can be found and the corresponding palm of the individual can be found. Verification refers to whether a pair of ROIs can be given and whether the two ROIs belong to the same palm.

[0124] For palmprint recognition tasks, the evaluation metric is accuracy (ACC), which represents the proportion of correctly classified samples out of the total number of samples; the higher the ACC, the better the recognition performance. The ACC calculation formula is as follows:

[0125]

[0126] In the above formulas, TP represents the number of True Positives, i.e., the number of samples that are actually positive and correctly classified by the classifier; TN represents the number of True Negatives, i.e., the number of samples that are actually negative and correctly classified by the classifier; FP represents the number of False Positives, i.e., the number of samples that are actually negative but are incorrectly classified as positive by the classifier; and FN represents the number of False Negatives, i.e., the number of samples that are actually positive but are incorrectly classified as negative by the classifier.

[0127] For palmprint verification tasks, the evaluation metric is the Equal Error Rate (EER). EER refers to the value at which the False Positive Rate (FPR) and False Negative Rate (FNR) are equal in the Receiver Operating Characteristic (ROC) curve. FPR refers to the proportion of samples that are actually negative but misclassified as positive by the classifier, and FNR refers to the proportion of samples that are actually positive but misclassified as negative. The formulas for calculating FPR and FNR are as follows:

[0128]

[0129] In the above formula, FP represents the number of false positives, that is, the number of samples that are actually negative examples but are misclassified as positive examples by the classifier; TN represents the number of true negatives, that is, the number of samples that are actually negative examples and are correctly classified by the classifier.

[0130]

[0131] In the formula, TP represents the number of True Positives, which is the number of samples that are actually positive and correctly classified by the classifier; FN represents the number of False Negatives, which is the number of samples that are actually positive but are incorrectly classified as negative by the classifier.

[0132] If the EER is high, it means that the recognition system performs worse in a certain aspect and needs further optimization; therefore, the lower the EER value, the better the performance of the classification system.

[0133] First, for two unconstrained databases—the Harbin Institute of Technology (HIT) palmprint database and the Tongji University Multi-purpose Palmprint Database (MPD)—the ROI extraction method proposed in this patent was used for ROI extraction, and the IBRN network was used for recognition. The ratio of the training set to the validation set was set to 1:1. The results were compared with existing palmprint recognition methods, including ROI extraction and subsequent recognition, as shown in Table 2. The palmprint recognition method proposed in this patent showed good experimental results on both databases, and its performance on the MPD database was superior to existing methods. This also proves that the ROI quality extracted by the proposed ROI extraction method can meet the requirements of subsequent recognition.

[0134] Table 2. Experimental results of the palmprint recognition method proposed in this invention.

[0135]

[0136] To verify the recognition accuracy of the IBRN network independently, comparative experiments were conducted on four databases to compare it with existing methods. The ROIs used on the MPD and HIT databases were extracted using the ROI extraction method proposed in this invention. Since the Xi'an Jiaotong University Unconstrained Palmprint Database (XJU_UP) and the Indian Institute of Technology Delhi (ITD) palmprint database do not publicly disclose original palm images, ROI images provided by the database creators were used. The comparison results are shown in Table 3. It can be seen that the IBRN network's recognition results are superior to existing methods, demonstrating that the IBRN network structure proposed in this invention can be well applied to palmprint recognition tasks in unconstrained environments and has practical application value.

[0137] Table 3. Comparison of IBRN with Existing Identification Methods

[0138]

[0139]

[0140] This invention addresses the technical problem of inaccurate Region of Interest (ROI) extraction when capturing hand images in unconstrained scenarios. It proposes training a Palm Region Segmentation Network (PRSN) using prior information on hand shape semantics to achieve accurate segmentation of the hand region from the background in the image, thus advancing the accurate localization of ROI regions.

[0141] Meanwhile, to address the issue of poor user-friendliness in traditional ROI extraction methods, a new ROI region segmentation method is adopted, using two reference points located at the edge of the palm to locate the ROI region; at the same time, a Key-Point Location Regression Network (KPLRN) is proposed to accurately locate key points, which can accurately extract ROIs for palm images of any orientation and also reduces the requirements for palm pose.

[0142] To address the technical problem of low recognition accuracy in traditional recognition methods, a new recognition network structure, the Inverted-Bottleneck Block based Recognition Network (IBRN), was developed for palmprint recognition tasks. By employing an appropriate training strategy, a breakthrough in recognition accuracy was achieved, and the results were verified on multiple databases.

[0143] However, the above description is merely a specific embodiment of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any substitution of equivalent components or equivalent changes and modifications made in accordance with the scope of protection of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A non-contact palmprint recognition method based on hand shape semantic prior and ViT, characterized by: This method employs two deep learning models, the palm segmentation network PRSN and the keypoint location regression network KPLRN, for region of interest (ROI) extraction, including the following steps: Step 1. Define the ROI region and train the deep learning model involved in adaptive ROI extraction; Step 1-1. Define the ROI region of the palm, set two key reference points, and determine the relationship between the reference point positions and the ROI region calculation; Step 1-2. Build the PRSN. The PRSN consists of all the downsampling layers of the FCN-32S network connected to 4 VisionTransformer Blocks, and finally connected to the upsampling layer of the FCN-32S. Then, using the existing database of hand images and background images captured under constrained scenarios, training samples and labels are automatically generated. The binary cross-entropy loss is used as the loss function to train the PRSN, enabling it to segment hand regions from the original image. Steps 1-3. Build KPLRN. KPLRN uses the first three blocks of the VGG16 network as the backbone of the network. In order to realize the regression of key point positions, an auxiliary layer consisting of a fully connected layer, a ReLU activation function layer, a normalization layer, and a fully connected layer is designed. The backbone is connected to the auxiliary layer to form KPLRN. Then, a palm region image with a black background was selected as the training sample, and the key point positions were manually labeled as labels. The mean squared error loss was used as the loss function to train KPLRN so that it could output the coordinates of the two key reference points defined in step 1-1 for any input palm region image with a black background. Step 2. Build and train the IBRN recognition network: Step 2-1. Construct the IBRN model: The IBRN model is built based on the Inverted BottleneckBlock structure proposed in the ConvNeXt network; IBRN uses a total of 12 Inverted Bottleneck Blocks and four ordinary convolutional layers to form the backbone network; Step 2-2. Train the IBRN model: After the model was built, palmprint ROI images were used as training samples, and the palm number to which the ROI belonged was used as the label to train the model, so that it has good palmprint feature extraction ability. Step 3. Apply the trained network to palmprint recognition: Step 3-1. Input the original image P of the hand with a natural background into PRSN. PRSN outputs a binary mask Mask. In the Mask, white pixels with a value of 1 represent the hand area, and black pixels with a value of 0 represent the background area. Perform a pixel-by-pixel AND operation between the Mask and the original image to obtain an image P1 with only a black background and the hand area. Step 3-2. Input P1 into the keypoint location regression network KPLRN, and the network outputs the position coordinates of two keypoints; Step 3-3. Based on the key point positions obtained in the previous step, the coordinates of the four vertices of the ROI region and the angle θ that the image needs to be rotated are calculated. Then, the image P1 is rotated according to θ and the coordinates of the four vertices of the rotated ROI region are calculated. Finally, the ROI region is directly cropped from the rotated P1 based on the rotated coordinates. The cropped ROI region can be directly used for subsequent recognition. Steps 3-4. Use the trained IBRN for palmprint recognition: For the input palmprint ROI image, the trained IBRN will output a 512-dimensional feature vector. This feature vector is then compared with the feature vectors in the previously extracted palmprint feature database using a specified standard to obtain the final recognition result.

2. The non-contact palmprint recognition method based on hand shape semantic prior and ViT as described in claim 1, characterized in that: In step 1-1, the steps for defining the ROI region and selecting key points are as follows: Select the two outermost points on the horizontal creases at the base of the index and little fingers of the palm as keypoints D1 and D2. After obtaining the coordinates of D1 and D2, denote the line connecting D1 and D2 as L and its length as l. Take the point on the line that is 0.15l away from D1 as D3 and the point that is 0.85l away as D4. Using D3 and D4 as two vertices, construct a square with a side length of 0.7L downwards as the final ROI. Take the direction from P1 to P2 as the positive direction, and denote the angle between L and the horizontal axis as θ, which is used for image rotation.

3. The non-contact palmprint recognition method based on hand shape semantic prior and ViT as described in claim 1, characterized in that: In steps 1-2, the detailed information of the PRSN network is as follows: (1) The input layer is a hand image with a size of 224×224×3, where the three parameters correspond to the height, width, number of pixels, and number of channels of the image, respectively. Since RGB image is used as input, the number of input channels is 3. After the feature extraction layer of the fully convolutional network FCN-32S segmentation network, a feature map of 7×7×512 is obtained. (2) The feature map is then fed into four consecutive ViT blocks for semantic feature learning, and finally outputs a 49×512 dimensional feature vector. (3) After transforming the feature vector into a feature map of size 7×7×512, it is fed into the upsampling layer of the FCN-32S network. The upsampling layer is implemented by transposed convolution. A direct connection is added between the feature extraction layer and the upsampling layer to increase information flow. Finally, the upsampling layer outputs a feature map of size 224×224×21. (4) The final feature map is processed by a 2D convolution with a kernel size of 1×1, 21 input channels and 2 output channels, resulting in a final segmentation result of 224×224×2. The three numbers correspond to the number of high pixels, the number of wide pixels and the number of channels of the image, respectively. In the output results of the two channels, the first channel corresponds to the confidence of the pixel belonging to the background class and the second channel corresponds to the confidence of the pixel belonging to the palm region class. (5) Take the label corresponding to the larger value of the two channels for each pixel position of the segmentation result to obtain the final black and white binary mask.

4. The non-contact palmprint recognition method based on hand shape semantic prior and ViT as described in claim 1, characterized in that: In steps 1-2, the automatic generation steps of PRSN training samples are as follows: (1) Use the original palm images provided by the palm print database collected under multiple constrained scenarios as the palm image library; use the images in the target detection database as the background image library; (2) Randomly select an image I from the palm image source library, first convert I to a Cr channel, and denote it as I. cr , to obtain I cr The corresponding image histogram records the number of pixels for each value in the image; (3) The threshold for image binarization is calculated using Otsu's method; this threshold is then used to binarize I. cr A binarization operation is performed, where pixel values ​​less than a threshold are changed to 0, and otherwise to 1; this yields a binarized mask M1 that distinguishes the background region from the palm region; the values ​​obtained using the OTSU algorithm can effectively separate the palm region pixels from the background pixels. (4) Obtain an image I2 containing only the palm portion by performing a pixel-by-pixel AND operation between the label and the original image; (5) Generate the affine transformation parameter matrix K, randomly select the rotation angle θ∈(-π,π), the scaling factor α∈(0.7,1.3), the horizontal offset x∈(-W / 4,W / 4) of the transformed image, and the vertical offset y∈(-H / 4,H / 4) of the transformed image; W and H are the width and height of the original image, and the generated matrix K; (6) Data augmentation: M2 and I3 are obtained by performing an affine transformation on M1 and I2 with parameter K. The position and direction of the palm can be randomly changed, which helps to improve the generalization ability and robustness of the recognition model. (7) Randomize the brightness of I3 to obtain I4. Randomly select an image B from the background image library and cover the palm area of ​​I4 at the corresponding position of B to obtain the final training sample image I5, whose corresponding training label is M2.

5. The non-contact palmprint recognition method based on hand shape semantic prior and ViT as described in claim 1, characterized in that: In steps 1-3, the information of the KPLRN network is as follows: (1) Input layer: The input is a hand image with only black background with a size of 56×56×3. After passing through the backbone layer of the network, a feature map of 7×7×256 is obtained, where 7×7 represents the height and width of the feature map and 256 is the number of channels of the feature map. (2) The feature map is then processed by the global average pooling layer and becomes a 1×256 one-dimensional feature vector. It is then fed into the fully connected layer 1 and outputs a 1×64 feature vector. After this vector is processed by the ReLU activation function and normalized, it is fed into FC2 and finally obtains a 1×4 coordinate vector, representing the x and y coordinates of the two selected key points. The values ​​of the four output coordinates are normalized to the range of (-1, 1). (3) In KPLRN, the dropout rates of Dropout1 and Dropout2 are set to 0.4 and 0.2, respectively, to prevent the model from overfitting.

6. The non-contact palmprint recognition method based on hand shape semantic prior and ViT as described in claim 1, characterized in that: In step 2-1, the information of the Inverted Bottleneck Block is as follows: (1) Input H×W×C feature map, first pass through a depthwise separable convolution with a kernel size of 7. The normalization operation of this depthwise separable convolution layer adopts layer normalization and no activation function is set; output H×W×C feature map; (2) Feature map output by depthwise separable convolution: Input a normal convolutional layer with a kernel size of 1, use GELU as the activation function, and output a feature map of H×W×4C. (3) The H×W×4C feature map is passed through a convolutional layer with a kernel size of 1. No activation function is set. The output feature map of size H×W×C is added to the initial input feature map to form the final output.

Images