Near-infrared palm vein ROI extraction method and system based on lightweight network

Abstract

The application discloses a palm vein ROI extraction method and system based on a light network, and the method comprises the following steps: first, inputting a palm vein image into an improved HRnet deep learning key point detection model to position key points; then, positioning a palm vein region of interest and a selected region position through key point coordinates obtained through geometric operation and fine adjustment; in order to correctly select the palm vein region of interest, the left and right palms are distinguished; finally, an affine transformation algorithm is used to obtain a final region containing the vein. Through the method, redundant network branches are deleted, the number of convolution channels is reduced, and a deep separable convolution mode is adopted for a residual block structure to reduce the network parameter quantity; while the key point detection accuracy is considered, the model calculation amount is greatly reduced, and the palm vein ROI extraction speed is accelerated. The application can be widely applied to the field of biometric recognition technology.

Description

Technical Field

[0001] This invention relates to the field of biometric recognition technology, and in particular to a method and system for extracting near-infrared palm vein ROIs based on lightweight networks. Background Technology

[0002] Biometric technology has demonstrated significant advantages and reliability in security authentication and identity recognition. Traditional biometric technologies, such as fingerprint recognition, facial recognition, iris recognition, and palm print recognition, are widely used in real life. Facial recognition is the most widely used, but due to the diversity of facial postures, it may fail to recognize the image. Fingerprints and palm prints, being exposed on the skin surface, are easily forged and damaged, leading to security issues. Palm vein recognition, as a new biometric technology, has many advantages and has attracted widespread attention. Hemoglobin in blood can be absorbed by near-infrared light in the 700nm–1000nm range, meaning that vein images are usually extracted under near-infrared light irradiation. Studies have shown that even twins… The palm veins of every individual are different, and the vast majority of palm veins do not fundamentally change with age. Because veins are located under the skin, they are not easily injured or forged. Therefore, it is very important to design a low-cost and effective acquisition device to obtain useful vein images for vein recognition. As we all know, palm veins contain more features than finger veins, so palm vein recognition is safer and more reliable than finger vein recognition. However, the most important part of palm vein recognition is to acquire the palm vein region. Due to the acquisition device and environment when acquiring palm vein data, the acquired palm vein images contain many irrelevant backgrounds, such as complex environments and extra parts of the wrist, which will increase the difficulty of extracting the palm vein region. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a near-infrared palm vein ROI extraction method and system based on lightweight networks. By deleting redundant network branches, reducing the number of convolution channels, and employing depthwise separable convolution on the residual block structure to reduce the number of network parameters, the computational load of the model is greatly reduced while maintaining the accuracy of key point detection, thereby accelerating the extraction speed of palm vein ROI.

[0004] The first technical solution adopted in this invention is: a near-infrared palmar vein ROI extraction method based on lightweight networks, comprising the following steps:

[0005] Acquire palm vein images and perform data preprocessing to obtain preprocessed palm vein images;

[0006] By removing redundant network branches and reducing the number of convolution channels in the HRnet network structure, a lightweight HRnet deep learning key point detection model is constructed.

[0007] The preprocessed palm vein image is input into a lightweight HRnet deep learning key point detection model to locate key points and obtain the coordinates of key points in the region of interest of the palm vein.

[0008] Based on the key point coordinates of the region of interest (ROI) of the palm vein, the ROI of the palm vein is selected by the preprocessed palm vein image using a vein ROI selection algorithm.

[0009] Furthermore, the step of acquiring palm vein images and performing data preprocessing to obtain preprocessed palm vein images specifically includes:

[0010] Set up a palm vein acquisition device to obtain palm vein images;

[0011] The Labelme labeling tool was used to annotate four key points in the palm portion of the palm vein image according to the Pascal VOC dataset format, namely the four valley points between the five fingers, starting from the thumb and numbered sequentially from 1 to 4, to obtain the annotated palm vein image.

[0012] Data augmentation processing was performed on the labeled palm vein images to obtain preprocessed palm vein images.

[0013] Furthermore, the step of removing redundant network branches and reducing the number of convolutional channels in the HRnet network structure to construct a lightweight HRnet deep learning keypoint detection model specifically includes:

[0014] The number of multi-resolution stacking operations in the HRnet network structure was reduced.

[0015] The high-resolution and second-high-resolution features in the reduced HRnet network structure are fused with the low-resolution feature maps of the last two stages. Specifically, the highest resolution features are fused with the lowest resolution features, and the second-highest resolution features are fused with the second-lowest resolution features.

[0016] In the HRnet network structure after feature fusion, downsampling is achieved by controlling the convolution stride and pooling operations, while upsampling is achieved by transposed convolution operations.

[0017] The standard convolutions of the residual modules in the transposed HRnet network structure are replaced with depthwise separable convolutions to construct a lightweight HRnet deep learning keypoint detection model.

[0018] Furthermore, the step of replacing the standard convolutions of the residual modules in the transposed HRnet network structure with depthwise separable convolutions specifically includes:

[0019] Depthwise separable convolution is a decomposable convolutional structure that can decompose standardized convolution into depthwise convolution and pointwise convolution. During operation, each input feature channel is first convolved by depthwise convolution, and then the output of the depthwise convolution is combined using pointwise convolution.

[0020] Furthermore, it also includes reflecting the reduction in computational complexity of the lightweight HRNet deep learning keypoint detection model by comparing the computational cost of constructing depthwise separable convolutions with that of standard convolutions. The expression for this reduction is:

[0021]

[0022] F1 = H × W × K 2 ×C in ×C out

[0023] F2=H×W×K 2 ×C in +H×W×C in ×C out

[0024] In the above formula, H×W represents a feature map of input size H×W, and K 2 C represents the kernel size. in C represents the number of channels in the input feature map. out F1 represents the number of output feature map channels, F2 represents the computation cost of standard convolution, and F2 represents the computation cost of depthwise separable convolution.

[0025] Furthermore, the step of inputting the preprocessed palm vein image into the lightweight HRnet deep learning keypoint detection model for keypoint localization and obtaining the keypoint coordinates of the region of interest of the palm vein specifically includes:

[0026] A Gaussian function is used to generate a heatmap for soft annotation of the preprocessed palm vein image, which guides the lightweight HRnet model to better learn the location of key points.

[0027] The image with soft annotations is input into the network, and the error between the key point coordinates predicted by the lightweight HRnet model and the actual key point coordinates is calculated using the mean squared error loss function (MSE).

[0028] The Adam optimization function is used to continuously update the weight parameters, making the prediction of key point coordinates more accurate. At the output end, each channel represents a key point position. By setting a threshold, the point with the largest response value in the output channel is the final output key point coordinate value.

[0029] After training for a certain number of epochs, the loss function gradually converges, and the keypoint detection accuracy on the test set no longer shows significant improvement. At this point, training is stopped, and the model is saved.

[0030] Furthermore, the step of selecting the ROI region of the palm vein based on the key point coordinates of the palm vein region of interest using a vein ROI selection algorithm, specifically includes:

[0031] Select three key points on the palm: points A and B, and the valley point of the thumb, i.e., reference point 1.

[0032] Draw a line parallel to AB and the perpendicular bisector of AB. At a distance of 1 / 3L from line segment AB, since there are two lines parallel to AB, they intersect at point P, P^. Passing through reference point 1, discard P^ which is far from reference point 1. At this point, select two points 1 / 2L away from point P to obtain points C and D.

[0033] Draw a line parallel to CD and perpendicular lines to points C and D. Similarly, through reference point 1, discard another incorrect parallel line. At this point, select a point L below the CD line segment, intersecting at points E and F. Connect points C, D, E, and F in sequence to form a rectangular area, thus obtaining the preliminary palm vein ROI.

[0034] The initial palm vein ROI is scaled and rotated for normalization, and the palm vein ROI region is obtained by affine transformation.

[0035] Furthermore, the method of obtaining the palm vein ROI region using affine transformation specifically refers to performing a linear transformation on a vector space within the geometric region, followed by a translation, to transform it into another vector space, and then performing translation, rotation, scaling, and cropping operations on the image.

[0036] Furthermore, it also includes constructing a keypoint detection metric, which serves as an evaluation standard for the keypoint detection accuracy of the lightweight HRnet deep learning keypoint detection model. The keypoint detection threshold is determined by calculating the Euclidean distance between the predicted point coordinates and the ground truth label, and its expression is:

[0037]

[0038]

[0039]

[0040] In the above formula, d represents the Euclidean distance between the predicted coordinates and the true label coordinates, (x dt ,y dt (x) represents the predicted coordinates. gt ,y gt) represents the true label coordinates, threshold represents a given threshold, when d is greater than threshold, P(x) is 1, otherwise it is 0, and accuracy represents the accuracy, which is the ratio of the sum of correctly predicted keypoints to the total number of keypoints.

[0041] The second technical solution adopted in this invention is: a near-infrared palm vein ROI extraction system based on a lightweight network, comprising:

[0042] The preprocessing module is used to acquire palm vein images and perform data preprocessing to obtain preprocessed palm vein images;

[0043] The building module is used to remove redundant network branches and reduce the number of convolution channels in the HRnet network structure, and build a lightweight HRnet deep learning key point detection model.

[0044] The localization module is used to input the preprocessed palm vein image into the lightweight HRnet deep learning keypoint detection model for keypoint localization and to obtain the keypoint coordinates of the region of interest of the palm vein.

[0045] The selection module uses the key point coordinates of the region of interest (ROI) of the palm vein to select the ROI region of the preprocessed palm vein image through a vein ROI selection algorithm, thereby obtaining the palm vein ROI region.

[0046] The beneficial effects of the method and system of this invention are as follows: This invention inputs palm vein images into an improved HRnet deep learning keypoint detection model for keypoint localization. Then, the obtained keypoint coordinates are used to locate the region of interest (ROI) of the palm vein and fine-tune the selected region position according to geometric operations. At the same time, in order to correctly select the ROI of the palm vein, the left and right palms are distinguished. Finally, the affine transformation algorithm is used to obtain the final region containing the vein. The improved HRnet deep learning keypoint detection model reduces the number of network parameters by deleting redundant network branches, reducing the number of convolution channels, and using depthwise separable convolution for the residual block structure. While maintaining the accuracy of keypoint detection, it greatly reduces the computational load of the model and accelerates the extraction speed of palm vein ROI. Attached Figure Description

[0047] Figure 1 This is a flowchart of the near-infrared palm vein ROI extraction method based on lightweight networks of the present invention;

[0048] Figure 2 This is a structural block diagram of the near-infrared palm vein ROI extraction system based on lightweight networks of the present invention;

[0049] Figure 3 These are palm vein images with different backgrounds obtained in specific embodiments of the present invention;

[0050] Figure 4 This is a schematic diagram of the imaging device for acquiring palm vein images according to the present invention;

[0051] Figure 5 This is a schematic diagram illustrating the results of annotating palm vein images using the Pascal VOC dataset format according to the present invention;

[0052] Figure 6 This is a schematic diagram illustrating the data enhancement of palm vein images according to the present invention;

[0053] Figure 7 This is a schematic diagram of the process for non-contact infrared palm vein image extraction ROI based on the lightweight HRnet algorithm of the present invention.

[0054] Figure 8 This is a schematic diagram of the existing HRnet network structure;

[0055] Figure 9 This is a schematic diagram of the lightweight HRnet network structure of the present invention;

[0056] Figure 10 This is a schematic diagram of the depthwise separable convolutional structure in the lightweight HRnet network of this invention;

[0057] Figure 11 This is a schematic diagram of the process for selecting palm vein ROI images in this invention;

[0058] Figure 12 This is a schematic diagram showing the changes in recognition accuracy and loss value between the model of this invention and existing methods;

[0059] Figure 13 This is a schematic diagram illustrating the ROI extraction effect under different thresholds of the present invention;

[0060] Figure 14 This is a schematic diagram illustrating the ROI extraction effect of traditional methods;

[0061] Figure 15 This is a schematic diagram illustrating the ROI extraction effect of the lightweight HRnet network according to the present invention;

[0062] Figure 16 This is a schematic diagram illustrating the extraction effect under three abnormal hand conditions using the method of this invention. Detailed Implementation

[0063] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0064] In recent years, scholars have conducted extensive research. For example, Tingting Chai et al. proposed a method to locate the region of interest (ROI) using feature points of the hand, namely the valley points between two fingers; Ahmed S. El Sayed et al. proposed a ROI localization method based on threshold segmentation, morphological and geometric operations; Wenxiong Kang et al. proposed an improved OTSU method to extract the hand contour from grayscale palm vein images, and then used the radial distance function between reference points and contour points to locate the peaks and valleys of the palm to extract the palm vein ROI; Sen Lin et al. proposed a method based on the maximum inscribed circle and centroid to extract the palm ROI; Yakno et al. discussed ROI extraction algorithms and proposed an improved algorithm for extracting larger ROIs; Damak et al. used hand boundary tracking, drawing the hand boundary distance contour by scanning contour lines, rotating the image so that the line connecting the first and third finger valleys becomes horizontal, and establishing the ROI region by selecting four hand boundaries (vertical left limit, vertical right limit, horizontal lower limit, and horizontal upper limit); Cimen segmented the hand image and determined the boundaries of the hand surface region, then scanned the entire image pixel by pixel from right to left and from top to bottom, finding that the first point reaching 255 pixels was the bone tip, and at this point, a 256×256 pixel square region was selected as the ROI region by descending 150 pixels; Wei Wu et al. proposed using image binarization to separate the palm, then drawing perpendicular lines to the four fingers (excluding the thumb), intersecting at eight points. The side length of the Region of Interest (ROI) was determined by the number of 255 pixels between adjacent points. The image was then rotated so that the vein region was parallel to the image boundary. G. Ananthi1 et al. proposed applying OTSU to palm vein images with wrist removal. In the connected regions of the generated binary image, the largest connected region represents the boundary of the palm region with fingers. An improved boundary rectangle strategy was used to extract the ROI from this palm region. However, the above methods all require a clean background for the infrared palm vein image. When the image background is complex, the above methods are difficult to extract the vein ROI.

[0065] With the rapid development of neural networks, they quickly became the mainstream algorithm for object detection. More and more deep learning network models for feature extraction were proposed, and to date, many classic neural networks have been proposed, mainly represented by R-CNN, Faster R-CNN, and Mask R-CNN algorithms. Tuan Linh Dang et al. proposed using the HRNet keypoint detection network to extract hand features and obtain hand joints for gesture prediction, achieving good detection results. Ruixin Zhang et al. proposed using the Tiny-YOLOv3 object detection algorithm, marking the palm vein region with bounding boxes, feeding it into the network for training, detecting gaps between fingers and the palm, determining keypoints by selecting the midpoint of the bounding box, and then obtaining the palm vein ROI through geometric calculation. However, their database of palm images lacks diversity in angle and shape. The detection of gaps between fingers is easily affected by the palm angle and shape when the palm is horizontal or vertical, or when the fingers are separated. When the palm is tilted, the strategy of directly detecting to obtain the ROI may not work. Luo Kai et al., in Ruxin... Zhang improved the detection method based on the algorithm proposed, taking the entire palm as the detection target. First, Tiny-YOLOV3 was used to detect the palm, and then MobilenetV2 was used to directly regress the key point coordinates in the target image. Although the method performed well on their self-built dataset, dividing the detection process into two parts was a bit cumbersome and increased the network computation.

[0066] To simplify the ROI extraction process, reduce model computation parameters, and improve running speed, this invention designs a lightweight keypoint detection network based on the HRnet concept. By deleting redundant network branches, reducing the number of convolution channels, and using depthwise separable convolution on the residual block structure, the number of network parameters is reduced. While maintaining the accuracy of keypoint detection, the computational load of the model is greatly reduced, thus accelerating the extraction speed of palm vein ROIs.

[0067] Reference Figure 1 This invention provides a near-infrared palmar vein ROI extraction method based on lightweight networks, which includes the following steps:

[0068] S1. Acquire palm vein images and perform data preprocessing to obtain preprocessed palm vein images;

[0069] Specifically, such as Figure 3As shown, palm vein images were collected under different environments. Therefore, after obtaining the palm vein images, if you want to extract the palm vein region, you first need to filter out irrelevant background. In this invention, the self-built palm vein dataset is collected by a self-built palm vein acquisition device, which mainly includes a lens, base, sensor, and PCB. It consists of an 850nm wavelength near-infrared lamp and a light shield. The interface uses micro USB. The acquisition device and shooting process are as follows. Figure 4 As shown, images were collected in May and June 2022 on the campus of South China Agricultural University in Guangzhou, Guangdong Province. The subjects were students and faculty members of different age groups. Image collection and personnel information labeling were conducted in student dormitories. A total of 300 people were sampled. Images were taken from each hand, distinguishing between left and right hands. Five images were taken from each hand, with five palm vein images taken from five random angles against both simple and complex backgrounds. The shooting distance was between 15cm and 20cm. A total of 3000 palm vein images were collected, each image measuring 1280×7 pixels. 20. In order to verify the effectiveness of the algorithm in the experiment, two public palm vein databases were also used in the paper: the Chinese Academy of Sciences Multispectral Palm Vein Database (CASIA) and the Tongji University Palm Vein Database (TJU-PV). The two databases contain 7,200 and 6,000 palm vein images, respectively. Similar to the palm vein images in the self-built database, the hand posture is not fixed. The CASIA palm vein images have complex backgrounds and the image size is 768×576, while the TJU-PV palm vein images have simple backgrounds and the image size is 800×600.

[0070] The acquired palm vein images were labeled using the Labelme tagging tool according to the Pascal VOC dataset format. In the task of extracting the infrared palm vein ROI region in this invention, four key points need to be labeled on the palm portion of the image, namely the four valley points between the five fingers, starting from the thumb and numbered sequentially from 1 to 4. The labeling effect is as follows. Figure 5 As shown, to enrich the training dataset and increase the diversity of training samples, data augmentation operations are performed on the training dataset, such as random angle reversal and adjustments to brightness and contrast. Figure 6 As shown.

[0071] S2. Redundant network branches are removed and the number of convolution channels is reduced in the HRnet network structure to construct a lightweight HRnet deep learning key point detection model.

[0072] S21. Existing HRnet network structure;

[0073] Specifically, in pose estimation, the resolution of the image feature map is crucial. Previous pose estimation methods have used a sequential high-to-low resolution approach to obtain high-resolution feature maps with strong semantic information. Compared to existing widely used networks, HRNet has two advantages: first, it connects multiple high-to-low resolution subnetworks in parallel, rather than in series; second, it adds features of different resolutions on top of the parallel connection. The interaction between maps allows HRNet to maintain high resolution instead of recovering high-resolution information through a low-to-high process, resulting in potentially more spatially accurate predicted heatmaps. Secondly, most existing fusion schemes directly fuse low- and high-level feature maps, while HRNet performs repeated multi-scale fusion, using low-resolution features of similar depth and level to enhance the representation of high-resolution features, and vice versa. Therefore, HRNet has a rich set of high-resolution features, sufficient for pose estimation. The HRNet network structure consists of two stages: a low-resolution stage and a high-resolution stage. The low-resolution stage generates feature maps at multiple resolutions, including original resolution, 1 / 2 resolution, 1 / 4 resolution, and 1 / 8 resolution. The high-resolution stage uses a feature fusion module to fuse the feature maps generated in the low-resolution stage to generate a high-resolution feature map. A schematic diagram of the HRNet algorithm is shown below. Figure 8 As shown, the Bottleneck module is the bottleneck layer of ResNet, used to deepen the network. The Basic module is a general ResNet structure, with each basic convolutional block including BN and ReLU layers. Up and Down represent upsampling and downsampling, respectively. The entire network generates reliable and location-sensitive high-resolution feature maps by repeatedly fusing multi-resolution stream representations. Finally, the number of feature map channels is determined based on the number of detected keypoints. In the keypoint prediction part, a heatmap approach is used. A Gaussian function is used to generate a heatmap of the same size as the network output feature map based on the actual coordinate labels of the keypoints. Each position represents the confidence level of whether a keypoint exists at that position. By setting a threshold, the exact location of the keypoint is finally output. During training, the mean squared error loss function is used, and the calculation formula is shown below:

[0074]

[0075] In the above formula, y i Represents the actual value. The predicted value is represented by the mean squared error loss function, which is a smooth function and can be minimized using the gradient descent method.

[0076] S22, Lightweight HRnet Deep Learning Keypoint Detection Model Structure;

[0077] Specifically, HRNet outperforms many object detection algorithms in keypoint detection tasks due to its repeated stacking and multi-resolution fusion. However, this network structure incurs huge computational and time overhead, making it unsuitable for real-time detection tasks. In particular, the computing and storage capabilities of ordinary CPUs in embedded systems cannot meet the deployment requirements of such large models. Since palm vein ROI localization tasks need to be completed in real time, the original HRNet structure is too redundant. Therefore, this invention modifies HRNet using the following method:

[0078] (1) Reduce the number of convolutional kernels and reduce the number of multi-resolution stacking operations;

[0079] (2) By fusing high-resolution and sub-high-resolution features with low-resolution feature maps in the last two stages, the high-resolution features are maintained by fusing the highest resolution features with the lowest resolution features, and the sub-high resolution features with the second-lowest resolution features.

[0080] (3) Downsampling is achieved by controlling the convolution stride and pooling operations (MaxPool, AveragePool), while upsampling is achieved by transposed convolution (Transposed Conv) operation;

[0081] (4) Replace the standard convolutions in the original HRnet residual module with depthwise separable convolutions (DSC);

[0082] Through the above improvements, not only is the network's fitting ability increased, but the number of network parameters is also greatly reduced, thereby speeding up the model's operation. The modified network model is as follows: Figure 7 As shown.

[0083] S23. For the lightweight HRNet deep learning keypoint detection model structure, depthwise separable convolutions are used.

[0084] Specifically, since the convolution kernel in standard convolution (Conv) operates on each channel of the input feature map, the computational cost is large. In this invention, depthwise separable convolution is used to replace standard convolution at the residual connection to reduce the number of network parameters. Depthwise separable convolution is a decomposable convolution structure that can decompose the standardized convolution into depthwise convolution and pointwise convolution. During operation, each input feature channel is first convolved by depthwise convolution, and then the output of the depthwise convolution is combined using pointwise convolution. This process of decomposing the convolution can effectively reduce the model parameter size and computational cost.

[0085] For a feature map with input size H×W, the convolution kernel size is K×K, C in For the number of channels in the input feature map, C outThe computational costs of standard convolution and depthwise separable convolution, respectively, are calculated based on the number of channels in the output feature map:

[0086] F1 = H × W × K 2 ×C in ×C out

[0087] F2=H×W×K 2 ×C in +H×W×C in ×C out

[0088] The computational cost ratio of depthwise separable convolution to standard convolution is:

[0089]

[0090] According to the formula for calculating the computational cost ratio of depthwise separable convolution to standard convolution, when an image has an input size of 12×12, 3 input channels, 128 output channels, and a convolution kernel size of 5×5, as shown... Figure 10 As shown, the computational cost of depthwise separable convolution is only 5% of the computational cost of standard convolution.

[0091] S3. Input the preprocessed palm vein image into the lightweight HRnet deep learning key point detection model to locate key points and obtain the coordinates of key points in the region of interest of the palm vein.

[0092] S4. Based on the key point coordinates of the palm vein region of interest, the ROI region of the preprocessed palm vein image is selected using a vein ROI selection algorithm to obtain the palm vein ROI region.

[0093] Specifically, the selection of the Region of Interest (ROI) is a crucial step. Correctly selecting the ROI region can preserve vein areas in the same location as much as possible. To avoid the effects of translation, rotation, and scaling, this invention uses key points to normalize the palm vein image. Three key points are selected in the palm: points A and B, and the valley point at the thumb position, i.e., reference point 1. The palm vein ROI extraction scheme proposed in this invention is as follows:

[0094] (1) Draw a line parallel to AB and the perpendicular bisector of AB. At a distance of 1 / 3L from line segment AB, since there are two parallel lines parallel to AB, they intersect at point P, P^. Passing through reference point 1, discard P^ which is far from reference point 1. At this time, select two points at a distance of 1 / 2L on the left and right sides of point P, namely point C and point D. Since the ROI selected in this invention is a rectangle, two more points need to be determined at this time.

[0095] (2) Draw a line parallel to CD and a line perpendicular to points C and D. Similarly, through reference point 1, discard another incorrect parallel line. Now select a point L below the CD line segment, intersecting at points E and F. Connect points C, D, E, and F in sequence to form a rectangular area, thus obtaining the palm vein ROI;

[0096] (3) Finally, the extracted ROIs are scaled and rotated for normalization. Affine transformation is then used to obtain the final matching palm vein ROI. Affine transformation, also known as affine projection, refers to the linear transformation of a vector space followed by a translation in geometry, transforming it into another vector space. This allows for operations such as translation, rotation, scaling, and cropping of images. Based on experience, a 256×256 ROI is typically selected. The selection process is as follows: Figure 11 As shown.

[0097] Experiments were conducted based on the method of the present invention described above;

[0098] (1) Experimental parameter settings;

[0099] The experiment used the Ubuntu operating system, Python distribution, Anaconda (Python 3.7) virtual environment, PyTorch 1.8.2 deep learning framework, and CUDA 10.2 accelerator. The CPU used was an Intel i7-9700F, and the GPU was an NVIDIA GeForce RTX 3080. SGD was used as the optimizer during the training of the Keypoint RCNN keypoint detection model. The beta1 parameter was adjusted to 0.5, the beta2 parameter to 0.999, the batch size was set to 32, and the initial learning rate was set to 0.001, using exponential decay to adjust the learning rate. The datasets used in this paper and the experimental data sample settings for two publicly available palm vein datasets are shown in Table 1.

[0100] Table 1 Experimental Data Setup

[0101] Database Train Test Total CASIA 1600 sheets 400 sheets 2000 TJU-PV 1600 sheets 400 sheets 2000 Shooting data 3600 900 4500 sheets

[0102] (2) Constructing evaluation indicators;

[0103] Unlike object detection, which determines the threshold by calculating the Intersection over Union (IoU) between the detected bounding box and the ground truth bounding box, keypoint detection requires calculating the Euclidean distance between the predicted point coordinates and the ground truth label to determine the threshold. The model evaluation criterion in this invention is calculated using the following formula:

[0104]

[0105] In the above formula, d represents the Euclidean distance between the predicted coordinates and the true label coordinates, (x dt,y dt (x) represents the predicted coordinates. gt ,y gt () represents the actual label coordinates;

[0106]

[0107] threshold is a given threshold. When d is greater than threshold, P(x) is 1, otherwise it is 0.

[0108]

[0109] The accuracy rate is defined as the ratio of the sum of correctly predicted keypoints to the total number of keypoints.

[0110] (3) Experimental results;

[0111] After training the improved model and the original HRNet model for 150 epochs, the training loss versus test set accuracy curves on the self-built database are as follows: Figure 12 As shown in the figure, the left figure a represents the keypoint training loss of the proposed model and the original model, and the right figure b represents the keypoint detection accuracy of the test set (threshold is 5). It can be seen from the figure that due to the complexity and depth of the original HRnet network, the convergence speed is faster than that of the improved HRnet network. The detection accuracy of the improved HRnet network is close to that of HRnet. At the same time, under the palm vein ROI extraction algorithm proposed in this invention, the keypoint detection accuracy and vein ROI extraction effect were verified on the self-built vein database and two public vein databases, CASIA and TJU-PV.

[0112] Table 2 shows that the detection accuracy of palm vein key points is related to the threshold setting. A higher threshold means only key points with scores below that threshold are considered valid, thus increasing accuracy. At a threshold of 3, the detection accuracy of palm vein key points is generally low across CASIA, TJU-PV, and self-built databases. The TJU-PV database, with its clean background, exhibits the highest detection accuracy. However, at a threshold of 6, palm vein key points perform well across all three databases. To determine the most suitable threshold, this paper... Figure 13 The improved HRnet network was verified to extract actual ROIs under different thresholds.

[0113] Table 2. Keypoint detection accuracy under different thresholds

[0114]

[0115] from Figure 13It can be seen that when the threshold is 5, the palm vein ROI region can be accurately extracted. When the threshold is 6, the entire palm vein ROI region shifts to the right due to the large deviation of the point with the number 4, resulting in poor extraction effect. Finally, the threshold is set to 5 as the accuracy threshold for calculating the palm vein key point localization.

[0116] Reference Figure 2 A near-infrared palm vein ROI extraction system based on lightweight networks includes:

[0117] The preprocessing module is used to acquire palm vein images and perform data preprocessing to obtain preprocessed palm vein images;

[0118] The building module is used to remove redundant network branches and reduce the number of convolution channels in the HRnet network structure, and build a lightweight HRnet deep learning key point detection model.

[0119] The localization module is used to input the preprocessed palm vein image into the lightweight HRnet deep learning keypoint detection model for keypoint localization and to obtain the keypoint coordinates of the region of interest of the palm vein.

[0120] The selection module uses the key point coordinates of the region of interest (ROI) of the palm vein to select the ROI region of the preprocessed palm vein image through a vein ROI selection algorithm, thereby obtaining the palm vein ROI region.

[0121] In summary, this paper proposes a lightweight network based on an improved HRnet keypoint detection system. This network utilizes keypoint localization methods to extract Regions of Interest (ROIs) from non-contact infrared palm vein images, unaffected by complex backgrounds or varied hand poses. The flowchart of the proposed algorithm is shown below. Figure 7 As shown, the palm vein image is first input into the trained improved HRnet deep learning key point detection model for key point localization. Then, the obtained key point coordinates are used to locate the palm vein region of interest and fine-tune the selected region position according to geometric operations. At the same time, in order to correctly select the palm vein region of interest, the paper will distinguish between the left and right palms. Finally, the affine transformation algorithm is used to obtain the final region containing the vein.

[0122] A comparative experiment was conducted between the algorithm proposed in this invention and existing methods.

[0123] (1) Comparison of ROI extraction results with traditional methods;

[0124] Generally, traditional palm vein ROI localization involves preprocessing steps including image thresholding, manual segmentation, contour detection, key point localization, and ROI extraction. For palm vein images with a clean background, the traditional ROI extraction method follows these steps: Figure 14 As shown. However, for palm vein images with complex backgrounds, the Otsu thresholding method for hand segmentation and hand boundary tracking is not ideal in complex scenes containing multiple objects in a single image. This leads to key point misalignment and inaccurate localization of the region of interest for the palm veins, such as... Figure 15 As shown, existing non-contact devices for acquiring palm vein images mostly rely on auxiliary tools to fix the position of the palm, resulting in relatively regular palm vein images. However, this limits the user's hand posture and angle, leading to a poor user experience. The algorithm proposed in this invention demonstrates ROI extraction effects under different backgrounds and in palm vein images with irregular hand postures, such as... Figure 15 and 16 As shown;

[0125] (2) Ablation experiment;

[0126] Table 3 compares the performance of the improved HRnet model with the original HRnet and three other network models: VGG16, Resnet-18, and Resnet-50. It can be seen that when the input image size is 512×512, the improved HRnet maintains a certain level of accuracy while having a model size of only 0.45m and a running speed of only 0.029s on a 3GHz CPU. This invention also verifies the comparison of the detection effects of different improvement methods on key points.

[0127] Table 3 Performance Comparison of Several Object Detection Algorithms

[0128] Algorithm ImageSize Params(M) Accuracy Speed ​​(s / sheet) VGG16 512×512 49.2 79.46％ 0.5 Resnet-18 512×512 42.7 78.64％ 0.23 Resnet-50 512×512 90.2 96.75％ 0.7 HRnet 512×512 109 98.87％ 1.3 Modifiednet 512×512 0.45 97.36％ 0.029

[0129] As shown in Table 4, by using the fusion of two high-resolution and low-resolution features and transposed convolution instead of upsampling, the keypoint detection accuracy reached 98.24%, while using only one high-resolution and low-resolution feature fusion resulted in a 4 percentage point decrease in accuracy. Furthermore, to further reduce the number of model parameters, this invention uses depthwise separable convolution, resulting in only approximately 1% accuracy loss.

[0130] Table 4. Impact of different improvement methods on the accuracy of keypoint detection

[0131] Transposed convolution Depthwise separable convolution One fusion Two fusions Accuracy √ √ \ √ 97.36％ √ \ \ √ 98.24％ \ √ \ √ 96.83％ √ √ √ \ 93.17％ √ \ √ \ 94.35％ \ √ √ \ 92.77％

[0132] Therefore, rapid and accurate extraction of the region of interest (ROI) of palm veins without contact is the foundation for subsequent palm vein recognition applications. In this invention, for complex backgrounds, irregular hand placement, and the presence of items on the hands, a lightweight ROI extraction algorithm is proposed based on an improved HRnet combined with geometric methods. Experimental results show that this method has good extraction accuracy, while the network model size is only 0.45m, and the running speed on a 3GHz CPU is only 0.029s. The key point detection accuracy reaches 97.36% on a self-built palm vein dataset, and the accuracy reaches 98.23% and 98.74% on two public palm vein databases, respectively.

[0133] Experimental analysis shows that the method proposed in this invention has a slow convergence speed during training, mainly due to the small differences at several target key points, certain errors in the output of heatmap activation point positions, and the small number of convolution channels used. Future research will focus on the problem of small target, i.e., key point detection, in complex backgrounds. In addition, further consideration will be given to the problem of balancing model lightweighting and accuracy, reducing the number of parameters while ensuring high accuracy, and subsequent research on palm vein recognition will be carried out.

[0134] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0135] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for extracting a near-infrared palm vein ROI based on a lightweight network, characterized in that, The method comprises the following steps: Collecting a palm vein image and performing data preprocessing to obtain a preprocessed palm vein image; Performing redundancy network branch deletion and convolution channel number reduction processing on the HRnet network structure to construct a lightweight HRnet deep learning key point detection model; Inputting the preprocessed palm vein image into the lightweight HRnet deep learning key point detection model for key point positioning to obtain key point coordinates of a palm vein region of interest; Based on the key point coordinates of the palm vein region of interest, performing ROI region selection processing on the preprocessed palm vein image through a vein ROI selection algorithm to obtain a palm vein ROI region; The step of performing redundancy network branch deletion and convolution channel number reduction processing on the HRnet network structure to construct a lightweight HRnet deep learning key point detection model specifically comprises: Performing multi-resolution stack number reduction processing on the HRnet network structure; Fusing high-resolution and sub-high-resolution features in the reduced HRnet network structure with low-resolution feature maps in the last two stages, specifically fusing the highest-resolution feature with the lowest-resolution feature and fusing the sub-high-resolution feature with the sub-low-resolution feature; Implementing down-sampling in the feature-fused HRnet network structure through control of convolution step length and pooling operation and implementing up-sampling through transposed convolution operation; Replacing standard convolution in the transposed HRnet network structure with deep separable convolution to construct a lightweight HRnet deep learning key point detection model; The step of performing ROI region selection processing on the preprocessed palm vein image through a vein ROI selection algorithm based on the key point coordinates of the palm vein region of interest to obtain a palm vein ROI region specifically comprises: Distinguishing left and right hands, selecting three key points in the palm, and defining them as points A and B and a valley point of the position of the thumb, i.e., a No. 1 reference point; Drawing parallel lines of AB and a vertical line of AB, and selecting two points at a distance of 1 / 2L from the line segment AB, to obtain points C and D; Drawing parallel lines of CD and vertical lines of points C and D, and selecting two points at a distance of L from the line segment CD, to obtain points E and F, and sequentially connecting points C, D, E, and F to form a rectangular region, thereby obtaining a preliminary palm vein ROI; Performing scaling and rotation normalization on the preliminary palm vein ROI to obtain a palm vein ROI region through affine transformation.

2. The method according to claim 1, wherein, The step of collecting a palm vein image and performing data preprocessing to obtain a preprocessed palm vein image specifically comprises: Building a palm vein collection device to obtain a palm vein image; The palm part four key points in the palm vein image are labeled by a Labelme label tool according to a Pascal VOC data set format, that is, four valley points between five fingers, and the four valley points are sequentially labeled from 1 to 4 from the thumb, to obtain a labeled palm vein image; The labeled palm vein image is subjected to data enhancement processing to obtain a preprocessed palm vein image.

3. The method according to claim 2, wherein, The step of replacing the standard convolution in the residual module in the transposed HRnet network structure with a depth separable convolution specifically includes: The depth separable convolution is a decomposable convolution structure, which can decompose the standard convolution into a depth convolution and a point-by-point convolution. During operation, the depth convolution is first used to convolve each input feature channel, and then the point-by-point convolution is used to combine the output of the depth convolution.

4. The method according to claim 3, wherein, The reduction amount of the calculation parameter of the lightweight HRnet deep learning key point detection model is embodied by constructing the ratio of the depth separable convolution to the standard convolution, and the expression is: In the above formula, denotes an input feature map with a size of denotes a convolution kernel size, denotes an input feature map channel number, denotes an output feature map channel number, denotes a standard convolution calculation amount, denotes a depth separable convolution calculation amount.​ 5. The method according to claim 4, wherein, The step of inputting the preprocessed palm vein image into the lightweight HRnet deep learning key point detection model to locate the key points and obtain the key point coordinates of the palm vein region of interest specifically includes: The preprocessed palm vein image is soft-labeled by generating a heat map using a Gaussian function to guide the lightweight HRnet model to better learn the positions of the key points; The image with soft labeling is input into the network, and the mse mean square error loss function is used to calculate the error between the predicted key point coordinates and the actual key point coordinates of the lightweight HRnet model; The Adam optimization function is used to continuously update the weight parameters, so that the predicted key point coordinates are more accurate. At the output end, each channel represents a key point position. The threshold setting method is used to output the point with the maximum response value in the output channel as the final output key point coordinate value. After a certain number of epochs are trained, the loss function gradually converges, and the key point detection accuracy of the test set is not significantly prompted. The training is stopped, and the model is saved.

6. The method according to claim 5, wherein, The palm vein ROI region obtained by affine transformation specifically includes performing linear transformation on a vector space and adding a translation to another vector space, and performing translation, rotation, scaling and shearing operation processing on the image.

7. The method according to claim 6, wherein, The key point detection index is also constructed, which is used as the key point detection accuracy evaluation standard of the lightweight HRnet deep learning key point detection model. The key point detection is determined by calculating the Euclidean distance between the predicted point coordinates and the true label, and the expression is: In the above formula, denotes the Euclidean distance between the predicted coordinate and the real label coordinate, denotes the predicted coordinate, denotes the real label coordinate, denotes a given threshold, when is greater than , is 1, otherwise 0, denotes the accuracy, i.e., the ratio of the total number of correctly predicted key points to the total number of key points.

8. A near-infrared palm vein ROI extraction system based on a lightweight network, characterized by, The method for extracting a near-infrared palm vein ROI based on a lightweight network according to claim 1 comprises the following modules: A preprocessing module is configured to collect palm vein images and perform data preprocessing to obtain preprocessed palm vein images; A construction module is configured to delete redundant network branches and reduce the number of convolution channels of an HRnet network structure to construct a lightweight HRnet deep learning key point detection model; The positioning module is configured to input the preprocessed palm vein image into a light-weight HRnet deep learning key point detection model to perform key point positioning and obtain key point coordinates of a palm vein region of interest. The selection module is configured to perform ROI region selection processing on the preprocessed palm vein image based on the key point coordinates of the palm vein region of interest through a vein ROI selection algorithm to obtain a palm vein ROI region.

Images