Back of hand or palm vein image key point and ROI positioning method based on improved UNet

Abstract

The application relates to the technical field of back-of-hand vein recognition, and discloses a back-of-hand or palm vein image key point and ROI positioning method based on an improved UNet, which comprises the following steps: S1, collecting an experimental system machine on site; S2, self-building a data set; S3, establishing a UNet model; S4, establishing a residual module; S5, an improved UNet model for back-of-hand or palm vein image key point positioning; S6, using an ROI calculation method to calculate the extraction of a region of interest; and S7, evaluating the model. Near-infrared imaging technology is used to acquire a back-of-hand or palm vein image. The improved UNet model can realize back-of-hand or palm key point positioning (ROI extraction) on a non-contact back-of-hand or palm vein image, and is suitable for actual deployment on a low-resource platform such as an edge embedded system.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of back-of-hand or palm vein recognition, in particular to a back-of-hand or palm vein image key point and ROI positioning method based on an improved UNet. BACKGROUND

[0002] With the rapid development of security systems and Internet economy, personal identity verification has become an indispensable technology for security purposes. Existing methods such as passwords, fingerprints, faces, palm prints and other biometric systems have limitations. Vein patterns stand out from numerous intrinsic biological characteristics and are used to develop an identification system that can eliminate the limitations of other existing systems. Facial recognition is a popular method for identifying individuals, but it also has some limitations. The face can be obscured by hair, glasses, masks and the like, which can reduce the recognition accuracy of the system. Fingerprints and palm prints, due to the inability to keep the hands clean at all times, may not be able to see the vein patterns under the obscuring of stains; there is also a part of the laborers working in a harsh environment causing the fingerprints and palm prints to change due to wear and tear, which is significantly different from the initial registered image, resulting in misidentification or false recognition. Moreover, during the epidemic period, face recognition requires the removal of masks, and fingerprints and palm prints require contact with the device, which can increase the probability of infection with the virus. Vein structure is the vascular structure existing under the skin, which is difficult to damage or replicate this structure. The advantage of vein technology is that it can only be obtained from a living person, and the pattern does not change over time. Therefore, the vein pattern can provide more accurate results and higher security. It has been proven that the vein network of each person is different, and even twins are not similar. Therefore, the proposed system is non-contact, low-cost and more secure than other popular biometric identification systems.

[0003] In recent years, dorsal hand vein (DHV) recognition has attracted much attention as an emerging biometric technology. It is well known that DHV recognition belongs to the family of hand-based biometrics. DHV recognition is a technology that uses automatic analysis of the subcutaneous vein network on the back of the hand to authenticate a person's identity. Region of interest (ROI) extraction is a key step in DHV recognition. The image of an open hand is binarized, the Euclidean distances between each edge pixel of the hand and the midpoint of the wrist are calculated, and a distance profile that is very similar in shape to the geometry of the back of the hand is constructed using these distances. The second and fourth fingerwells are selected as reference points to define a square ROI. The gray-scale image is binarized, and then the edge region is extracted by the Sobel operator. Multiple tangent circles are drawn along the edge line, and rules related to the number of intersection points between the tangent circles and the edge line are set to determine the positions of the corner points. After the midperpendiculars between the corner points are obtained, the slopes of the midperpendiculars are calculated to perform a rotation correction on the original vein image, and the ROI position is obtained, overcoming the disadvantages of uneven translucency and accidental rotation when establishing a database. The ROI extraction algorithm is discussed, and an improved algorithm for larger ROI extraction is proposed. Otsu thresholding is used to segment the hand, and the hand boundary is tracked. The hand boundary distance profile is drawn by scanning the contour lines, and the image is rotated so that the line connecting the first and third finger valleys becomes horizontal. The ROI region is established by selecting four hand boundaries (vertical left limit, vertical right limit, horizontal lower limit, and horizontal upper limit). The hand image is segmented, and the boundary of the hand surface region is determined. Then, the entire image is scanned pixel by pixel from right to left and from top to bottom, and the first point that reaches 255 pixels is found to be the bone tip. A 256x256 pixel square region is selected as the ROI region 150 pixels below this point. The image of a clenched hand is converted into a binary image, and the hand back boundary is located using morphological methods. After the distances between each point on the boundary and the midpoint of the wrist are calculated, it is found that the valley points between the fingers are the corresponding valley points in the distance profile. Finally, a fixed-size sub-image based on valley points 1 and 3 is extracted. A general ROI extraction algorithm typically includes the following main steps: (1) convert the hand image into a binary image using a segmentation algorithm; (2) perform hand boundary tracking; (3) calculate the distance profile between the contour points (points located on the hand contour) and the reference points. The midpoint of the wrist is generally used as the reference point; (4) position the ROI based on the detected points; and (5) crop the ROI sub-image.

[0004] Many researchers have used some classic image processing methods to extract ROI of hand dorsal vein image, in which the key step is to determine the valley point (hereinafter referred to as key point) by calculating the distance, and then to extract ROI according to the obtained key point. The current research objects are basically hand dorsal vein images collected in a constrained environment where the position of the hand is limited and the background is very clean. The traditional image processing algorithm is very beneficial to the segmentation of such images; however, it is difficult to segment the complete image of the hand in a complex background, and it is difficult to meet the actual use scenario. So far, deep learning technology has not been used in the ROI extraction of hand dorsal vein images. Nowadays, deep learning has become one of the most important technologies in the field of computer vision. In recent years, the rapid development of computer vision, robotics and other fields almost all depends on deep learning technology. In the field of biometric identification, especially in the field of face recognition, deep learning has become the most mainstream technology. For image-based biometric identification technology, neural networks are the most commonly used deep learning technology. So far, many classic neural networks have been proposed and have achieved impressive results in many identification tasks. With the continuous improvement of deep learning network architecture and the continuous increase of data volume, the recognition accuracy of deep learning in different biological indicator tasks is also continuously improved. For example, in the field of face recognition, the recognition accuracy of deep learning has far exceeded that of traditional manual algorithms; therefore, deep learning has successfully promoted the large-scale application of face recognition technology.

[0005] Currently, the key point detection research mainly includes two methods: one is to directly regress the key point coordinates through the fully connected layer of the neural network, and the other is to directly output the heat map through the neural network without the fully connected layer, and the coordinates corresponding to the peak value of the heat map are the key point coordinates. The method of directly regressing the key point coordinates through the fully connected layer is mostly used in face key point detection, because the face can be regarded as a rigid body, and the relative positions between points are basically unchanged, the spatial information of the feature map is lost, and the spatial generalization is lacking. However, the hand is very flexible and has various postures. When collecting non-contact hand dorsal vein images, because there is no hand fixing device, the hand will appear in different situations such as pitching, bending and opening and closing, and the relative positions of the key points change greatly, so the heat map method is selected to locate the key points of the hand dorsal vein image in this study. The UNet network model is a classic model in fully convolutional neural networks, which is divided into down-sampling and up-sampling paths, and the low-level features and high-level features are fused through jump connection to obtain more rich feature information, which can more accurately locate the key points of the hand dorsal vein image.

[0006] Deep learning has developed rapidly in the field of computer vision, and its ability in image detection and recognition under complex conditions is better than that of traditional image processing algorithms. Therefore, it is crucial to study the method of deep learning in non-contact hand vein image key point positioning for ROI extraction. To this end, the study collects unconstrained hand vein images by self-developed infrared image acquisition equipment to construct a hand vein dataset, selects a heat map to predict key point coordinates to improve the UNet network, adds a residual module to the down-sampling path of the original UNet network to solve the model degradation problem caused by deepening the network, changes the transpose convolution in the up-sampling path of the UNet network to bilinear interpolation up-sampling to reduce network parameters, introduces a JS divergence loss function to supervise the final feature map, and introduces SoftArgmax to decode the key point coordinates from the feature map to realize end-to-end training. The traditional image processing algorithm cannot quickly and accurately extract ROI from non-contact hand vein images under complex backgrounds, such as translation, rotation, scaling, and slight pitch. However, deep learning algorithms are large in size and require high computing resources, making it difficult to deploy them on edge embedded systems. Therefore, the study proposes a hand or palm vein image key point and ROI positioning method based on improved UNet. SUMMARY

[0007] (I) Technical problems solved

[0008] To solve the above problems, the present application provides a hand or palm vein image key point and ROI positioning method based on improved UNet.

[0009] (II) Technical solutions

[0010] To achieve the above purpose, the present application provides the following technical solutions: a hand or palm vein image key point and ROI positioning method based on improved UNet, comprising the following steps:

[0011] S1: Experimental system machine collects on-site settings;

[0012] S2: Self-built dataset;

[0013] Acquire hand or palm vein images using near-infrared imaging technology;

[0014] The left and right hands of the subject are irradiated with near-infrared light, the bottom of the lens is provided with an 850nm filter to prevent visible light from entering the camera, and the package shell can be optionally provided with an 850nm filter. The current supply of the collection system remains unchanged during the entire experiment. The collection site of the hand back or palm vein image is South China Agricultural University. The researchers have no restrictions on the angle, distance and shooting background of the subject's hand. Five pictures of the left and right hands of each person are taken respectively, and the picture size is 1920 pixels x 1080 pixels. Thus, a total of 2200 unconstrained hand back or palm vein image datasets of 220 people are created, including clean background, background interference, incomplete shielding and wearing ring conditions.

[0015] The public dataset is adopted. The infrared camera of the vein image collection device is placed horizontally on the desktop. The hand back or palm surface is parallel to the collection device, and can be at any angle in the parallel plane. It is stopped at 10-15 cm above the lens. A total of 222 people are collected in the dataset. Each person takes 10 pictures of the right hand, a total of 2220 hand back or palm vein pictures.

[0016] S3: Establishing a UNet model;

[0017] S4: Establishing a residual module;

[0018] With the deepening of the network, the network can extract more rich feature information, and the performance of the network will be improved. However, research shows that with the deepening of the network, the problem of gradient explosion and gradient disappearance will occur, which will lead to network degradation. In view of the problem of gradient explosion and gradient disappearance in network training caused by the deepening of network layers, He Kai-ming et al. proposed a residual module to solve the problem of network degradation. There are two paths. The first path is that the input x is output as a residual F(x) after passing through at least two convolutional layers and a Relu activation function. The second path is that the original input x is directly added to the output F(x) of the first path through a short connection. The final output of the residual module is H(x) = F(x) + x. When the residual F(x) is 0, the residual module only performs an identity mapping, and the network performance will not decrease. However, in fact, the residual F(x) will not be 0, which will enable the residual module to learn new features based on the input features, thereby improving the performance of the network.

[0019] The residual module is divided into a conventional residual module and a bottleneck residual module, as shown in Figure 5 The bottleneck structure refers to the change of the number of channels, i.e. the original number of channels is reduced and then increased, so as to reduce and increase the number of channels, thereby reducing the dimension of the input x and inputting it into a 3x3 convolutional layer for convolution operation, thereby improving the calculation efficiency.

[0020] S5: improved UNET model for key point positioning of back of hand or palm vein image;

[0021] S6: extraction of region of interest by using ROI calculation method;

[0022] S7: evaluation of the model;

[0023] Preferably, in the S6, the extraction of the region of interest specifically comprises the following steps:

[0024] S601: the key points of the four finger webs from the thumb to the little finger are in the order of 0, 1, 2, and 3;

[0025] S602: a line segment L13 is drawn with the key points 1 and 3 as the endpoints, and the length of the line segment L13 is l13;

[0026] S603: a perpendicular line L45 is drawn with the midpoint 4 of the line segment L13 as the center, and the length of the perpendicular line L45 is l13 / 3;

[0027] S604: a perpendicular line L67 is drawn with the point 5 as the center, and the length of the perpendicular line L67 is l13;

[0028] S605: a square 6789 is drawn with the line segment L67 as the side;

[0029] S606: the region of interest image of the square 6789 is extracted, and the size of the region of interest image is adjusted to 256x256 pixels by linear interpolation.

[0030] Preferably, in the S7, the model evaluation standard is calculated as follows:

[0031]

[0032]

[0033]

[0034] Preferably, the (1) is the Euclidean distance between the predicted coordinate (x1, y1) and the label coordinate (x2, y2);

[0035] The (2) is that a distance less than 40 pixels is considered as correct prediction and marked as 1, otherwise marked as 0, and 40 pixels is an adjustable threshold value which needs to be adjusted according to the resolution of the picture in actual application;

[0036] The (3) is the proportion of the points correctly predicted in all points, which is the accuracy rate.

[0037] Preferably, in the S5, the improved UNet network model comprises:

[0038] S501: residual module realizes down-sampling;

[0039] S502: bilinear interpolation realizes up-sampling;

[0040] S503: JS divergence loss function;

[0041] S504: SoftArgmax obtains key point coordinates.

[0042] Preferably, in S3;

[0043] The UNet model is different from the convolutional neural network in that the UNet model does not have a full connection layer and directly outputs a feature map to realize end-to-end semantic segmentation of an image. The left path is down-sampled by 2 times through 2 times 3*3 convolution and maximum pooling at each layer, and the feature map is reduced by 1 / 2 in length and width. The right path is up-sampled by 2 times through 2 times 3*3 convolution and 2*2 transposed convolution at each layer, and the feature map is enlarged by 2 times in length and width. The down-sampling path extracts low-level features layer by layer, and the up-sampling path recovers high-level features. The low-level features and the high-level features are fused through the skip connection to further enrich the detailed features of the feature map and improve the segmentation accuracy of the model.

[0044] Preferably, in S501, the existing UNet network model uses a normal convolution layer for down-sampling. Network degradation occurs when the network is deepened. The use of a residual module can avoid gradient disappearance and explosion and solve the problem of network degradation. In the first layer of the down-sampling path, a 3*3 convolution kernel is used to expand a 1-channel image to a 20-channel image, and then a 3*3 convolution kernel is used to reduce the 20-channel image to a 10-channel image. Edge padding is used to ensure that the sizes of the feature maps are the same. A residual module is used for the output feature map of the first layer. The first layer in the residual module is a 3*3 convolution kernel, and a convolution with a step of 2 is used to down-sample the feature map to reduce the length and width by 1 / 2. A short-circuit connection uses a 1*1 convolution kernel with a step of 2 to adjust the channel number and size of the feature map, so that the outputs of the two paths can be added because their channel numbers and sizes are the same. The down-sampling path uses a residual module four times to realize down-sampling, and the size of the feature map is reduced to 1 / 16 of the length and width of the original input image.

[0045] Preferably, in S502, the up-sampling of the existing UNet network model uses transposed convolution, and the up-sampling is realized by zero padding and then convolution, but because there is no sub-pixel interpolation, it is easy to cause a checkerboard effect; the bilinear interpolation selects 4 adjacent points and performs interpolation in the x and y directions, so the bilinear interpolation is selected to replace the transposed convolution for up-sampling. The down-sampling output feature map of the corresponding layer is first convolved using a 1*1 convolution kernel, so that the number of channels of the down-sampling output feature map is the same as that of the output feature map of the corresponding up-sampling layer, thereby directly adding to realize the fusion of low-level features and high-level features to further enrich the detailed features of the feature map.

[0046] Preferably, in S503, the label heat map is Gaussian distribution, and only the MSE loss function is used for supervised training, and the output heat map only tends to the label heat map in value and cannot tend to the Gaussian distribution in shape. The network model outputs a 4-channel heat map, and each channel heat map corresponds to a key point. The heat map can be regarded as a probability distribution map. During training, there may be a case that there are multiple peaks in a heat map. In order to accurately predict the coordinates of the key points, it is necessary to make the output heat map as Gaussian distribution as possible. The peak coordinates of the Gaussian distribution are the coordinates of the key points. The KL divergence is a method for measuring the similarity of two probability distributions. The smaller the KL divergence, the more similar the two probability distributions are. When they are completely equal, the KL divergence is 0. The JS divergence is a variant of the KL divergence, which solves the non-symmetry problem of the KL divergence.

[0047] Preferably, in S504, the loss value between the Gaussian heat map label and the feature map output by the neural network is calculated by the MSE loss function for supervised learning, and the maximum response point index is calculated by using Argmax to realize key point detection. Because Argmax is not derivable, it cannot be connected to the last feature map output layer of the neural network to realize a full differential model, resulting in the inability to realize end-to-end training. SoftArgmax is derivable, and the maximum response point index can be directly connected to the feature map output layer of the network to calculate the loss value between the key point coordinate label and the maximum response point index, and supervised learning is realized, thereby realizing end-to-end training.

[0048] (Three) beneficial effects

[0049] Compared with the prior art, the present application provides a hand back or palm vein image key point and ROI positioning method based on an improved UNet, which has the following beneficial effects:

[0050] The back of the hand or palm vein image key point and ROI positioning method based on the improved UNet, solves the model degradation problem caused by gradient disappearance by adding a residual module in the down-sampling path of the UNet network, improves the feature information extraction capability of the network; the transposed convolution layer in the up-sampling path of the UNet network is improved to bilinear interpolation up-sampling to avoid the chessboard effect and reduce the network parameters; the JS divergence loss function is used to supervise the distribution of the last feature map, so that the output feature map tends to be Gaussian distribution, improves the multi-peak problem of the feature map, and improves the model accuracy; the SoftArgmax is used to calculate the key point coordinates of the last feature map to realize end-to-end training, which is convenient for network model training; the results show that (accuracy, parameter quantity, operation quantity, operation time): the accuracy of the UNet network is 97.6%, the accuracy of the improved UNet network model is 98.6%, which is 1% higher than that of the original UNet network model; the size of the original Unet network model is 35.0M, and the file of the improved UNet network model is 1.16M, which is 96.7% smaller than that of the original UNet network model, and higher accuracy than the original UNet network model is obtained under the condition of greatly reducing the model parameters; the running time of the original UNet network is 146.12ms, and the running time of the improved UNet network is 59.57ms, which is 59.2% higher than that of the original UNet network. The results show that the improved UNet model of the present study can realize the positioning of the back of the hand or palm key point (ROI extraction) of the non-contact back of the hand or palm vein image, and is suitable for practical deployment in edge embedded system and other low resource platforms. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. In the drawings:

[0052] Figure 1 Some sample pictures of the back of the hand vein image data set built by the present application;

[0053] Figure 2 Some sample pictures of the back of the hand vein image data set built by the present application;

[0054] Figure 3 The U-Net network structure diagram of the present application;

[0055] Figure 4 The residual module diagram of the present application;

[0056] Figure 5 The two common residual module diagrams of the present application;

[0057] Figure 6 The improved UNet network structure diagram of the present application;

[0058] Figure 7 Residual module for improved UNet network of the application;

[0059] Figure 8 Figure for the method for extracting the region of interest of the self-built hand dorsal vein image dataset of the application;

[0060] Figure 9 Training change curve diagram of the improved UNet model of the application;

[0061] Figure 10 Thermal map obtained by the improved UNet model of the application with or without short circuit connection;

[0062] Figure 11 Thermal map obtained by the improved UNet model of the application using different up-sampling methods;

[0063] Figure 12 Thermal map obtained by the improved UNet model of the application using different loss functions for training;

[0064] Figure 13 Thermal map obtained by the improved UNet model of the application before and after the improvement;

[0065] Figure 14 Test results of the improved UNet model of the application on the self-built database;

[0066] Figure 15 Test results of the improved UNet model of the application on the Jilin University dataset;

[0067] Figure 16 Positioning key points of the Jilin University hand dorsal vein image dataset by the traditional image processing method of the application;

[0068] Figure 17 Positioning key points of the self-built hand dorsal vein image dataset by the traditional image processing method of the application;

[0069] Figure 18 Figure for positioning key points of the hand heart vein image dataset of the application.

[0070] In the figure: Figure 1 In the figure, (a) clean background, (b) background with fluorescent light, (c) background with subject clothing pattern, (d) incomplete occlusion, (e) wearing a ring; Figure 2 In the figure, (a) no jewelry placed in the center, (b) offset placement, (c) wearing a watch, (d) wearing a ring; Figure 5 In the figure, (a) conventional residual module, (b) bottleneck residual module; Figure 8 In the figure, (a) hand dorsal vein image key point positioning and region of interest selection, (b) region of interest image adjusted by linear interpolation;Figure 15 In the middle, (a) background clean, (b) background with fluorescent light, (c) background with subject clothing pattern, (d) incomplete occlusion; Figure 16 In the middle, (a) original image, (b) image binarization, (c) detect hand back contour, (d) use convex hull and convex defect to detect locatable key points; Figure 17 In the middle, (a) positioning effect when occlusion is not complete, (b) positioning effect when background has clothing reflection. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all.

[0072] The hand back or palm vein image key point and ROI positioning method based on improved UNet includes the following steps:

[0073] S1: Experimental system machine collects on-site settings;

[0074] S2: Self-built dataset;

[0075] Near-infrared imaging technology is used to obtain hand back or palm vein images;

[0076] The left and right hands of the subject are irradiated with near-infrared light, the lens bottom is provided with an 850nm filter to prevent visible light from entering the camera, and the packaged shell can be optionally provided with an 850nm filter. The current supply of the collection system remains unchanged during the entire experiment. The hand back or palm vein image collection site is at South China Agricultural University. The researchers have no restrictions on the angle, distance and shooting background of the subject's hand. Five pictures are taken for each person's open left and right hands, and the picture size is 1920 pixels x 1080 pixels. Thus, a total of 2200 unrestricted hand back or palm vein image datasets are created for 220 people. The dataset has clean background, background interference, incomplete occlusion and wearing a ring, and some sample pictures are shown in Figure 1 ;

[0077] Public datasets are adopted. The infrared camera of the vein image collection device is placed horizontally on the desktop, the hand back or palm surface is parallel to the collection device, and the angle can be any within the parallel plane. It is stopped 10-15cm above the lens. A total of 2220 hand back or palm vein pictures are collected for 222 people, 10 pictures for each person's right hand. Some sample pictures are shown in Figure 2 ;

[0078] S3: Establish UNet model;

[0079] The UNet model is different from the convolutional neural network in that the UNet model does not have a full connection layer to realize classification or regression, but directly outputs a feature map to realize end-to-end semantic segmentation of an image. The UNet model is left-right symmetrical and U-shaped as shown in Figure 3 The left path of the UNet model is first subjected to 2 times of 3*3 convolution and then subjected to maximum pooling downsampling, so that the length and width of the feature map are each reduced by 1 / 2, and the downsampling is performed 4 times. The right path of the UNet model is first subjected to 2 times of 3*3 convolution and then subjected to 2*2 transposed convolution upsampling, so that the length and width of the feature map are each enlarged by 2 times, and the downsampling feature map of the same layer is fused after being cropped. The downsampling path is related to the encoder for extracting low-level features layer by layer, and the upsampling path is equivalent to the decoder for recovering high-level features. The low-level features and the high-level features are fused through the skip connection to further enrich the detailed features of the feature map and improve the segmentation accuracy of the model.

[0080] S4: Establishing a residual module;

[0081] With the deepening of the network, the feature information that can be extracted by the network is more abundant, and the performance of the network is also improved. However, research shows that with the deepening of the network layers, the problem of gradient explosion and gradient disappearance will occur, leading to network degradation. In view of the problem of gradient explosion and gradient disappearance caused by the deepening of the network layers during network training, He Kai-ming et al. proposed a residual module to solve the problem of network degradation. The residual module is shown in Figure 4 The residual module is divided into two paths. The first path is that the input x is output as a residual F(x) through at least two convolution layers and a Relu activation function. The second path is that the original input x is directly added to the output F(x) of the first path through a short circuit connection. The final output of the residual module is H(x) = F(x) + x. The reason for this is that residual learning is easier than direct learning of original features. When the residual F(x) is 0, the residual module only performs an identity mapping, and the network performance will not decrease. However, in fact, the residual F(x) will not be 0, which will enable the residual module to learn new features based on the input features, thereby improving the performance of the network.

[0082] The residual module is divided into a regular residual module and a bottleneck residual module as shown in Figure 5 The bottleneck structure refers to the change in the number of channels, that is, the original number of channels is reduced and then increased. The 1*1 convolution has the function of adjusting the number of channels (or the function of reducing and increasing the feature dimension), so as to reduce the dimension of the input x and input it into the 3*3 convolution layer for convolution operation, thereby improving the calculation efficiency.

[0083] S5: An improved UNET model for key point positioning of a dorsal or palm vein image;

[0084] S6: calculating the extraction of the region of interest by using the ROI calculation method;

[0085] S7: evaluating the model;

[0086] In S6, the extraction of the region of interest specifically includes the following steps:

[0087] S601: the key points of the four finger webs from the thumb to the little finger are in the order of 0, 1, 2, and 3;

[0088] S602: a line segment L13 is drawn with the key points 1 and 3 as the endpoints, and the length is l13;

[0089] S603: a mid-perpendicular line L45 is drawn with the midpoint 4 of the line segment L13 as the center, and the length is l13 / 3;

[0090] S604: a perpendicular line L67 is drawn with the point 5 as the center, and the length is l13;

[0091] S605: a square 6789 is drawn with the line segment L67 as the side;

[0092] S606: the region of interest image of the square 6789 is extracted, and the size of the region of interest image is adjusted to 256x256 pixels by linear interpolation.

[0093] In S7, the model evaluation standard is calculated as follows:

[0094]

[0095]

[0096]

[0097] (1) to calculate the Euclidean distance between the predicted coordinates (x1, y1) and the label coordinates (x2, y2);

[0098] (2) if the distance is less than 40 pixels, it is considered as correct prediction and marked as 1, otherwise marked as 0. 40 pixels is an adjustable threshold, which needs to be adjusted according to the resolution of the picture in actual application;

[0099] (3) the proportion of points that are correctly predicted in all points is the accuracy.

[0100] In S5, the improved UNet network model includes:

[0101] S501: the residual module realizes down-sampling;

[0102] The existing UNet network model uses ordinary convolution layers for downsampling, which can cause network degradation when the network is deepened. The use of residual modules can avoid gradient disappearance and explosion to solve the problem of network degradation. In the first layer of the downsampling path, a 3x3 convolution kernel is used to expand the 1-channel image to 20 channels, and then a 3x3 convolution kernel is used to reduce the 20-channel image to 10 channels. Edge padding is required to ensure that the feature map sizes are the same. The output feature map of the first layer is processed using a residual module as shown in Figure 7 The first layer in the residual module is a 3x3 convolution kernel, and the step size of the convolution is 2 to achieve downsampling, which reduces the feature map length and width by 1 / 2. The short connection uses a 1x1 convolution kernel with a step size of 2 to adjust the number of channels and the size of the feature map to ensure that the outputs of the two paths are the same in terms of channel number and size, which can be added together. The downsampling path uses 4 residual modules to achieve downsampling, and the feature map size is reduced to 1 / 16 of the original input image length and width.

[0103] S502: Bilinear interpolation for upsampling;

[0104] The existing UNet network model uses transpose convolution for upsampling, which increases the feature map size by zero padding and then performs convolution to achieve upsampling. However, because there is no sub-pixel interpolation, it is easy to cause a checkerboard effect. Bilinear interpolation selects 4 adjacent points and performs interpolation in the x and y directions, so the upsampling path uses bilinear interpolation instead of transpose convolution for upsampling. First, the output feature map of the corresponding layer is convolved using a 1x1 convolution kernel to make the number of channels of the downsampling output feature map the same as that of the corresponding upsampling layer output feature map, so that the low-level features and high-level features can be directly added to further enrich the detailed features of the feature map.

[0105] S503: JS divergence loss function;

[0106] The label heat map is a Gaussian distribution, and only the MSE loss function is used for supervised training. The output heat map only approaches the label heat map in value and cannot approach the Gaussian distribution in shape. The network model outputs a 4-channel heat map, and each channel corresponds to a key point. The heat map can be considered as a probability distribution map. During training, there may be multiple peaks in a heat map. In order to accurately predict the coordinates of the key points, the output heat map needs to be as Gaussian as possible. The peak coordinates of the Gaussian distribution are the coordinates of the key points. KL divergence is a method to measure the similarity of two probability distributions. The more similar the two probability distributions are, the smaller the KL divergence is. When they are completely equal, the KL divergence is 0. JS divergence is a variant of KL divergence, which solves the non-symmetry problem of KL divergence.

[0107] S504: SoftArgmax to obtain key point coordinates;

[0108] The loss value between the Gaussian heat map label and the feature map output by the neural network is calculated by the MSE loss function for supervised learning, and the maximum response point index is calculated by Argmax to realize key point detection. Because Argmax is not derivable, it cannot be connected to the last feature map output layer of the neural network to realize the full differential model, which leads to the inability to realize end-to-end training. SoftArgmax is derivable and can be directly connected to the feature map output layer of the network to calculate the maximum response point index, and then use the L2 loss function to calculate the loss value between the key point coordinate label and the maximum response point index for supervised learning to realize end-to-end training.

[0109] Experimental results and analysis:

[0110] Model training

[0111] The training server environment of this study is Windows 10, 64-bit operating system, Intel(R) Core I7-10700F CPU, 32GB memory, NVIDIA Quadro RTX5000 graphics card, using Python and PyTorch framework. 1100 images are randomly selected from 2200 images, and the 4 key points of the images are labeled using the open source data labeling software Labelme. The training set and the test set are randomly divided in the ratio of 8:2, the batch training size is 32, the Adam optimization method is used, the initial learning rate is 0.001, the equal interval adjustment learning rate strategy (StepLR) is used, the total training period number epochtotal=200, the image is horizontally and vertically flipped during training, and random brightness contrast and histogram equalization are added to the data enhancement means to improve the generalization ability of the model. The training loss function is the average of the JS divergence loss function and the Euclidean distance loss function. The average accuracy and loss value change curve of the model training is shown in Figure 9 It can be seen that when the iteration number is close to 100, the model begins to gradually converge.

[0112] Residual module implementation of down-sampling comparison test

[0113] In order to evaluate whether the residual module improves the performance of the network model, the improved UNet model with short circuit connection and without short circuit connection is used for training.

[0114] (1) Qualitative comparison

[0115] The network model without short circuit connection is used for training, and when the back of the hand or the palm has a certain pitch angle, the heat map obtained is not very accurate, and the heat map of 2 points does not tend to Gaussian distribution. Therefore, the residual module with short circuit connection is used to obtain more accurate heat map, as shown in Figure 10 ;

[0116] (2) Quantitative comparison

[0117] The test results on the self-built data set are shown in Table 1.

[0118] As can be seen from Table 1, the model accuracy is reduced by 0.6 percentage points, the inference time is shortened by about 5 ms, and the model is reduced by 0.02M when using a network model without short circuit connection for training. The short circuit connection path has a convolution layer that consumes a certain inference time, and the short circuit connection can extract more rich feature information, solve the gradient vanishing problem, and improve the model accuracy.

[0119] Table 1 Comparison test of improved UNet model with and without short circuit connection

[0120]

[0121]

[0122] Comparison test of improved transposed convolution to bilinear interpolation

[0123] In order to evaluate whether the improved transposed convolution to bilinear interpolation improves the performance of the network model, the improved UNet model using transposed convolution and bilinear interpolation is trained.

[0124] (1) Qualitative comparison

[0125] When the network model using transposed convolution is trained, the finger opening is not enough and the adjacent finger web is misjudged, so the bilinear interpolation is used for upsampling, and the output heat map is more accurate. The specific situation is shown in Figure 11 ;

[0126] (2) Quantitative comparison

[0127] The test results on the self-built data set are shown in Table 2.

[0128] As can be seen from Table 2, the model accuracy is improved by 1.9 percentage points, the inference time is shortened by about 8 ms, and the model is reduced by 0.02M when using a network model with bilinear interpolation for training. The convolution operation of transposed convolution consumes a certain inference time, and the convolution is performed after the feature map size is increased by zero padding interpolation, without sub-pixel interpolation, while bilinear interpolation is sub-pixel interpolation, and the color transition is more natural, improving the model accuracy.

[0129] Table 2 Comparison test of improved UNet model using transposed convolution and bilinear interpolation

[0130]

[0131] Comparison test of supervision effect of different loss functions

[0132] In order to evaluate whether the use of JS divergence loss function to supervise the Gaussian distribution of feature map can improve the performance of network model, different loss functions are used to supervise the training of improved UNet model.

[0133] (1) Qualitative comparison

[0134] When only MSELoss is used for supervised training, the feature map will appear multi-peak phenomenon, which affects the judgment of peak value. Therefore, MSELoss, JS divergence loss function and Euclidean distance loss function are used for training to solve the multi-peak problem of heat map, and the specific situation is shown in Figure 12

[0135] (2) Quantitative comparison

[0136] The test results of the self-built data set are shown in Table 3.

[0137] From Table 3, it can be seen that when only MSELoss is used for supervised training, the model accuracy is 92.3%, when only JS divergence loss function is used to supervise the probability distribution of feature map, the model accuracy is 95%, when only Euclidean distance loss function is used to supervise the coordinate point, the model accuracy is 95.5%, and when the probability distribution of feature map is supervised on the basis of the coordinate point, the model accuracy is improved by 3.1 percentage points. JS divergence loss function makes the probability distribution of feature map tend to Gaussian distribution, but the peak coordinate point does not tend to the label coordinate point; Euclidean distance loss function makes the peak coordinate point tend to the label coordinate point, but the probability distribution of feature map does not tend to Gaussian distribution, and there are multiple peaks, which affects the judgment of SoftArgmax to the maximum response point; using MSELoss, JS divergence loss function and Euclidean distance loss function for common supervision can simultaneously make the probability distribution of feature map and the peak coordinate point tend to the probability distribution and coordinate point of the label, and improve the accuracy of the model.

[0138] Table 3 Comparison of improved UNet model using different loss functions

[0139]

[0140] Comparison of improved and unimproved UNet models

[0141] In order to be able to deploy network model on platforms with limited computing resources such as embedded systems to realize practical application, it is necessary to evaluate the performance of the model before and after improvement. The inference time, accuracy and model size obtained by training the improved UNet network and the original UNet network under the same training set and loss function and testing on the same test set are shown in Table 4, and the obtained heat map is shown in Figure 13 .​

[0142] Table 4 Comparison of UNet model before and after improvement

[0143]

[0144] As can be seen from Table 4, the accuracy of the original UNet network model is 97.6%, and the model file is 35.0M, which will occupy a lot of embedded system memory and is not suitable for deployment on an embedded system. The improved UNet network model has an accuracy of 1 percentage point, and the model file is 1.16M, which can be deployed on an embedded system for practical application. Because the residual module avoids gradient explosion and disappearance, it solves the problem of deep network model degradation, so that the network can still extract rich feature information even with a significant reduction in model parameters, and the model size is reduced to nearly 1 / 30, still achieving a higher accuracy than the original UNet network model. The test results of the improved UNet model for hand back or palm key point positioning and ROI extraction of hand back or palm vein images according to the key points are shown in Figure 14 As can be seen from the figure, the improved UNet model can achieve high-accuracy hand back or palm key point positioning in complex backgrounds;

[0145] Test results of improved UNET on Jilin University dataset

[0146] Directly using the network model trained on the self-built dataset to test 220 Jilin University dataset images, the accuracy is 80%, and directly using the Jilin University dataset for training, the test accuracy is 98.1%. Using 1100 self-built dataset images and 300 Jilin University dataset images as the training set, 220 Jilin University dataset images as the test set, and other training parameters and strategies unchanged, the accuracy on the test set is 99.5%, and the test results of the hand back or palm vein image ROI extraction of the Jilin University dataset are shown in Figure 15 Because there is a large difference between the self-built dataset and the Jilin University dataset, the resolution of the Jilin University dataset image is only 640x480 pixels, and the hand back or palm is basically located at the center of the image, and the self-built dataset lacks similar image data to the Jilin University dataset. By supplementing a small amount of Jilin University dataset images on the self-built dataset, the network model trained can achieve high accuracy on both datasets;

[0147] Comparison of key point positioning by traditional image processing method

[0148] First, the original image is converted to a grayscale image, then binarized, the contour of the binarized image is detected, and the key points are positioned by performing convex hull and convex defect detection on the hand back or palm contour. The traditional image processing method is used to position the key points of the Jilin University dataset, and the implementation method and effect areFigure 7 The same key point positioning method was used on the self-built dataset, and the results are shown in FIG. 6. The results of the traditional image processing method and the improved UNet network model are shown in FIG. 7. Figure 16 and Figure 17 It can be seen that in the case of clean background and complete hand, the traditional image processing method can achieve good key point positioning, but in the case of occlusion caused by shadow when the back of the hand or the palm is curved and the existence of background interference, the correct back of the hand or palm binary image cannot be obtained after the image is binarized, resulting in that the correct key points cannot be positioned after contour detection, convex hull and convex defect detection;

[0149] In view of the difficulty of traditional image processing algorithm in key point positioning of non-contact back of the hand or palm vein image, a method for positioning the back of the hand or palm key point based on deep learning is proposed. Based on the UNet network structure, a residual module is added to the down-sampling path of the UNet network to solve the model degradation problem caused by gradient disappearance, improve the network feature information extraction ability; the transpose convolution layer in the up-sampling path of the UNet network is improved to bilinear interpolation up-sampling to avoid the chessboard effect and reduce the network parameters; the JS divergence loss function is used to supervise the distribution of the final feature map, so that the output feature map tends to be Gaussian distribution, which improves the feature map multi-peak problem and improves the model accuracy; the SoftArgmax is used to calculate the key point coordinates of the final feature map to realize end-to-end training, which is convenient for network model training; the improved UNet network model proposed in this paper has an accuracy of 98.6%, which is 1% higher than the original UNet network model; the improved UNet network model file is 1.16M, which is reduced to nearly 1 / 30 of the original UNet network model file; in the case of a substantial reduction in model parameters, a higher accuracy than the original UNet network model is achieved, which is suitable for deployment on embedded systems.

Claims

1. A method for locating key points and ROIs in hand dorsum or palm vein images based on an improved UNet, characterized in that, Includes the following steps: S1: Experimental system data acquisition on-site settings; S2: Self-built dataset; Near-infrared imaging technology was used to acquire vein images of the back or palm of the hand; an unconstrained dataset of vein images of the back or palm of the hand was created, including images with clean backgrounds, background interference, incomplete occlusion, and images of people wearing rings. S3: Build the UNet model; S4: Establish the residual module; the residual module is the bottleneck residual module; S5: An improved UNet model for keypoint localization in images of veins on the back or palm of the hand; the improved UNet network model includes: S501: The residual module implements downsampling; in the first layer of the downsampling path, a 3×3 convolution kernel is used to expand the 1-channel image to 20 channels, and then a 3×3 convolution kernel is used to reduce the 20-channel image to 10 channels. Edge padding is used to ensure that the feature map size is the same. The output feature map of the first layer is processed by the residual module. The first layer of the residual module is a 3×3 convolution kernel with a stride of 2 to achieve downsampling, which reduces the length and width of the feature map by 1 / 2. The short-circuit connection uses a 1×1 convolution kernel with a stride of 2 to adjust the number of channels and size of the feature map to ensure that the outputs of the two paths can be added when the number of channels and size are the same. The downsampling path uses the residual module to implement downsampling 4 times, and the feature map size is reduced to 1 / 16 of the length and width of the original input image. S502: Bilinear interpolation is used to achieve upsampling. Bilinear interpolation uses four adjacent points and performs interpolation in the x and y directions respectively. Therefore, the upsampling path uses bilinear interpolation to replace the transposed convolution for upsampling. First, the downsampled output feature map of the corresponding layer is convolved with a convolution kernel of size 1×1 so that the number of channels of the downsampled output feature map is the same as that of the corresponding upsampled layer output feature map, so that they are directly added to achieve the fusion of low-level features and high-level features. S503: JS divergence loss function; uses the JS divergence loss function to supervise the distribution of feature maps, making the output feature maps tend to a Gaussian distribution; S504: SoftArgmax obtains keypoint coordinates; SoftArgmax directly connects to the feature map output layer of the network to calculate the maximum response point index, and then uses the L2 loss function to calculate the loss value between the keypoint coordinate label and the maximum response point index for supervised learning, realizing end-to-end training; S6: Use ROI calculation methods to calculate the extraction of the region of interest; S7: Evaluate the model.

2. The method for locating key points and ROIs in hand dorsum or palm vein images based on the improved UNet according to claim 1, characterized in that: In step S6, the extraction of the region of interest specifically includes the following steps: S601: The key points of the four webs from the thumb to the little finger are in the order of 0, 1, 2, 3; S602: Draw line segment L13 with key points 1 and 3 as endpoints, with a length of l13; S603: Draw a perpendicular line L45 with a length of l13 / 3, using the midpoint 4 of line segment 13 as the midpoint. S604: Draw a perpendicular line L45 with point 5 as the center, with length l13; S605: Construct a square 6789 with line segment L67 as its side; S606: Extract the region of interest image of square 6789 and resize the region of interest image to 256×256 pixels using linear interpolation.

3. The method for locating key points and ROIs in hand dorsum or palm vein images based on the improved UNet according to claim 1, characterized in that: In S7, the calculation formula for the model evaluation criteria is as follows: (1) (2) (3)； The (1) is to calculate the Euclidean distance between the predicted coordinates (x1, y1) and the label coordinates (x2, y2); the (2) is to record 1 if the distance is less than 40 pixels as a correct prediction, otherwise it is recorded as 0. 40 pixels is an adjustable threshold. In practical applications, it needs to be adjusted according to the resolution of the image; the (3) is to calculate the proportion of correctly predicted points to all points, which is the accuracy.

4. The method for locating key points and ROIs in hand dorsum or palm vein images based on the improved UNet according to claim 1, characterized in that: In S3; The UNet model implements semantic segmentation tasks. In the left path, each layer first performs two 3×3 convolutions followed by max pooling downsampling, reducing the feature map's width and height by half, for a total of four downsampling operations. In the right path, each layer first performs two 3×3 convolutions followed by 2×2 transposed convolutions for upsampling, enlarging the feature map's width and height by a factor of 2. The feature maps are then cropped and fused with the downsampled feature maps from the same layer. The downsampling path is equivalent to the encoder extracting low-level features layer by layer, while the upsampling path is equivalent to the decoder recovering high-level features. Low-level and high-level features are fused through skip connections.

5. The method for locating key points and ROIs in hand dorsum or palm vein images based on the improved UNet according to claim 1, characterized in that: In the S503, the label heatmap is a Gaussian distribution, and the network model outputs a 4-channel heatmap. Each channel of the heatmap corresponds to a key point. The heatmap can be regarded as a probability distribution map, and the peak coordinates of the Gaussian distribution are the coordinates of the key point.

Images