Gesture-based non-contact dynamic video palmprint identification method

Abstract

The application relates to a gesture-based non-contact dynamic video palmprint identification method, which comprises the following steps: S1: collecting a hand video sample and detecting hand key points; S2: calculating the weighted sum of distance scores and displacement scores of each frame image as frame scores, and selecting key frames according to minimum frame scores; S3: adopting a two-stage palmprint preprocessing method, positioning and assisting positioning through hand key points and an improved Deeplab V3+ semantic segmentation network, segmenting interdigital valley bottom regions and detecting interdigital valley bottom points to position a palmprint region of interest ROI; and S4: identifying the palmprint region of interest ROI. The application fuses action recognition and palmprint recognition to improve the security of palmprints, can eliminate frames that are relatively fuzzy and not suitable for palmprint sample extraction, evaluates the best frames as library samples and query samples, and can accurately position the palmprint region of interest ROI in a complex background, so that the complexity and positioning error rate are reduced.

Description

Technical Field

[0001] This application relates to the field of biometric recognition technology, specifically to a gesture-based non-contact dynamic video palmprint recognition method. Background Technology

[0002] Biometric identification technologies, such as palmprint, fingerprint, and facial recognition, offer significant advantages in terms of convenience and security as novel identity authentication methods. However, most palmprint and fingerprint recognition devices currently employ contact-based methods. In certain settings, such as hospitals, contact-based devices can easily lead to cross-infection and virus transmission. Furthermore, repeated contact inevitably causes mechanical wear and contamination of the collection devices, especially in high-traffic areas, which can interfere with the collected samples. Therefore, contactless identification is of great importance.

[0003] Most existing palmprint recognition research focuses on images, with relatively little research on video palmprint recognition. Compared to static images, dynamic videos contain more information, making them more conducive to feature extraction and less susceptible to forgery. However, videos also have a larger data volume and more redundant information. Therefore, the key to video palmprint recognition is how to eliminate redundant data and extract the most suitable data for palmprint recognition.

[0004] Meanwhile, some non-contact palmprint recognition technologies suffer from low robustness to complex background interference during palmprint sample extraction, and some unconstrained palmprint preprocessing methods perform poorly in complex backgrounds. Existing low-constraint palmprint preprocessing methods require the hand to align with auxiliary positioning points or lines displayed in the system, or to maintain a specific position relative to the camera, in order to be recognized. While these auxiliary positioning methods ensure the accuracy of palmprint extraction, they place high demands on hand placement and posture, limit the location of the detection device, and reduce the user experience. Summary of the Invention

[0005] The purpose of this invention is to provide a gesture-based non-contact dynamic video palmprint recognition method that integrates action recognition and palmprint recognition to improve palmprint security. This method can eliminate frames that are too blurry or unsuitable for palmprint sample extraction, and evaluate the best-performing frame in the palmprint video as a library sample and query sample for matching to make identity verification decisions. Furthermore, it can accurately locate the region of interest (ROI) of the palmprint in complex backgrounds, effectively avoiding most background interference while ensuring computational speed, thus reducing complexity and positioning error rate.

[0006] The technical solution adopted in this invention is: a non-contact dynamic video palmprint recognition method based on gestures, comprising the following steps:

[0007] S1: Collect hand video samples and extract the key points of the hand in each frame of the hand video samples;

[0008] S2: Calculate the distance score and displacement score of each frame image based on the hand key points, where the distance score represents the distance of the palm from the camera and the displacement score represents the distance the palm moves between two adjacent frames; calculate the weighted sum of the distance score and displacement score as the frame score, and select the image with the smallest frame score as the key frame. If two or more frames have the same smallest frame score, then randomly select one frame as the key frame.

[0009] S3: The keyframes are processed using the two-stage palmprint preprocessing method TSPP. In the first stage, the approximate range of the finger valley bottom is delineated by using hand key points for assisted localization. In the second stage, the improved Deeplab V3+ semantic segmentation network is used to segment the finger valley bottom region, and the lowest point of the finger valley bottom is used as a reference point to locate the palmprint region of interest (ROI).

[0010] The improved Deeplab V3+ semantic segmentation network replaces the original backbone feature extraction network Xception with a lightweight MobileNetV2 network, and adds a feature fusion layer of 1 / 2 size to the decoder of the original Deeplab V3+ semantic segmentation network; the improved Deeplab V3+ semantic segmentation network is then trained.

[0011] S4: Perform palmprint recognition on the Region of Interest (ROI) of the palmprint.

[0012] Furthermore, the specific method of step S1 is as follows:

[0013] S101: The hands, especially the area between the fingers, remain open and stationary relative to the camera in shots that avoid skin-toned backgrounds;

[0014] S102: Extract key points of the hand from each frame of the image using a key point localization model.

[0015] Furthermore, the specific method of step S2 is as follows:

[0016] S201: Calculate the coordinates of the center point P of the palm based on the coordinates of key points on the hand;

[0017] S202: Based on the maximum x-coordinate among all hand key point coordinates h1 Minimum x-coordinate h2 , the maximum value of the ordinate y h1 and the minimum value of the ordinate y h2Calculate the ratio of the hand area to the overall image area in each frame of the hand image, and use this ratio as the distance score S for each frame of the hand image. a ;

[0018] S203: Based on the weighted sum of displacement fractions of the same hand keypoint in each frame of the hand image and the adjacent frames, calculate the displacement fraction D between each frame of the hand image and the previous frame. a+ And the displacement fraction D of the next frame a- And take the displacement fraction D a+ and displacement fraction D a- The mean value is used as the displacement fraction D for each frame of the hand image. a Among them, in calculating the displacement fraction D a+ and displacement fraction D a- At that time, the farther the key hand points are from the center point P of the palm, the lower their weighting.

[0019] S204: The distance fraction S of each frame of the hand image a and the displacement fraction D of each frame of the hand image a Weighted, as the frame score W for each frame of the image. a The image with the lowest score is selected as the keyframe. If two or more images have the same lowest frame score, one frame is randomly selected as the keyframe. The frame score W for each image is calculated as follows: a At that time, the distance fraction S of the hand image a The weight is greater than the displacement fraction D of the hand image. a The weight.

[0020] Furthermore, in step S102, the MediaPipe model is used to extract key points of the hand, and the number of key points of the hand extracted by the MediaPipe model is 21.

[0021] The formula for calculating the coordinates of the center point P of the palm is as follows:

[0022]

[0023]

[0024] Where, x P Let y be the x-coordinate of the center point P of the palm. P Let x be the x-coordinate of the center point P of the palm; n Let y be the x-coordinate of the nth hand key point. n Let be the ordinate of the nth hand key point, where n is the hand key point number, n = 0, 1, ..., 20;

[0025] The distance fraction S a The specific calculation formula is as follows:

[0026]

[0027] x h1 =max(x0, x1, ... x 20 );

[0028] x h2 =min(x0, x1, ... x) 20 );

[0029] y h1 =max(y0, y1, ... y 20 );

[0030] y h2 =min(y0, y1, ... y 20 );

[0031] Where a is the frame number, w is the length of the hand image, and s is the width of the hand image;

[0032] The displacement fraction D a The specific calculation formula is as follows:

[0033]

[0034]

[0035] D a =(D a- +D a+ ) / 2;

[0036] Where, d n- Let d be the displacement of the nth hand keypoint between the current frame and the previous frame. n+ Let d be the displacement of the nth hand keypoint between the current frame and the next frame. np This is the distance between the nth hand keypoint in the current frame and the center point P of the palm.

[0037] Score W per frame a The specific calculation formula is as follows:

[0038] W a =D a +S a ×16.

[0039] Furthermore, the specific steps for extracting the Region of Interest (ROI) from the palm print in step S3 are as follows:

[0040] S301: Calculate the coordinates (x, y, y) of the upper left corner of the positioning frame between the index and middle fingers based on key hand points. f1 y f1 ) and the coordinates of the lower right corner (xf2 y f2 ), and the coordinates of the upper left corner of the positioning frame between the ring finger and little finger (x r1 y r1 ) and the coordinates of the lower right corner (x r2 y r2 This allows for the definition of the finger gaps between the index and middle fingers, as well as the finger gaps between the ring and little fingers. The finger gap areas between the index and middle fingers and between the ring and little fingers are then cut out using these definition frames.

[0041] S302: Reduce the area between the index and middle fingers and the area between the ring and little fingers by 10% inward from the boundary to reduce interference from non-skin background areas.

[0042] S303: The cropped finger gap regions of the index and middle fingers and the finger gap regions of the ring and little fingers are used as sample sets and input into the improved DeepLab V3+ semantic segmentation network for training. The sample set is randomly divided into a training set and a validation set. The improved DeepLab V3+ semantic segmentation network adds a feature fusion layer of size 1 / 2 to the original DeepLabV3+ semantic segmentation network structure. The output of the feature fusion layer is fused with the features of the decoder to segment the final finger gap images of the index and middle fingers and the finger gap images of the ring and little fingers.

[0043] S304: Binarize the segmented finger gap image and scan the pixels line by line to find the lowest point of the pixels in the non-hand area and match it with the coordinates (x, y) of the upper left corner of the finger gap positioning box between the index and middle fingers. f1 y f1 The coordinates (x) of the upper left corner of the positioning frame between the ring finger and the little finger. r1 y r1 Add the two points together to obtain the positions of the valley points between the index and middle fingers and the valley points between the ring and little fingers in the entire image; let the length of the line connecting the two valley points be d, and let the position be 0.3d perpendicular to the midpoint of the line and downward. Use 0.9d as the side length to cut out the region of interest (ROI) of the palm print.

[0044] Furthermore, the upper left corner coordinate (x) of the positioning frame between the index and middle fingers... f1 y f1 ) and the coordinates of the lower right corner (x f2 y f2 The specific calculation formula is as follows:

[0045] x f1 =min(x5, x6, x9, x) 10 )+[max(x5, x6, x9, x 10 )-min(x5, x6, x9, x10 ) / 10];

[0046] y f1 =min(y5, y6, y9, y 10 )+[max(y5, y6, y9, x 10 )-min(y5, y6, y9, y 10 ) / 10];

[0047] x f2 =max(x5, x6, x9, x 10 )-[max(x5, x6, x9, x 10 )-min(x5, x6, x9, x 10 ) / 10];

[0048] y f2 =max(y5, y6, y9, y 10 )-[max(y5, y6, y9, x 10 )-min(y5, y6, y9, y 10 ) / 10];

[0049] The coordinates of the upper left corner of the positioning frame between the ring finger and little finger (x r1 y r1 ) and the coordinates of the lower right corner (x r2 y r2 The specific calculation formula is as follows:

[0050] x r1 =min(x 13 x 14 x 17 x 18 )+[max(x 13 x 14 x 17 x 18 )-min(x 13 x 14 x 17 x 18 ) / 10];

[0051] y r1 =min(y 13 y 14 y 17 y 18 )+[max(y 13 y 14 y 17 y 18 )-min(y 13 y 14 y 17y 18 ) / 10];

[0052] x r2 =max(x 13 x 14 x 17 x 18 )-[max(x 13 x 14 x 17 x 18 )-min(x 13 x 14 x 17 x 18 ) / 10];

[0053] y r2 =max(y 13 y 14 y 17 y 18 )-[max(y 13 y 14 y 17 y 18 )-min(y 13 y 14 y 17 y 18 ) / 10].

[0054] The beneficial effects of this invention are as follows:

[0055] (1) The present invention can filter key frames (KFS) from hand motion information screening videos. It estimates the weighted average by the amplitude of hand movement and the distance between the hand and the acquisition device to eliminate frames that are too blurry or unsuitable for palm print sample extraction. It also evaluates the best-performing frame in the palm print video as a library sample and query sample for matching to make identity verification decisions.

[0056] (2) This invention proposes a two-stage palmprint preprocessing method, TSPP (Two-Stage Palmprint Preprocessing). TSPP is a low-constraint palmprint preprocessing algorithm for complex backgrounds, which can improve the user's freedom of use and the robustness of the video palmprint recognition system to complex backgrounds. TSPP adopts a two-stage structure, first narrowing the localization range of key points in the region of interest (ROI) of the palmprint, and then using semantic segmentation to perform secondary localization of key points within the range. This can effectively avoid most background interference while ensuring computational speed. Experimental results show that TSPP has low user constraints and can accurately locate the region of interest (ROI) of the palmprint in complex backgrounds. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of key points of the hand and the center point of the palm in an embodiment of the present invention; wherein, (a) is a schematic diagram of the location of key points of the hand, and (b) is a schematic diagram of the location number of key points of the hand and the location of the center point of the palm;

[0060] Figure 3 This is a schematic diagram of the keyframe filtering results in an embodiment of the present invention; wherein, (a) is the filtered keyframe, (b) is the frame where the hand is blurred due to motion, and (c) is the frame where the hand target is too small;

[0061] Figure 4 This is a flowchart of the two-stage palmprint preprocessing method TSPP in an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the auxiliary positioning area in the first stage of the two-stage palmprint preprocessing method TSPP according to an embodiment of the present invention, and a schematic diagram of the coordinate system establishment;

[0063] Figure 6 This is a schematic diagram of the improved Deeplab V3+ semantic segmentation network structure in an embodiment of the present invention;

[0064] Figure 7This is a schematic diagram of the finger gap area extracted by the two-stage palm print preprocessing method TSPP in an embodiment of the present invention; wherein, (a) is the finger gap area directly cut out in the first process; (b) is the area of ​​the reduced area relative to the original area; and (c) is the reduced finger gap area.

[0065] Figure 8 This is a schematic diagram of the Region of Interest (ROI) location extracted from the palm print in an embodiment of the present invention;

[0066] Figure 9 Images showing the effects of different methods for dividing finger gaps;

[0067] Figure 10 This is a schematic diagram illustrating the extraction of the Region of Interest (ROI) of the palm print of the same hand under different conditions in an embodiment of the present invention.

[0068] Figure 11 This is a schematic diagram illustrating the extraction of the Region of Interest (ROI) of palm prints for different hands under different conditions in an embodiment of the present invention.

[0069] Figure 12 ROC curves for different encoding methods. Detailed Implementation

[0070] To better understand the above-described objects, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention; however, the invention may be practiced in other ways different from those described herein, and therefore, the invention is not limited to the specific embodiments disclosed below.

[0071] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art described herein. The terms “first,” “second,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “connected” or “linked” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” “right,” etc., are used only to indicate relative positional relationships, which change accordingly when the absolute position of the described object changes.

[0072] like Figures 1 to 8 As shown, a gesture-based non-contact dynamic video palmprint recognition method includes the following steps:

[0073] S1: Collect hand video samples and extract the key points of the hand in each frame of the hand video samples. The specific method is as follows:

[0074] S101: The hands, especially the area between the fingers, remain open and stationary relative to the camera in shots that avoid skin-toned backgrounds;

[0075] S102: Extract key points of the hand from each frame of the image using a key point localization model.

[0076] In this embodiment of the invention, the hand video sample NCHU_DP consists of 120 video segments, each consisting of six segments filmed under different backgrounds and lighting conditions for twenty hands. Each video segment is in a compatible 1920*1080*60fps format, and the duration of each segment is controlled to be 8 to 10 seconds. During video capture, the hand needs to be open for approximately five seconds to facilitate background processing of previously captured frames during gesture recognition, thus shortening processing time and improving efficiency. This embodiment of the invention uses the MediaPipe model to extract hand key points. The MediaPipe model extracts 21 hand key points, and the positions of these key points are as follows: Figure 2 As shown.

[0077] S2: Calculate the distance score and displacement score for each frame of image based on the key points of the hand. The distance score represents the distance between the palm print and the camera. The distance from the camera is measured by the area occupied by the hand in the entire video frame. The closer the hand is to the camera, the more palm print information can be obtained. The displacement score represents the distance the palm print part moves between two adjacent frames. The closer the key point of the current frame is to the key point of the previous frame, the smaller the displacement between the two frames. That is, the smaller the displacement of the palm in the two frames, which can reduce the blurring of the palm print image caused by the movement of the camera relative to the lens.

[0078] The distance score and displacement score are weighted to obtain the frame score for each image. The image with the lowest frame score is selected as the keyframe. If two or more images have the same lowest frame score, one of them is randomly selected as the keyframe. The specific method for selecting keyframes is as follows:

[0079] S201: Calculate the coordinates of the center point P of the palm based on the coordinates of key hand points. The specific calculation formula is as follows:

[0080]

[0081]

[0082] Where, x P Let y be the x-coordinate of the center point P of the palm. P Let x be the x-coordinate of the center point P of the palm;n Let y be the x-coordinate of the nth hand key point. n Let y be the ordinate of the nth hand key point, where n is the hand key point number, n = 0, 1, ..., 20.

[0083] S202: Based on the maximum x-coordinate among all hand key point coordinates h1 Minimum x-coordinate h2 , the maximum value of the ordinate y h1 and the minimum value of the ordinate y h2 Calculate the ratio of the hand area to the overall image area in each frame of the hand image, and use this ratio as the distance score S for each frame of the hand image. a The specific calculation formula is as follows:

[0084]

[0085] x h1 =max(x0, x1, ... x 20 );

[0086] x h2 =min(x0, x1, ... x) 20 );

[0087] y h1 =max(y0, y1, ... y 20 );

[0088] y h2 =min(y0, y1, ... y 20 );

[0089] Where a is the frame number, w is the length of the hand image, and s is the width of the hand image.

[0090] S203: Based on the weighted sum of displacement fractions of the same hand keypoint in each frame of the hand image and the adjacent frames, calculate the displacement fraction D between each frame of the hand image and the previous frame. a+ And the displacement fraction D of the next frame a- And take the displacement fraction D a+ and displacement fraction D a- The mean value is used as the displacement fraction D for each frame of the hand image. a Among them, in calculating the displacement fraction D a+ and displacement fraction D a- At that time, the farther the key hand points are from the center point P of the palm, the lower their weighting. The specific calculation formula is as follows;

[0091]

[0092]

[0093] D a =(D a- +D a+ ) / 2;

[0094] Where, d n- Let d be the displacement of the nth hand keypoint between the current frame and the previous frame. n+ Let d be the displacement of the nth hand keypoint between the current frame and the next frame. np This is the distance between the nth hand keypoint in the current frame and the center point P of the palm.

[0095] S204: The distance fraction S of each frame of the hand image a and the displacement fraction D of each frame of the hand image a Weighted, as the frame score W for each frame of the image. a The image with the lowest frame score is selected as the keyframe. If two or more images have the same lowest frame score, one frame is randomly selected as the keyframe. The filtering results are as follows: Figure 3 As shown. Specifically, in calculating the frame score W for each frame of the image... a At that time, the distance fraction S of the hand image a The weight is greater than the displacement fraction D of the hand image. a The weights. The frame score W for each frame of the image. a The specific calculation formula is as follows: In this embodiment of the invention, the distance fraction S of each frame of the hand image is... a The weight is the displacement fraction D of each frame of the hand image. a Sixteen times the weight:

[0096] W a =D a +S a ×16.

[0097] This invention uses hand displacement calculation to filter out clearer palm print images. The calculation of distance relative to the lens can avoid misjudgment caused by the hand's relatively small area leading to large actual hand movements and small displacement. Furthermore, the distance calculation ensures the proportion of the hand in the image, thereby ensuring the amount of palm print texture information extracted.

[0098] To accommodate finger tremors caused by certain diseases, such as Parkinson's syndrome and hyperthyroidism, or physiological factors such as emotional stress and overwork; and considering that finger movements have a smaller impact on image acquisition compared to palm movements, and that a certain degree of finger tremor does not affect keyframe selection, this embodiment of the invention reduces the weight of the fingers, especially the fingertips, and increases the weight of the palm. This effectively reduces the impact of fingertip movements and appropriately amplifies the influence of the palm on displacement, thereby achieving the goal of selecting keyframes with smaller hand displacements and reducing the impact of blurring caused by tremors.

[0099] S3: As Figure 4 As shown, the keyframes are processed using the two-stage palmprint preprocessing method TSPP. The first stage uses hand key points to assist in localization, delineating the approximate range of the finger gaps required for locating the palmprint region of interest (ROI). The second stage uses an improved Deeplab V3+ semantic segmentation network to segment the finger gaps, and binarizes the segmentation results to scan the pixels line by line to obtain the position of the finger gap valley points. Based on the position of the finger gap valley points, the palmprint region of interest (ROI) is cut out.

[0100] The structure of the improved Deeplab V3+ semantic segmentation network is as follows: Figure 7 As shown, the original backbone feature extraction network Xception of the Deeplab V3+ semantic segmentation network is replaced with the lightweight MobileNetV2 network, and a feature fusion layer of 1 / 2 size is added to the decoder of the original Deeplab V3+ semantic segmentation network. The constructed improved Deeplab V3+ semantic segmentation network still needs to be trained.

[0101] S301: Calculate the coordinates (x, y, y) of the upper left corner of the positioning frame between the index and middle fingers based on key hand points. f1 y f1 ) and the coordinates of the lower right corner (x f2 y f2 ), and the coordinates of the upper left corner of the positioning frame between the ring finger and little finger (x r1 y r1 ) and the coordinates of the lower right corner (x r2 y r2 This involves defining the positioning frames for the space between the index and middle fingers, and the space between the ring and little fingers. The positioning frames are then used to cut out the areas for the space between the index and middle fingers, and the space between the ring and little fingers. The specific locations are as follows: Figure 5 As shown.

[0102] The coordinates of the upper left corner of the positioning frame between the index and middle fingers (x) f1 y f1 ) and the coordinates of the lower right corner (xf2 y f2 The specific calculation formula is as follows:

[0103] x f1 =min(x5, x6, x9, x) 10 )+[max(x5, x6, x9, x 10 )-min(x5, x6, x9, x 10 ) / 10];

[0104] y f1 =min(y5, y6, y9, y 10 )+[max(y5, y6, y9, x 10 )-min(y5, y6, y9, y 10 ) / 10];

[0105] x f2 =max(x5, x6, x9, x 10 )-[max(x5, x6, x9, x 10 )-min(x5, x6, x9, x 10 ) / 10];

[0106] y f2 =max(y5, y6, y9, y 10 )-[max(y5, y6, y9, x 10 )-min(y5, y6, y9, y 10 ) / 10].

[0107] The coordinates of the upper left corner of the positioning frame between the ring finger and little finger (x r1 y r1 ) and the coordinates of the lower right corner (x r2 y r2 The specific calculation formula is related to the upper left corner coordinate (x) of the positioning frame between the index and middle fingers. f1 y f1 ) and the coordinates of the lower right corner (x f2 y f2 The calculation formula is the same; simply replace the coordinates of keypoints 5, 6, 9, and 10 with the coordinates of keypoints 13, 14, 18, and 17 respectively to obtain the following:

[0108] x r1 =min(x 13 x 14 x 17 x 18 )+[max(x 13 x 14 x 17 x 18)-min(x 13 x 14 x 17 x 18 ) / 10];

[0109] y r1 =min(y 13 y 14 y 17 y 18 )+[max(y 13 y 14 y 17 y 18 )-min(y 13 y 14 y 17 y 18 ) / 10];

[0110] x r2 =max(x 13 x 14 x 17 x 18 )-[max(x 13 x 14 x 17 x 18 )-min(x 13 x 14 x 17 x 18 ) / 10];

[0111] y r2 =max(y 13 y 14 y 17 y 18 )-[max(y 13 y 14 y 17 y 18 )-min(y 13 y 14 y 17 y 18 ) / 10].

[0112] S302: Reduce the area between the index and middle fingers and the area between the ring and little fingers by 10% inward from the boundary to reduce interference from non-skin-colored background areas.

[0113] S303: Input the cropped index and middle finger gap areas and the ring and little finger gap areas as sample sets into the improved Deeplab V3+ semantic segmentation network for training. The sample sets are randomly divided into training and validation sets; Figure 6As shown, the improved Deeplab V3+ semantic segmentation network, compared to the original Deeplab V3+ semantic segmentation network, adds the original... Figure 1 A feature fusion layer of size / 2 is used. The output of this feature fusion layer is fused again with the features extracted by the decoder to improve segmentation accuracy, resulting in the final images of the gaps between the index and middle fingers and the gaps between the ring and little fingers, thus improving the accuracy of the gap image boundaries. In this embodiment of the invention, the gap region extracted by the two-stage palmprint preprocessing method TSPP is as follows: Figure 7 As shown.

[0114] To improve positioning accuracy and reduce computational load, this embodiment of the invention replaces the core feature extraction network Xception of the DeeplabV3+ semantic segmentation network with the lightweight MobileNetV2 network, forming an improved DeeplabV3+ semantic segmentation network. The improved Deeplab V3+ semantic segmentation network uses convolutional modules to learn and extract the finger gap contours of the finger gap region, and then uses deconvolutional modules to upsample and restore the image resolution until it matches the input image resolution, outputting a segmentation map marking the finger gap region. Since palmprint ROI localization has strict requirements for the effectiveness of finger gap segmentation, in order to better locate the boundaries of hand skin and improve the segmentation details so as to better cope with more complex situations in actual deployment, this embodiment of the invention divides the training samples into training and test sets in a 9:1 ratio for training. It integrates DeepLabV3+ and MobilenetV2 to solve the problems of large number of network parameters, high computational complexity, slow speed, and heavy dependence on hardware resources. Furthermore, this embodiment of the invention adds a feature fusion of size 1 / 2 on the basis of the original DeepLabV3+ network structure, which further improves the segmentation accuracy of the network, enabling the improved DeepLab V3+ semantic segmentation network to achieve accurate segmentation of finger gap regions in complex backgrounds.

[0115] S304: Binarize the segmented finger gap image and scan the pixels line by line to find the lowest point of the pixels in the non-hand area and match it with the coordinates (x, y) of the upper left corner of the finger gap positioning box between the index and middle fingers. f1 y f1 The coordinates (x) of the upper left corner of the positioning frame between the ring finger and the little finger. r1 y r1 Adding these together gives the positions of the valley points between the index and middle fingers, and between the ring and little fingers, within the entire image; for example... Figure 8 As shown, let the length of the line connecting the two finger gap valleys be d. Perpendicular to the midpoint of the line and downward at a distance of 0.3d, use 0.9d as the side length to cut out the region of interest (ROI) of the palm print.

[0116] S4: Perform palmprint recognition on the Region of Interest (ROI) of the palmprint.

[0117] The following experiments verify the palmprint Region of Interest (ROI) segmentation results of this invention. In the semantic segmentation results, compared to the original DeepLabV3+ network with MobilenetV2 as the backbone feature extraction network, the improved DeepLabV3+ semantic segmentation network, under the same conditions, achieves improved accuracy in semantic segmentation due to the addition of a feature fusion step. The segmentation results are as follows: Figure 9 As shown, (a) is the original image of the finger gap; (b) is the segmentation result of the Ground truth network; (c) is the segmentation result of the UNet with ResNet50 network; (d) is the segmentation result of the UNet with VGG16 network; (e) is the segmentation result of the PSPNet with ResNet50 network; (f) is the segmentation result of the PSPNet with MobilenetV2 network; (g) is the segmentation result of the DeepLabV3+ with Xpection network; (h) is the segmentation result of the DeepLabV3+ with MobilenetV2 network; and (i) is the network segmentation result of the embodiment of the present invention.

[0118] Figure 10 and Figure 11 The figures show the extraction results of the Region of Interest (ROI) of palm prints for the same hand under different scenarios and ambient lighting conditions in this embodiment of the invention, as well as the extraction results of the ROI of palm prints for different hands under different scenarios and lighting conditions. As can be seen from the figures, the ROIs of palm prints collected in this embodiment of the invention are relatively stable under different conditions.

[0119] In this embodiment of the invention, different networks were used to extract the Region of Interest (ROI) of the palmprint from the hand video sample NCHU_DP. After extracting the ROI, different encoding methods, such as PalmCode, OrdinalCode, FusionCode, CompCode, RLOC, BOCV, E-BOCV, DRCC, DOC, MTCC, and 2TCC, were used to verify the accuracy of palmprint recognition. The comparison results are shown in Table 1. If a sample collected from a single hand is called a palmprint sample class, then the matching between different palmprint samples collected from the same hand under different conditions is called intra-class matching, that is, matching between samples collected from the same hand under different conditions. The matching between samples collected from different hands is called inter-class matching. A person's left and right hands cannot be considered samples of the same class, but rather belong to different classes. If a palm print collected from the same hand is mistakenly identified as a palm print from a different hand, the probability of this incorrect identification is called the False Reject Rate (FRR); the probability that samples collected from different hands are identified as the same hand is called the False Accept Rate (FAR); when the two probabilities are equal, the value obtained is the Equal Error Rate (EER). In this embodiment of the invention, when using the DHN method for verification, the EER is 0.0877%, meaning that the probability of palm print recognition error is approximately 0.0877%.

[0120] Table 1 EER of different encoding methods on NCHU-DP

[0121]

[0122]

[0123] Samples collected through different semantic segmentation networks can be encoded using different palmprint encoding methods. Under the same sample conditions, the accuracy of semantic segmentation results is positively correlated with the degree of palmprint matching. The better the performance of the semantic segmentation network, the more accurate the ROI localization. Due to limitations of the acquisition equipment and environment, the ROI of palmprints acquired without constraints will have relatively lower texture detail compared to those acquired using specific instruments and controlled lighting. The relatively small number of palmprint samples in the dataset can lead to relatively high EER (Earning Error) for some palmprint recognition methods.

[0124] The Receiver Operating Characteristic (ROC) curve is an evaluation curve that integrates information from the False Acceptance Rate (FAR) and False Rejection Rate (FRR). In plotting the ROC curve, the relationship between FAR and FRR at different thresholds is used to comprehensively measure these two rating indicators. When plotting FAR on the x-axis and FRR on the y-axis, the smaller the area enclosed by the curve and the coordinate axis, the better the authentication effect of the palmprint recognition system. Furthermore, the y-axis of the ROC curve can also be replaced by the Genuine Acceptance Rate (GAR), i.e., as shown below. Figure 12 As shown, the closer the ROC curve, plotted with the true acceptance rate (GAR) as the vertical axis, is to the upper part of the coordinate axis, the better the certification effect.

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A gesture-based non-contact dynamic video palmprint identification method, characterized in that, The method comprises the following steps: S1: collecting a hand video sample and extracting hand key points of each frame of image in the hand video sample; S2: calculating a distance score and a displacement score of each frame of image based on the hand key points, wherein the distance score represents a distance of a palm from a camera, and the displacement score represents a moving distance of the palm between adjacent two frames; calculating a weighted sum of the distance score and the displacement score as a frame score, and selecting an image with the minimum frame score as a key frame; if two or more images have the same minimum frame score, randomly selecting one frame as the key frame; S3: processing the key frame by using a two-stage palm print preprocessing method TSPP, in a first stage, positioning by hand key points to demarcate a general range of interdigital valley bottoms, in a second stage, segmenting the interdigital valley bottom region by using an improved Deeplab V3+ semantic segmentation network, taking the lowest point of the interdigital valley bottom as a reference point to position a palm print region of interest ROI; The improved Deeplab V3+ semantic segmentation network uses a lightweight MobileNetV2 network to replace an original backbone feature extraction network Xception of the Deeplab V3+ semantic segmentation network, and adds a feature fusion layer with a size of 1 / 2 in a decoder decoder of the original Deeplab V3+ semantic segmentation network; and the improved Deeplab V3+ semantic segmentation network is trained; S4: performing palm print recognition on the palm print region of interest ROI.

2. The gesture-based non-contact dynamic video palmprint identification method according to claim 1, wherein, The specific method of the step S1 is: S101: the hand, especially the interdigital region, is kept open and still relative to the lens in the picture avoiding the skin color background; S102: extracting hand key points of each frame of image by a key point positioning model.

3. The gesture-based non-contact dynamic video palmprint identification method according to claim 2, characterized in that, The specific method of the step S2 is: S201: calculating a coordinate of a palm center point P according to hand key point coordinates; S202: Based on the maximum x-coordinate among all hand key point coordinates h1 Minimum x-coordinate h2 , the maximum value of the ordinate y h1 and the minimum value of the ordinate y h2 The ratio of the hand area to the overall image area in each frame of the hand image is calculated and used as the distance score S for each frame of the hand image. a ; S203: According to the displacement score of each hand key point in the same hand image and the adjacent frames before and after, respectively calculate the displacement score D of each hand image and the previous frame a+ and the displacement score D of the next frame a- , and take the average of the displacement score D a+ and the displacement score D a- as the displacement score D of each frame of hand image a ; wherein, when calculating the displacement score D a+ and the displacement score D a- , the farther the hand key point from the palm center point P, the lower the weighted weight. S204: the distance score S of each frame of hand image is weighted with the displacement score D of each frame of hand image as the frame score W of each frame of image a a a The image with the minimum frame score is selected as the key frame. If two or more images have the same minimum frame score, a frame is randomly selected as the key frame. a a a The weight of the distance score S of the hand image is greater than the weight of the displacement score D of the hand image.​​​​ 4. The gesture-based non-contact dynamic video palmprint identification method according to claim 3, characterized in that, In the step S102, the MediaPipe model is used to extract the hand key points, and the number of the hand key points extracted by the MediaPipe model is 21; The coordinate calculation formula of the palm center point P is as follows: wherein x P is the horizontal coordinate of the palm center point P, y P is the horizontal coordinate of the palm center point P; x n is the horizontal coordinate of the nth hand key point, y n is the vertical coordinate of the nth hand key point, n is the hand key point number, n = 0, 1, …, 20; The distance score S a The specific calculation formula is as follows: x h2 = min(x0, x1,... x 20 ); y h1 = max(y0, y1,... y 20 ); y h2 = min(y0, y1,... y 20 ); Wherein, a is the frame number, w is the length of the hand image, and s is the width of the hand image; The displacement fraction D a The specific formula for the calculation is as follows: D a = (D a- + D a+ ) / 2; wherein d n- is the displacement of the n-th hand key point between the current frame and the previous frame, d n+ is the displacement of the n-th hand key point between the current frame and the next frame, d np is the distance between the n-th hand key point and the palm center point P in the current frame. The score W of each frame of image a The specific calculation formula is as follows: W a = D a + S a x 16.

5. The gesture-based non-contact dynamic video palmprint identification method according to claim 3, characterized in that, The specific steps of cutting out the palm print region of interest ROI in the step S3 are: S301: Calculate the upper left corner coordinates (x f1 ,y f1 ) and the lower right corner coordinates (x f2 ,y f2 ) of the index finger and middle finger interdigital positioning frame and the upper left corner coordinates (x r1 ,y r1 ) and the lower right corner coordinates (x r2 ,y r2 ) of the ring finger and little finger interdigital positioning frame according to the hand key points, thereby delimiting the index finger and middle finger interdigital positioning frame and the ring finger and little finger interdigital positioning frame, and cutting out the index finger and middle finger interdigital region and the ring finger and little finger interdigital region using the positioning frame; S302: reducing the cut-out index and middle finger interdigital region and the cut-out ring finger and thumb interdigital region by 10% towards the boundary, so as to reduce the interference of non-skin color background region; S303: inputting the cut-out index and middle finger interdigital region and the cut-out ring finger and thumb interdigital region as a sample set into the improved Deeplab V3+ semantic segmentation network for training, and randomly dividing the sample set into a training set and a verification set; the improved Deeplab V3+ semantic segmentation network adds a feature fusion layer with a size of 1 / 2 in the structure of the original DeepLab V3+ semantic segmentation network, the output of the feature fusion layer is twice fused with the features of the decoder, and finally the index and middle finger interdigital picture and the ring finger and thumb interdigital picture are segmented. S304: binarize the segmented interdigital picture, and scan the pixels row by row to find the lowest point of the pixels in the non-hand region and add the coordinates (x f1 ,y f1 ) of the upper left corner of the positioning frame of the interdigital space between the index finger and the middle finger and the coordinates (x r1 ,y r1 ) of the upper left corner of the positioning frame of the interdigital space between the ring finger and the little finger to obtain the positions of the valley points of the interdigital spaces between the index finger and the middle finger and between the ring finger and the little finger in the entire picture; record the length of the line connecting the two valley points as d, the position vertically downward from the midpoint of the line by a distance of 0.3d, and cut out a palm print region of interest (ROI) with a side length of 0.9d.

6. The gesture-based non-contact dynamic video palmprint identification method according to claim 5, characterized in that, The specific calculation formula of the left upper corner coordinate (x f1 ,y f1 ) and the right lower corner coordinate (x f2 ,y f2 ) of the index finger and middle finger interdigital gap positioning frame is as follows: x f1 = min(x5, x6, x9, x 10 )+ [max(x5, x6, x9, x 10 )-min(x5, x6, x9, x 10 ) / 10]; y f1 = min(y5, y6, y9, y 10 )+ [max(y5, y6, y9, x 10 )- min(y5, y6, y9, y 10 ) / 10]; x f2 = max(x5, x6, x9, x 10 )- [max(x5, x6, x9, x 10 )-min(x5, x6, x9, x 10 ) / 10]; y f2 = max(y5, y6, y9, x 10 )- [max(y5, y6, y9, x 10 )- min(y5, y6, y9, y 10 ) / 10]; The specific calculation formula of the left upper corner coordinate (x r1 ,y r1 ) and the right lower corner coordinate (x r2 ,y r2 ) of the ring and little finger gap positioning frame is as follows: x r1 = min(x 13 ,x 14 ,x 17 ,x 18 )+[max(x 13 ,x 14 ,x 17 ,x 18 )-min(x 13 ,x 14 ,x 17 ,x 18 ) / 10] ; y r1 = min(y 13 ,y 14 ,y 17 ,y 18 )+[max(y 13 ,y 14 ,y 17 ,y 18 )-min(y 13 ,y 14 ,y 17 ,y 18 ) / 10] x r2 = max(x 13 , x 14 , x 17 , x 18 )-[max(x 13 , x 14 , x 17 , x 18 )-min(x 13 , x 14 , x 17 , x 18 ) / 10] y r2 = max(y 13 ,y 14 ,y 17 ,y 18 )- [max(y 13 ,y 14 ,y 17 ,y 18 )- min(y 13 ,y 14 ,y 17 ,y 18 ) / 10]

Images