Palm vein heart rate in-vivo detection and palm recognition method and palm vein heart rate in-vivo detection and palm recognition device

Abstract

The invention discloses a palm vein heart rate in-vivo detection and palm recognition method and device, and relates to the field of biological information recognition, identity authentication and security defense of automobile door unlocking, and the method comprises the following steps: step 1, shooting a video containing a palm, and tracking the movement of the palm; the video is an infrared video; 2, the ROI of the palm is determined, a heart rate signal is extracted, and the heart rate signal is enhanced; 3, whether the palm is a living palm or not is judged according to whether the heart rate signal shows periodicity or not; and 4, performing palm recognition on the living palm, and outputting a palm vein recognition result. According to the technical scheme provided by the invention, living body detection is carried out based on camera non-contact video palm heart rate detection, the method is suitable for illumination change or dark conditions, prosthesis attacks can be effectively prevented, and the anti-attack capability and safety of a palm vein recognition system are improved.

Description

Technical Field

[0001] The present invention relates to the field of biometric information recognition, identity authentication and security defense for unlocking automobile doors, and in particular to a method and device for palm vein heart rate liveness detection and palm recognition. Background Art

[0002] The principle of palm vein recognition is to use the property of hemoglobin in human blood to absorb near-infrared light to obtain palm vein images, extract feature values ​​through positioning, enhancement, feature extraction and other algorithms, and compare them with the original palm vein template to verify personal identity.

[0003] Palm vein recognition technology is a highly secure biometric technology. It has the following characteristics: 1) Highly anti-counterfeiting: Veins are hidden under the skin and are difficult to forge or copy.

[0004] 2) Unique identification: Each person’s palm vein pattern is unique.

[0005] 3) Non-contact: No need to directly contact the equipment, reducing the risk of cross infection, more hygienic and convenient.

[0006] Due to the high security and convenience of palm vein recognition technology, it is widely used in various fields, such as finance, security, medical care, access control systems, etc., bringing higher security and convenience to people's lives and work.

[0007] In actual use, palm vein recognition still faces the problem of prosthetic attacks, such as using materials similar to human tissue to make palm models, trying to imitate the palm vein characteristics of real palms; using high-definition palm photos or playing palm videos. If the attacker successfully conducts a prosthetic attack, it will lead to the leakage of sensitive information, property loss or other serious consequences. Once the palm vein recognition system is frequently attacked by live bodies and successfully broken through, the trust of users and related institutions in this technology will drop significantly.

[0008] In the existing technology for dealing with prosthetic attacks, methods such as multimodal recognition fusion or adding a deep learning liveness detection module are usually adopted.

[0009] Multimodal recognition fusion is to combine other biometric recognition technologies, such as fingerprint recognition, face recognition, etc., to perform multimodal fusion recognition. This can increase the difficulty of attack and improve the security of the system. For example, on the basis of palm vein recognition, users are required to perform fingerprint recognition or face recognition at the same time, and the identity is confirmed only when multiple biometric features match. However, this method requires multiple biometric sensors or multiple cameras, such as fingerprint recognition requires fingerprint sensors, face recognition requires cameras, iris recognition requires iris scanners, etc. Each sensor has its own unique working principle and technical requirements, which increases the technical complexity and hardware complexity. At the same time, a better hardware platform is required, such as a higher-performance processor and larger memory, which increases the cost of hardware.

[0010] The deep learning liveness detection module uses a deep neural network model to learn and distinguish between real live bodies and fake attacks (such as photos, video replays, 3D molds, etc.). However, this method is highly dependent on the training data set. If there are situations outside the data set, the detection accuracy will decrease or misjudgment will occur.

[0011] Therefore, technicians in this field are committed to developing a palm vein heart rate liveness detection and palm recognition method and device to solve the above-mentioned problems existing in the prior art. Summary of the invention

[0012] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to effectively deal with the attack of the prosthesis in palm vein recognition.

[0013] To achieve the above object, the present invention provides a method for palm vein heart rate liveness detection and palm recognition, comprising the following steps: Step 1, shooting a video containing a palm and tracking the movement of the palm; the video is an infrared video; Step 2, determining the ROI area of ​​the palm and extracting the heart rate signal, and enhancing the heart rate signal; Step 3: judging whether the palm is a live palm according to whether the heart rate signal shows periodicity; Step 4: Perform palm recognition on the living palm and output the palm vein recognition result.

[0014] Furthermore, when the video is shot in step 1, a stable light source is irradiated on a fixed area on the surface of the palm skin.

[0015] Furthermore, step 2 includes the following sub-steps: Step 2.1, determining the ROI area according to the key points of the palm, wherein the key points include the palm, the base of the palm, or the metacarpophalangeal joint; Step 2.2, taking spatial average of the two-dimensional matrix corresponding to the single-channel palm vein of each frame of the infrared video, and extracting a set of discrete time signal sequences including the heart rate signal, namely, the PPG signal; Step 2.3, extracting the heart rate signal from the PPG signal; Step 2.4: Enhance the heart rate signal by filtering and noise reduction.

[0016] Furthermore, the video shot in step 1 also includes a color video, and correspondingly, in step 4, a recognition result of the fusion of palm print and palm vein is output.

[0017] Furthermore, in the step 2.2, the two-dimensional matrices corresponding to the single-channel palm vein of each frame of the infrared video and the three color channels of R, G, and B of the color video are spatially averaged to extract four groups of discrete time signal sequences containing the heart rate signal, namely, the PPG signal.

[0018] Furthermore, in step 2.2, the spatial average may be replaced by variance or standard deviation, median or extreme point detection.

[0019] Furthermore, before step 1, a palm detection algorithm is used to perform palm detection. Step 1 is initiated only when it is detected that the captured image contains a palm, wherein the palm detection algorithm may be an edge detection algorithm, a template matching method or a deep learning model.

[0020] Furthermore, in step 1, a palm tracking algorithm is used to track the movement of the palm; wherein the palm tracking algorithm includes the following steps: (1) using the key point information of the palm in the first frame image detected by the palm detection algorithm as feature points; (2) when entering the next frame image, searching for the position most similar to the feature point in the first frame image in the new image area; (3) estimating the movement of the palm based on the position change of the feature point between consecutive frames; (4) continuously updating the set of feature points as time goes by; if some feature points cannot be successfully matched in several consecutive frames, or the motion trajectory is unstable, they are deleted from the feature point set and new feature points are re-extracted from the image; at the same time, the motion estimation is continuously adjusted according to the new matching results to achieve continuous tracking of the palm.

[0021] The present invention also provides a palm vein heart rate liveness detection and palm recognition device, the device comprising: A collection module, which includes an infrared light source, an infrared camera and an image sensor, is located at the front end or in the area interacting with the user, and can also be installed on a movable part of the device, and is responsible for capturing a video containing the palm; A main control module, which is located inside the device and may be installed on a circuit board or integrated in an embedded system, is connected to the acquisition module, receives the video captured by the acquisition module, and processes it; A human body detection module, which is located at the outer edge of the palm vein module or close to the side where the user may appear, and maintains a bidirectional connection with the acquisition module and the main control module; when the human body detection module detects that a human body is approaching, it will send a wake-up signal to the acquisition module, and the acquisition module will switch from a low-power state to a working state; the acquisition module will also feedback status information to the human body detection module to help the human body detection module adjust its own working mode; the human body detection module will send the detected human body approaching signal to the main control module, and the main control module will decide whether to start the workflow of the entire palm vein module; the main control module will also send configuration information and control instructions to the human body detection module.

[0022] Furthermore, the acquisition module also includes a narrowband filter or a color camera; the narrowband filter can eliminate interference caused by external light sources; the color camera is used to acquire color video.

[0023] The present invention provides a palm vein heart rate liveness detection and palm recognition method and device which have at least the following technical effects: 1. The technical solution provided by the present invention is to detect liveness based on the heart rate of the palm of a camera contactless video. It is adaptable to changes in lighting or in darkness, and can accurately distinguish between live palms and prostheses (such as photos, videos, 3D prosthetic hands, etc.), effectively prevent prosthesis attacks, and improve the anti-attack capability and security of the palm vein recognition system.

[0024] 2. The technical solution provided by the present invention can be used to upgrade the algorithm of existing equipment (palm vein recognition equipment, palm print recognition equipment, face recognition equipment) without replacing hardware, and can also be integrated with existing palm vein recognition equipment and systems without large-scale hardware modification.

[0025] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram of the optical model of skin tissue; Figure 2 It is a method flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0028] In order to deal with the problem of prosthesis attack during palm vein recognition, an embodiment of the present invention proposes a palm vein heart rate liveness detection and palm recognition method and device, which uses infrared light to penetrate the skin, irradiate the hemoglobin in the blood, and detect the light absorption and reflection of the hemoglobin, so as to infer the heart rate. Specifically, the color change signal caused by the heartbeat is extracted from the palm area, and the heart rate signal is further detected. When the heart contracts, the blood flow rate will accelerate, and the color of the blood will also change due to the fullness of the hemoglobin. These changes will cause the light reflected on the skin to have slight intensity and fluctuations. Although this slight color change cannot be observed by the human eye, it can be captured by a camera and then amplified by an image processing method. The embodiment of the present invention also designs a non-contact palm vein recognition device with integrated hardware and software, which can realize fast and real-time palm vein identity authentication.

[0029] Example 1 like Figure 2 As shown, an embodiment of the present invention provides a method for palm vein heart rate liveness detection and palm recognition, comprising the following steps: Step 1: Shoot a video containing the palm and track the movement of the palm; the video is infrared video; Step 2: determine the ROI area of ​​the palm and extract the heart rate signal, and enhance the heart rate signal; Step 3: judging whether the palm is a live palm based on whether the heart rate signal shows periodicity; Step 4: Perform palm recognition on the live palm and output the palm vein recognition result.

[0030] Example 2 Based on Example 1, when shooting a video in step 1, a stable light source is irradiated on a fixed area on the surface of the palm skin. When a stable light source is irradiated on a fixed area on the surface of the human skin, the surface of the skin will show periodic color changes.

[0031] In particular, when a palm is detected, if the infrared light source is not on, the infrared light is turned on to illuminate the palm and shoot a video.

[0032] When the heart is squeezed and beats, the blood volume in the blood vessels will change within a certain period of time. The main components of blood are water and hemoglobin. Compared with other tissues of the human body, such as bones and muscles, hemoglobin is more sensitive to light. Hemoglobin can absorb more light sources of different lengths in the light. Every time the heart beats, the blood volume in the blood vessels will increase or decrease accordingly, and the corresponding amount of hemoglobin will increase or decrease accordingly. Because bones and muscles are relatively constant. The amount of hemoglobin absorbed by part of the length of the light source and reflected or refracted back will change periodically with the beating of the heart, and the periodic changes are amplified by digital signals, which is the heartbeat signal, that is, the heart rate. Although this weak color change cannot be observed by the human eye, it can be captured by the camera and amplified by image processing methods.

[0033] Figure 1 This is a schematic diagram of the optical model of skin tissue. In the epidermis, when light hits the skin, part of the light will directly produce mirror reflection and propagate to the camera through the air, and part of the light will be absorbed by the skin as it enters. After absorbing part of the incident light, the skin tissue will also return the remaining light to the body through diffuse reflection and be captured by the camera. The skin is divided into the epidermis, dermis, and subcutaneous tissue. The dermis is located in the second layer, and the subcutaneous tissue is located in the third layer. These two layers are the main locations for the distribution of capillaries and the main places for the skin to absorb and scatter light. The diffusely reflected light is absorbed by the blood vessels and capillaries, and the changes presented are consistent with the heart rate information. This slight change in light cannot be detected by the naked eye, but it can be reflected in the image.

[0034] Slight color changes in the skin area will change the pixel value on the image, and will also change accordingly with the increase or decrease of blood volume. By extracting the color change information, analyzing it, and removing noise, the real heart rate signal can be obtained.

[0035] Example 3 Based on Example 1 or 2, step 2 includes the following sub-steps: Step 2.1, determine the ROI area according to the key points of the palm, including the palm, the base of the palm, or the metacarpophalangeal joint; Step 2.2, take spatial average of the two-dimensional matrix corresponding to the single-channel palm vein of each frame of the infrared video, that is, calculate the arithmetic mean of all pixel values ​​in a matrix, and extract a set of discrete time signal sequences containing heart rate signals as the PPG signal at the moment of the frame of video image; Step 2.3, extract the heart rate signal from the PPG signal; during a heart rate cycle, blood flow will cause changes in pressure in the blood vessels. When the pressure in the blood vessels reaches its maximum, it means that the heart is contracting and pumping blood into the arteries. At this time, the blood's ability to absorb light increases, resulting in a decrease in the mean pixel value in the image, which is reflected in the PPG signal to form a trough; on the contrary, when the blood vessel pressure is the lowest, that is, when the heart is in diastole, the blood flow decreases, the ability to absorb light decreases, and the mean pixel value in the image increases, which is manifested as a peak in the PPG waveform. Through the analysis of the PPG signal, a variety of important physiological parameters can be extracted, including blood oxygen saturation, heart rate, blood pressure, cardiac output, etc., and it can also be used to monitor breathing; Step 2.4: Enhance the heart rate signal through filtering and noise reduction. The color change of the skin area is very small and is easily affected by light and movement. The original pulse wave signal has a lot of high-frequency noise and baseline drift, so the extracted heart rate signal needs to be filtered and denoised first, such as detrending and standardization, low-pass, high-pass, band-pass, band-stop, full-pass filtering, adaptive filters, blind source separation algorithms, independent component analysis (ICA) and other noise reduction algorithms.

[0036] In particular, before starting step 2, image preprocessing is performed. The light signal reflected by the palm changes very slightly, but through advanced image processing algorithms, the acquired image sequence can be analyzed and processed. Specifically, preprocessing operations such as denoising, enhancement and filtering are performed on the image to remove background noise and other interference factors and improve the image quality.

[0037] In particular, in the ROI region extraction of step 2.1, the region of interest of the palm is calculated based on the position of the key points of the palm. For example, a rectangular or elliptical region can be determined as the ROI based on the position of the joint points at the base of the fingers, with the center of the palm as the center. When calculating the ROI, the size and shape of the palm also need to be considered. The size of the palm can be estimated based on the distance between the key points or the contour information of the palm, and then the size and shape of the ROI can be adjusted based on the shape characteristics of the palm (such as aspect ratio, finger length ratio, etc.). If palm tracking is performed in a video, the ROI can be dynamically adjusted based on the movement and deformation of the palm. For example, when the palm moves or deforms, the position and size of the ROI are adjusted accordingly to ensure that the key area of ​​the palm is always included. After determining the ROI, it may be necessary to further optimize it to remove possible noise and interference. Image filtering, morphological processing and other methods can be used to remove noise and interference in the background to make the ROI more accurate and clear.

[0038] In particular, in step 2.2, a single-channel signal consisting of pixel mean values ​​in each region of interest (ROI) is generated frame by frame from the acquired video image frames.

[0039] In particular, in step 2.2, the spatial average can also be replaced by variance or standard deviation, median or extreme point detection. Among them, the variance or standard deviation is to calculate the variance or standard deviation of the pixel gray value in each channel to reflect the degree of change in the image area. This can provide dynamic information on the change of blood volume in the region. The median is to use the median instead of the mean, which can reduce the impact of noise, especially when there are abnormal values ​​in the signal. Extreme point detection is to find the maximum and minimum values ​​of the gray value of each channel to obtain the degree of fluctuation of blood volume.

[0040] Example 4 Based on Embodiment 1, 2 or 3, the video shot in step 1 also includes a color video, and correspondingly, in step 4, a recognition result of the fusion of palm print and palm vein is output.

[0041] In particular, if the acquisition device includes both an infrared camera and a color camera, the infrared camera obtains a single-channel palm vein under near-infrared light, and the color camera obtains an RGB palm print image under visible light. Turn on the infrared light, and simultaneously shoot the palm print and palm vein images under infrared light to obtain a color (RGB) video and an infrared video. By fusing the information of these two videos, a more stable heart rate signal can be obtained. The same system can capture palm prints and veins well aligned to provide more accurate information for fusion.

[0042] In particular, in step 2.2, the two-dimensional matrices corresponding to the single-channel palm vein of each frame of infrared video and the three color channels of R, G, and B of the color video are spatially averaged, and four groups of discrete time signal sequences containing heart rate signals in the time domain are extracted as the PPG signal at the moment of the frame of video image.

[0043] In particular, before step 1, a palm detection algorithm will be used to perform palm detection. Only when it is detected that the captured image contains a palm, step 1 is started, wherein the palm detection algorithm can be an edge detection algorithm, a template matching method, or a deep learning model. If the acquisition device includes both an infrared camera and a color camera, the color camera is used for palm detection in the palm detection stage under good light conditions such as daytime, and the color camera generally has better performance in detection; at night or in low-light environments such as darkness, an infrared light source is turned on for fill light, and the infrared camera is used for palm detection (in low-light or no-light conditions, the infrared camera generally has better performance in palm detection). The infrared camera and the color camera can also be used for palm detection at the same time. By fusing the information of the two cameras, the characteristics of the palm can be more comprehensively understood, and the accuracy of detection and environmental adaptability can be improved. However, using an infrared camera and a color camera at the same time requires additional hardware equipment, more computing resources and time, which will increase the cost and complexity of the system. If the acquisition device only includes an infrared camera, in daytime and other conditions with good lighting, you also need to turn on the infrared light source for fill light during the palm detection stage, and use the infrared camera for palm detection; at night or in low-light environments such as darkness, turn on the infrared light source for fill light, and use the infrared camera for palm detection.

[0044] Palm vein image acquisition can only be performed effectively when the image actually contains a palm. Without the palm detection link, the system may incorrectly collect and process without a palm, wasting computing resources and possibly producing erroneous results. Palm vein heart rate measurement relies on the detection of changes in blood flow in the palm. If there is no palm in the image, the changes in light reflection related to heart rate cannot be accurately measured. By detecting whether the image contains a palm, it can ensure that the system only calculates heart rate when there is a palm, improving the accuracy and reliability of heart rate detection.

[0045] In addition, when the user places the palm in the collection area, the palm detection function can quickly determine the position and status of the palm and provide timely feedback to the user. For example, if the system detects that the palm position is incorrect or incomplete, it can guide the user to adjust the palm position through the display module or sound prompts, so as to better perform palm vein collection and heart rate recognition. This can improve the convenience and success rate of user operation and enhance the user experience.

[0046] Traditional edge detection algorithms, such as the Canny edge detection algorithm, can be used to detect edge information in the image. Possible palm edges can be identified based on edge continuity, curvature and other features, and combined with prior knowledge such as the approximate shape and size of the palm to determine whether it is a palm edge. Alternatively, the template matching method can be used to pre-create palm template images of multiple people, multiple angles and postures, match the template with the image to be detected, and calculate the similarity between the template and each area in the image. The area with higher similarity may be the location of the palm. Deep learning methods can also be used, which specifically include the following steps: 1. Data preparation: a) Collect a large amount of image data containing palms and non-palms, mark key points, and mark the position and range of the palms in the images; b) The palm, the base of the palm, and the metacarpophalangeal joints are usually taken as key points. Among them, the palm is located in the center of the palm and is an important reference point of the hand. The position of the palm can help determine the spatial position and orientation of the entire palm. The base of the palm connects the wrist and the palm, which is important for determining the overall position and range of motion of the hand. For example, in hand tracking and motion analysis, the position change of the base of the palm can reflect the large movement of the hand. The metacarpophalangeal joint is where the fingers connect to the palm. The metacarpophalangeal joint is relatively stable, unlike the ends of the fingers or the edges of the palm, which are easily deformed or displaced by external factors. In actual applications, the palm may be deformed to a certain extent due to posture, pressure changes or slight movements, but the position of the metacarpophalangeal joint is relatively fixed and used as a reference to determine the center position and angle of the palm; c) Divide the data into training set, validation set and test set for model training, adjustment and evaluation.

[0047] 2. Model selection and training: a) Choose a deep learning model suitable for the target task, such as deep learning, such as common ResNet, VGG, YOLO series (YOLOv1, YOLOv2, YOLOv3, YOLOv4, YOLOv5), SSD (Single Shot MultiBox Detector) and other architectures; b) Use the training set to train the model and continuously adjust the model parameters through the back propagation algorithm to minimize the loss function so that the model can accurately recognize the palm; c) During the training process, data augmentation techniques such as rotation, flipping, and scaling can be used to increase data diversity and improve the generalization ability of the model.

[0048] 3. Model evaluation and optimization: a) Use the validation set to evaluate the trained model and check the model's accuracy, recall and other indicators; b) According to the evaluation results, adjust the model structure, parameters or training strategy for optimization. For example, you can adjust the number of network layers, convolution kernel size, learning rate and other parameters, or use different optimization algorithms.

[0049] 4. Model application: Apply the optimized model to the actual palm detection task. Input the image to be detected into the model, and the model will output whether the image contains a palm, the location information of the palm, and the key point information of the palm.

[0050] In particular, after the palm is detected, the palm tracking algorithm is started to track the movement of the palm in the captured video, which has the following important functions: 1) Stabilize the acquisition signal: Slight movement of the palm may cause changes in the acquired palm vein image and heart rate signal. By tracking the movement of the palm, the parameters of the acquisition device can be adjusted in real time to ensure that the correct position of the palm is always aligned, so as to stably acquire high-quality palm vein images and heart rate signals. For example, if the palm moves during the acquisition process, the tracking algorithm can adjust the camera's viewing angle so that it always focuses on the palm to avoid image blur or signal loss; 2) Compensate for motion interference: When the palm inevitably moves during the detection process, the tracking algorithm can compensate for the acquired signal. By analyzing the palm's motion trajectory and speed, the position of the palm at the next moment can be predicted, and the parameters of the heart rate calculation can be adjusted accordingly to eliminate the influence of motion on the heart rate measurement. For example, if the palm shakes rapidly during the detection process, the tracking algorithm can filter and correct the heart rate signal according to the amplitude and frequency of the movement to improve the accuracy of heart rate measurement; 3) Convenient user operation: The palm tracking algorithm can make palm heart rate detection more convenient and natural. The user does not need to deliberately keep the palm still, just place the palm near the collection area, and the tracking algorithm can automatically find the palm and detect it. This greatly improves the user experience and makes heart rate detection more convenient and comfortable.

[0051] The palm tracking algorithm includes the following steps: (1) Use the key point information of the palm in the first frame image detected by the palm detection algorithm as the feature point; (2) When entering the next frame image, search for the position most similar to the feature point in the first frame image in the new image area; (3) Estimate the movement of the palm based on the position change of the feature point between consecutive frames; (4) Continuously update the set of feature points over time; if some feature points cannot be successfully matched in several consecutive frames, or the motion trajectory is unstable, delete them from the feature point set and re-extract new feature points in the image; at the same time, continuously adjust the motion estimation based on the new matching results to achieve continuous tracking of the palm. This method greatly improves the running speed of the program while ensuring the tracking accuracy.

[0052] In particular, in step 3, if it is a live palm, the heart rate signal obtained from step 2 will show obvious periodicity; if it is a fake palm such as a photo, video, mobile phone image, 3D prosthetic hand, etc., the obtained signal will be chaotic. The video for liveness detection does not need to be shot for too long to obtain a complete heart rate signal, only obvious periodicity needs to be detected. For example, the heart rate of an adult at rest is usually between 60 and 100 beats per minute. This means that the heart completes a cycle of contraction and relaxation approximately every 0.6 seconds to 1 second, thereby generating a heart rate signal with a fixed period. In addition to periodicity, short-time energy spectrum, zero-crossing rate, frequency component, deep learning and other methods can also be used to comprehensively detect whether the signal obtained from step 2 is a heart rate signal, thereby improving the accuracy of liveness detection.

[0053] In particular, in the palm recognition of step 4, if the captured video is judged to be a live palm through liveness detection, the palm recognition is performed through the trained model deployed in the system. The model outputs the palm vein recognition result, such as the user's identity information, whether it is a legitimate user, etc. According to the recognition result, corresponding operations can be performed, such as authorized access, log recording, etc.

[0054] If the acquisition device captures both palm print and palm vein videos, the image quality detection is notified to select a frame of palm print and palm vein images with the best quality from the captured video, and the two images are fused for multimodal palm recognition to authenticate identity information. The two images can be fused through pixel-level fusion, feature-level fusion, or decision-level fusion. The specific method used is limited by factors such as hardware size, network model, training method, system required time, etc. In actual applications, it is necessary to select the appropriate fusion method and network model according to specific needs and hardware resources to achieve the best fusion effect.

[0055] If the acquisition device captures palm vein video, it notifies the image quality detection to select a frame with the best quality from the captured video for palm recognition and identity authentication.

[0056] Example 5 The embodiment of the present invention provides a palm vein heart rate liveness detection and palm recognition device, comprising: The acquisition module is responsible for capturing videos containing the palm, including an infrared light source, an infrared camera, and an image sensor. It is located at the front end or in the area where the user interacts. For example, in some palm vein recognition devices, the acquisition module may be installed on a specific window or platform where the user places the palm for vein acquisition. It may also be installed on the movable part of the device to better adapt to the palm position and angle of different users. The main control module, located inside the device, is the core control part of the entire palm vein recognition system. It may be installed on a circuit board or integrated into an embedded system. The location of the main control module usually takes into account factors such as heat dissipation and electromagnetic compatibility to ensure its stable operation. The main control module is connected to the acquisition module, receives the video captured by the acquisition module, and analyzes and processes it; the main control module may also be connected to other external devices, such as power adapters, storage devices, network devices, etc., and is responsible for managing the power supply, data storage, and communication functions of the entire system.

[0057] The human body detection module is located at the outer edge of the palm vein module or on the side close to the direction where the user may appear, and maintains a two-way connection with the acquisition module and the main control module; when the human body detection module detects that a human body is approaching, it will send a wake-up signal to the acquisition module, and the acquisition module will switch from a low-power state to a working state; the acquisition module will also feedback status information to the human body detection module to help the human body detection module adjust its own working mode; the human body detection module will send the detected human body approaching signal to the main control module, and the main control module will decide whether to start the workflow of the entire palm vein module; the main control module will also send configuration information and control instructions to the human body detection module.

[0058] In particular, the device also includes the method included in any one of embodiments 1 to 4.

[0059] In particular, the infrared light source in the acquisition module continuously emits near-infrared light (0.78-1.5um). When near-infrared light hits the surface of an object or penetrates an object, the spectrum is distorted, which can be extracted by the infrared camera. The light source emitted in the infrared region of the spectrum (700nm to 1000nm) (i) ensures user comfort because it is invisible to the human eye, and (ii) allows remote heart rate signal measurement.

[0060] The function of the image sensor is to capture the image information formed after the infrared light is reflected and scattered by the palm, and convert the light signal into an electrical signal.

[0061] The role of infrared cameras is to complete video recording in dark conditions. They are less sensitive to changes in the external environment and lighting, and their sensitivity to lighting changes is much lower than that of RGB cameras. Therefore, they are more suitable for heart rate measurement in conditions of drastic lighting changes or complete darkness.

[0062] In particular, the microcontroller (MCU) or processor, as the core of the main control module, is responsible for running and executing various algorithms, control logic and data processing tasks. Storage chip: used to store palm vein recognition algorithms, template data and other related system configuration information. It may include flash memory or random access memory (RAM), etc. Power management chip: ensures stable power supply for the main control module and the entire system, and regulates, converts and monitors the power supply. Communication interface chip: realizes data communication with other devices or modules, such as USB, Ethernet, Bluetooth and other interface chips. Image sensor interface circuit: if image acquisition is involved, the corresponding interface circuit is required to connect and control the image sensor to obtain palm vein image data. Clock circuit: provides clock signal to the main control module to ensure the synchronous operation of various components. Logic gate circuit: used to realize logical operation and control of digital signals.

[0063] In particular, infrared sensors, ultrasonic sensors, Doppler microwave sensors, radar sensors, etc. can be selected to detect human approach. When a human body is detected approaching the module, the entire module is awakened from low power consumption mode to start working. The functions of the human body detection module include: 1) Reducing power consumption: When no human body is approaching, the palm vein module can be in low power consumption mode to save energy. When the human body detection module detects that a human body is approaching, the entire module is awakened to perform operations such as palm vein collection and recognition. This can greatly reduce the power consumption of the module and extend the use time of the device. 2) Improving response speed: The human body detection module can quickly detect the approach of the human body and wake up the entire module in time, thereby improving the response speed of palm vein recognition. Users can perform palm vein collection and recognition without waiting for the module to be fully started, which improves the user experience. 3) Enhancing security: The human body detection module can enhance the security of the palm vein module to a certain extent. When the human body detection module detects that a human body is approaching, the palm vein collection and recognition function is turned on, avoiding the risk of malicious attacks or information theft when no one is approaching.

[0064] For example, in a palm vein recognition access control system, the human body detection module and the acquisition module are installed at the front end of the access control device, and the user places the palm in the acquisition area for vein acquisition. The main control module is installed inside the access control device, responsible for processing the collected palm vein data and controlling the opening and closing of the access control. If the display module is selected, it may be installed above or next to the access control device to display the user's identity information and access control status. When the user approaches the recognition device, it can quickly and accurately detect the approach of the human body, immediately send a signal, wake up the entire system, and switch it from low power mode to working mode. When the user places the palm in the acquisition area, the acquisition module collects palm vein data and transmits it to the main control module. After the main control module processes the data, if the recognition is successful, it will control the opening of the access control and transmit the user's identity information to the display module for display. If the recognition fails, the main control module will send a corresponding prompt message to the display module to let the user re-collect.

[0065] Example 6 Based on Example 5, the acquisition module further includes a narrow-band filter or a color camera.

[0066] Among them, the narrow-band filter can largely eliminate the interference caused by external light sources during measurement; the color camera is used to collect two-dimensional color images under ordinary light sources for analysis and comparison.

[0067] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A palm vein heart rate liveness detection and palm recognition method, characterized in that: The method comprises the following steps: Step 1, shooting a video containing a palm and tracking the movement of the palm; the video is an infrared video; Step 2, determining the ROI area of ​​the palm and extracting the heart rate signal, and enhancing the heart rate signal; Step 3: judging whether the palm is a live palm according to whether the heart rate signal shows periodicity; Step 4: Perform palm recognition on the living palm and output the palm vein recognition result.

2. The method for palm vein heart rate liveness detection and palm recognition as claimed in claim 1, characterized in that: When shooting the video in step 1, a stable light source is irradiated on a fixed area on the surface of the palm skin.

3. The method for palm vein heart rate liveness detection and palm recognition as claimed in claim 1, characterized in that: The step 2 includes the following sub-steps: Step 2.1, determining the ROI area according to the key points of the palm, wherein the key points include the palm, the base of the palm, or the metacarpophalangeal joint; Step 2.2, taking spatial average of the two-dimensional matrix corresponding to the single-channel palm vein of each frame of the infrared video, and extracting a set of discrete time signal sequences including the heart rate signal, namely, the PPG signal; Step 2.3, extracting the heart rate signal from the PPG signal; Step 2.4: Enhance the heart rate signal by filtering and noise reduction.

4. The method for palm vein heart rate liveness detection and palm recognition as claimed in claim 3, characterized in that: The video shot in step 1 also includes color video. Correspondingly, in step 4, the recognition result of the fusion of palm print and palm vein is output.

5. The method for palm vein heart rate liveness detection and palm recognition as claimed in claim 4, characterized in that: In the step 2.2, the two-dimensional matrices corresponding to the single-channel palm vein of each frame of the infrared video and the three color channels of R, G, and B of the color video are spatially averaged to extract four groups of discrete time signal sequences containing the heart rate signal, namely the PPG signal.

6. The method for palm vein heart rate liveness detection and palm recognition as claimed in claim 5, characterized in that: In step 2.2, the spatial average may be replaced by variance or standard deviation, median or extreme point detection.

7. The method for palm vein heart rate liveness detection and palm recognition as claimed in claim 1, characterized in that: Before step 1, a palm detection algorithm is used to perform palm detection. Step 1 is initiated only when it is detected that the captured image contains a palm. The palm detection algorithm may be an edge detection algorithm, a template matching method or a deep learning model.

8. The method for palm vein heart rate liveness detection and palm recognition as claimed in claim 7, characterized in that: In step 1, a palm tracking algorithm is used to track the movement of the palm; wherein the palm tracking algorithm includes the following steps: (1) using the key point information of the palm in the first frame image detected by the palm detection algorithm as feature points; (2) when entering the next frame image, searching for the position most similar to the feature point in the first frame image in the new image area; (3) estimating the movement of the palm based on the position change of the feature point between consecutive frames; (4) continuously updating the set of feature points as time goes by; if some feature points cannot be successfully matched in several consecutive frames, or the motion trajectory is unstable, they are deleted from the feature point set and new feature points are re-extracted from the image; at the same time, the motion estimation is continuously adjusted according to the new matching results to achieve continuous tracking of the palm.

9. A palm vein heart rate liveness detection and palm recognition device, characterized in that: The device comprises: A collection module, which includes an infrared light source, an infrared camera and an image sensor, is located at the front end or in the area interacting with the user, and can also be installed on a movable part of the device, and is responsible for capturing a video containing the palm; A main control module, which is located inside the device and may be installed on a circuit board or integrated in an embedded system, is connected to the acquisition module, receives the video captured by the acquisition module, and processes it; A human body detection module, which is located at the outer edge of the palm vein module or close to the side where the user may appear, and maintains a bidirectional connection with the acquisition module and the main control module; when the human body detection module detects that a human body is approaching, it will send a wake-up signal to the acquisition module, and the acquisition module will switch from a low-power state to a working state; the acquisition module will also feedback status information to the human body detection module to help the human body detection module adjust its own working mode; the human body detection module will send the detected human body approaching signal to the main control module, and the main control module will decide whether to start the workflow of the entire palm vein module; the main control module will also send configuration information and control instructions to the human body detection module.

10. The palm vein heart rate liveness detection and palm recognition device according to claim 9, characterized in that: The acquisition module also includes a narrowband filter or a color camera; the narrowband filter can eliminate interference caused by external light sources; the color camera is used to acquire color video.

Images