Ultrasonic and infrared combined palm print and palm vein recognition method, equipment and device

Abstract

The invention discloses an ultrasonic and infrared combined palm print and palm vein recognition method, equipment and device, and relates to the field of biological information recognition and identity authentication, and the method comprises the steps: 1, collecting original images, including a palm vein image and an ultrasonic palm print image; step 2, ROI extraction is carried out, a palm biological characteristic information area contained in the original image is identified, and two ROIs are obtained and are respectively the ROI of the palm vein image and the ROI of the ultrasonic palm print image; 3, the ROI of the palm vein image is input into a feature encoder 1, the ROI of the ultrasonic palm print image is input into a feature encoder 2, and palm vein features and palm print features are extracted respectively; 4, in the feature extraction process, bidirectional feature fusion is synchronously carried out, and finally multi-scale features are obtained; and 5, inputting the multi-scale features into a classifier for classification and identification, and outputting a classification result.

Description

Technical Field

[0001] The present invention relates to the field of biometric identification and identity authentication, and particularly to a palmprint and palm vein recognition method, device and apparatus combining ultrasound and infrared. Background Art

[0002] With the rapid development of information technology, people have more urgent needs for security and convenience, and biometric technology has gradually entered people's production and life and is applied to important fields such as financial payment, intelligent security, and information security. Palmprint recognition and palm vein recognition are both biometric technologies.

[0003] Palmprint recognition is to utilize the unique wrinkle, line, protrusion and other texture features on the surface of the human palm, collect palm images through a camera, and then use a special algorithm to extract and analyze palmprint features. Compare the collected palmprint features with the existing palmprint templates to identify personal identity. However, palmprint belongs to the epidermal features of the palm, and it is easy for lawbreakers to illegally obtain and forge.

[0004] Palm vein recognition is a biometric technology that uses the uniqueness of palm veins for identity recognition. Its main principle is to utilize the absorption characteristics of deoxyhemoglobin in vein blood vessels for near-infrared light and the difference in the absorption of near-infrared light by other physiological tissues (skin, fat), so that the palm irradiated by near-infrared light will show darker lines at the subcutaneous veins, thereby obtaining vein blood vessel information.

[0005] Combining palmprint and palm vein as a biometric technology has the advantages of high security, high accuracy, being unaffected by the environment, high-speed recognition, and biometric diversity, and is applicable to various identity verification fields requiring high security and high accuracy.

[0006] 1. High security: Combining the biometric features of palmprint and palm vein can improve the security of the recognition system. The palmprint and palm vein features are not easy to be imitated or forged, and are difficult to be stolen, increasing the ability of the recognition system to resist attacks and fraud.

[0007] 2. High accuracy: Combining palmprint and palm vein recognition technologies can improve the accuracy of recognition. The combination of the two biometric features can more comprehensively describe the individual body structure, reduce the false recognition rate, and improve the accuracy and reliability of recognition.

[0008] 3. Unaffected by the environment: The palmprint and palm vein biometric features are not affected by environmental factors during the recognition process. Whether in a wet, dry or strong or weak light environment, the palmprint and palm vein recognition technologies can stably perform recognition.

[0009] 4. Biometric Diversity: Palmprint and palm vein are different biometric features. When combined, they can increase the feature diversity of the biometric recognition system, thereby improving the robustness of the system. Even if some features are damaged or incomplete, recognition can still be completed through other biometric features.

[0010] Although the palmprint and palm vein recognition systems in the prior art have achieved high-precision recognition capabilities in using a single binocular camera sensor to collect palmprint and palm vein images, there is still a risk of being attacked by a high-precision 3D prosthetic hand. This type of attack may deceive the recognition system by using high-fidelity 3D printing technology or simulation technology to generate a prosthetic hand with palmprint and palm vein similar to those of a real palm.

[0011] Therefore, those skilled in the art are committed to developing a new palmprint and palm vein recognition method, device, and apparatus to solve the above problems existing in the prior art. Summary of the Invention

[0012] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to improve the accuracy of the palmprint and palm vein recognition method and reduce the risk of being attacked by a high-precision 3D prosthetic hand.

[0013] To achieve the above object, the present invention provides a palmprint and palm vein recognition method combining ultrasound and infrared, including the following steps: Step 1: Collect original images, including palm vein images and ultrasonic palmprint images; Step 2: Perform ROI extraction to identify the regions containing palm biometric feature information in the original images, obtaining two ROIs, namely the ROI of the palm vein image and the ROI of the ultrasonic palmprint image; Step 3: Input the ROI of the palm vein image into Feature Encoder 1 and the ROI of the ultrasonic palmprint image into Feature Encoder 2 to extract palm vein features and palmprint features respectively; Step 4: During the feature extraction process, perform two-way feature fusion synchronously. On the one hand, input the palm vein features and the palmprint features extracted from different layers into the feature fusion network for fusion to obtain fusion features. On the other hand, feedback the obtained fusion features to the feature extraction process to optimize the extraction of the palm vein features and the palmprint features, and finally obtain multi-scale features; Step 5: Input the multi-scale features into a classifier for classification recognition and output the classification result; Among them, the ROI extraction in Step 2 adopts a deep learning method; The deep learning method includes: 1) performing a convolution operation on the original image using the Canny edge detection operator to find the edge pixels in the original image, and then extracting the contour information of the palm according to the threshold segmentation algorithm; 2) using a localization network to obtain the normalized coordinates of the palm and detect the key landmarks of the palm, where the normalized coordinates are used to define the position and shape of the ROI; 3) transforming the original image using a spatial transformation network based on the key landmarks and extracting the ROI.

[0014] Further, the ROI extraction in step 2 can also adopt a traditional ROI extraction method; The traditional ROI extraction method includes: 1) performing a convolution operation on the original image using the Canny edge detection operator to find the edge pixels in the original image, and then extracting the contour information of the palm according to the threshold segmentation algorithm; 2) calculating the geometric shape parameters of the palm and locating the region of the palm biometric information; 3) using the seed filling method to collect all the points in the region of the palm biometric information and extracting the ROI.

[0015] Further, the feature extraction strategies adopted by the feature encoder 1 and the feature encoder 2 in step 3 are dilated convolution, depthwise separable convolution, residual network + feature pyramid network, or U-Net.

[0016] Further, the fusion methods adopted by the feature fusion network in step 4 include: element-wise addition or concatenation, adaptive pooling, or attention mechanism.

[0017] Further, the classifier in step 5 includes a convolutional layer, a normalization layer, a pooling layer, and a fully connected layer. Among them, the fully connected layer is the last layer. The multi-scale features are expanded into a vector through multiple fully connected operations, and then non-linearly transformed through an activation function to map the high-dimensional features to the dimension of the classification label, and finally the classification result is output.

[0018] The present invention also provides a palmprint and palm vein recognition device combining ultrasound and infrared, including the palmprint and palm vein recognition method combining ultrasound and infrared as described above, and further including a hardware part and a software algorithm part; The hardware part includes an optical module and an ultrasound module; the software algorithm part includes a palm detection and localization module, a feature extraction module, a bidirectional feature fusion module, and a backend classifier module; Among them, the optical module is responsible for obtaining palm vein images, and the ultrasonic module is responsible for obtaining ultrasonic palmprint images. The obtained palm vein images and ultrasonic palmprint images are sent to the software algorithm part for processing. After the palm detection and positioning module detects a palm, it enters the feature extraction module and the bidirectional feature fusion module for feature recognition, and finally obtains the final classification result through the backend classifier module.

[0019] Further, the optical module includes several optical modules, including a camera, an infrared lamp, an indicator light, and a CMOS; the ultrasonic module is composed of an ultrasonic array, and the ultrasonic array is an array structure composed of several ultrasonic transducers arranged.

[0020] Further, the arrangement of the optical module and the ultrasonic module is as follows: the optical module is located on one side of the device, and the ultrasonic module is located on the other side of the device.

[0021] Further, the arrangement of the optical module and the ultrasonic module is as follows: the optical module is located at the bottom of the device, and the ultrasonic module is located at the top of the device; or the ultrasonic array in the ultrasonic module surrounds the optical module.

[0022] The present invention also provides a palmprint and palm vein recognition device combining ultrasound and infrared, including the palmprint and palm vein recognition device combining ultrasound and infrared according to any one of the above.

[0023] The palmprint and palm vein recognition method, device and device provided by the present invention at least have the following technical effects: 1. In the technical solution provided by the present invention, ultrasonic technology is used, which has the advantages of being not affected by light, not affected by surface contamination, and non-contact, thereby avoiding the influence of illumination conditions on the clarity and contrast of images. Too strong or too weak illumination may lead to recognition failure; at the same time, ultrasonic technology can detect the blood flow in the palm, and judge whether the palm information is real through this information, improving the safety of live detection; 2. The ultrasonic palmprint recognition technology used in the technical solution provided by the present invention can obtain palmprint images with high resolution. Compared with traditional optical palmprint recognition technology, ultrasonic palmprint recognition has better anti-forgery performance and adaptability, and can obtain accurate palmprint images in humid, dry and other environments. The palmprint image information obtained by the ultrasonic module can be combined with the surface palmprint image information obtained by the optical module to achieve multi-modal palmprint recognition and improve the recognition accuracy; 3. The technical solution provided by the present invention aims at the multi-modal recognition task of palmprint and palm vein, and proposes a multi-modal recognition network, including a feature encoder, a bidirectional feature fusion, and a backend classifier, to achieve more accurate multi-modal recognition, improve the uniqueness and forgery resistance of biometric features. The use of the backend classifier can achieve efficient feature extraction and classification, reduce the number of model parameters and computational complexity, and improve the operating efficiency and response speed of the system.

[0024] The following will further illustrate the concept, specific structure and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features and effects of the present invention. Brief Description of the Drawings

[0025] Figure 1 is a flowchart of the method of a preferred embodiment of the present invention; Figure 2 is a schematic diagram of the device composition of a preferred embodiment of the present invention; Figure 3 is a schematic diagram of the arrangement of the optical module and the ultrasonic module of a preferred embodiment of the present invention. Detailed Embodiments

[0026] The following introduces multiple preferred embodiments of the present invention with reference to the accompanying drawings of the specification, making its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0027] The embodiments of the present invention utilize an ultrasonic module and an optical module to implement a multi-modal palmprint and palm vein recognition technology of multi-sensor fusion. Compared with the prior art and devices, through the use of the multi-sensor fusion technology combining ultrasonic and optical modules, the features of palmprint and palm vein can be obtained and recognized more comprehensively, improving the system's ability to distinguish real biometric features from prostheses and effectively preventing prosthesis attacks. By using the method of multi-modal fusion, the ultrasonic and optical feature information can be comprehensively utilized to improve the accuracy and precision of the palmprint and palm vein recognition system, reduce the false recognition rate, and enhance the recognition performance. The technology of multi-sensor fusion increases the complexity of recognition, further improves the security of the system, and makes the recognition result more reliable and secure. This device can be applied to fields such as intelligent door locks, financial payments, access control and attendance.

[0028] Embodiment 1 As Figure 1 shown, it is a schematic flowchart of a palmprint and palm vein recognition method combining ultrasonic and infrared. The method includes the following steps: Step 1: Collect original images, including palm vein images and ultrasonic palmprint images; Step 2: Perform ROI extraction to identify the regions in the original image that contain palm biometric information, obtaining two ROIs, namely the ROI of the palm vein image and the ROI of the ultrasonic palmprint image, where ROI is the region of interest image. Step 3: Input the ROI of the palm vein image into Feature Encoder 1 and the ROI of the ultrasonic palmprint image into Feature Encoder 2 to extract palm vein features and palmprint features respectively. Step 4: During the feature extraction process, perform bidirectional feature fusion synchronously. On one hand, input the palm vein features and palmprint features extracted from different layers into the feature fusion network for fusion to obtain fused features. On the other hand, feedback the obtained fused features to the feature extraction process to optimize the extraction of palm vein features and palmprint features. This feedback mechanism helps to optimize the performance of feature extraction, enabling the feature extraction process to be adjusted and optimized according to the feedback information of the fused features, thus better realizing the extraction of palmprint and palm vein features. Fuse the extracted palmprint features, palm vein features, and fused features to obtain multi-scale features, further fully integrating palmprint features and palm vein features to improve the feature expression ability and model performance. Step 5: Input the multi-scale features into the classifier for classification and recognition, and output the classification result.

[0029] Specifically, the palm vein image in Step 1 is obtained through an infrared lamp and a camera. The ultrasonic palmprint image is such that ultrasonic signals propagate along the palm surface and inside the skin in a certain way. Due to the different tissue structures of the skin and under the skin, the propagation speed of ultrasonic waves is also different in different tissues, resulting in the reflection and refraction of ultrasonic waves when passing through the palmprint and the skin. The reflected and refracted ultrasonic signals contain information about different tissue structures, such as palm print lines (ridge line and valley line information) and blood vessels under the skin, etc. The received ultrasonic signals are transmitted to the signal processing unit for signal processing and analysis. By processing features such as the amplitude and frequency of the ultrasonic signals, detailed information such as the texture of the palmprint image and even the blood vessel distribution under the skin can be extracted to form an ultrasonic palmprint image similar to an optical palmprint image.

[0030] Specifically, before performing ROI extraction in Step 2, preprocess the collected original images (ultrasonic palmprint images, palm vein images), mainly including operations such as image denoising, grayscale processing, and edge detection, to improve the clarity and accuracy of the images. ROI extraction refers to identifying and extracting the regions that contain palm biometric information from the obtained original images (ultrasonic palmprint images, palm vein images), and these information are the key information in the recognition module. The ROI extraction process helps to improve the accuracy and efficiency of the recognition system while reducing resource consumption. Usually, there are two ideas for ROI extraction, namely based on traditional methods and based on deep learning, and it can be flexibly selected according to actual needs.

[0031] In particular, the feature encoders 1 and 2 in step 3 can use different feature extraction strategies for feature extraction, such as DCNN (dilated convolution), MobileNets (depthwise separable convolution), ResNet (residual network) + FPN (feature pyramid network), or structures like U-Net. Additional skip connections can also be added in feature extraction networks at different depths to enhance the network's ability to capture image details, such as residual connections, dense connections, etc. According to the actual application requirements, the feature extraction network can flexibly adjust the pooling layer, batch normalization layer, and convolutional layer to achieve better results, such as changing the pooling or normalization strategy, using the RELU activation function and its variants (LeakyReLU, PReLU), adjusting the convolutional kernel size, and changing the convolutional kernel form (dilated convolution, deformable convolution).

[0032] In particular, in step 4, during the feature extraction process, the design of synchronous bidirectional feature fusion can effectively reduce the computational and memory overhead, and at the same time helps to better utilize the correlation between different features. Among them, the feature fusion network can adopt the structure of a deep neural network, including convolutional layers, pooling layers, fully connected layers, etc., to effectively integrate and fuse these features. The feature fusion network can be fused in several ways: element-wise addition or concatenation, adaptive pooling, attention mechanism, etc. Element-wise addition or concatenation: Upsample or convolve the high-level feature map to make its size the same as the low-level feature map, and then add or concatenate them element-wise to achieve the fusion of features at different scales. Adaptive Pooling: Perform adaptive pooling operations on feature maps at different scales, compress them to the same size, and then fuse the compressed feature maps. Adaptive pooling can retain important information in the feature map, helping to improve the representational ability of the fused features. Attention Mechanism: Introducing the attention mechanism can help the network dynamically learn the importance between features at different scales and perform feature fusion according to the importance, such as introducing attention modules like SENet, STN, or CBAM. By introducing the attention mechanism, features of different modalities, different channels, and different spatial positions in the input are adaptively fused. In this way, the network can pay more attention to important feature regions, thus better utilizing the correlation between palmprint information and palm vein information and improving the perception ability of the target.

[0033] Specifically, in step 5, the fused features are input into a classifier for the final recognition and classification process. The classifier includes convolutional layers, normalization layers, pooling layers, fully connected layers, etc., and structures such as residual connections or dense connections can be used internally. Appropriate structures and layer types can be selected according to specific tasks and resource limitations to build an efficient and accurate model. The last layer of the network is usually a fully connected layer (Fully Connected Layer). These layers are used to integrate and summarize the multi-scale features extracted by the network before, and finally output the classification result. In the fully connected layer, the network unfolds the multi-scale feature maps into a vector through multiple fully connected operations, and then performs a non-linear transformation through an activation function to map the high-dimensional features to the dimension of the classification labels, and finally outputs the classification result.

[0034] The ROI extraction in step 2 adopts a deep learning method. This method includes the following steps: 1) Use the Canny edge detection operator to perform a convolution operation on the original image to find the edge pixels in the original image, and then extract the contour information of the palm according to the threshold segmentation algorithm; 2) Use a localization network to obtain the normalized coordinates of the palm and detect the key landmarks of the palm, where the normalized coordinates are used to define the position and shape of the ROI; 3) According to the key landmarks, use a spatial transformation network to transform the original image and extract the ROI.

[0035] The deep learning method uses a localization network and a spatial transformation network to output the image ROI. Among them, the localization network is responsible for detecting the key landmarks in the hand image and transforming the image according to these landmarks to correct the elastic deformation and non-affine transformation of the hand image. The transformed ROI image is sent to the subsequent module for feature extraction and recognition.

[0036] Embodiment 2 Based on Embodiment 1, the deep learning method adopted for ROI extraction in step 2 specifically includes: 1) Contour extraction: Use the edge detection operator - Canny operator - to perform a convolution operation on the image to find the edge pixels in the image. The Canny operator is a multi-stage algorithm, including steps such as Gaussian filtering, gradient calculation, non-maximum suppression, and double-threshold processing, and finally obtains an edge image. From the edge image obtained by the edge detection operator, the contour of the palm is extracted according to a certain threshold segmentation algorithm, and some small linear objects or spots generated due to threshold segmentation errors are removed through some additional steps. Commonly used contour extraction algorithms include methods such as the findContours function and contour approximation; 2) Localization network: It includes a feature extraction network and a fully connected regression network, which are used to output the normalized coordinates of the palm. These coordinates are used to define the position and shape of the palmprint ROI. Among them, the hyperparameters of the feature extraction network can be adjusted according to actual needs, and the backbone network can be selected, such as ResNet50 or VGG-16, etc. The extracted features are connected to a fully connected regression network, and the hyperparameters of the network can also be flexibly adjusted. Usually, the hidden layer is followed by a ReLU activation function and a Dropout layer to prevent overfitting.

[0037] 3) Spatial transformation network: It includes a grid generator and a bilinear sampler, which can perform effective spatial image transformation operations, such as affine transformation, projective transformation or thin plate spline transformation, etc. Among them, the grid generator creates a deformed sampling network according to the normalized landmark coordinates output by the localization network. The bilinear sampler receives the deformed sampling network and the original image, and converts the original image into a regular grid hand ROI image through sampling.

[0038] Embodiment 3 Based on Embodiment 1, the ROI extraction in step 2 adopts a traditional ROI extraction method.

[0039] The traditional ROI extraction method includes: 1) Contour extraction: Use an edge detection operator - Canny operator - to perform convolution operations on the image to find the edge pixels in the image. The Canny operator is a multi-stage algorithm, including steps such as Gaussian filtering, gradient calculation, non-maximum suppression and double threshold processing, and finally an edge image is obtained. For the edge image obtained by the edge detection operator, the contour of the palm is extracted according to a certain threshold segmentation algorithm, and some small linear objects or spots generated due to threshold segmentation errors are removed through some additional steps. Commonly used contour extraction algorithms include methods such as the findContours function and contour approximation; 2) Locate the ROI: According to the extracted contour information, the geometric shape parameters of the palm can be calculated, such as area, perimeter, centroid coordinates, etc. Through these parameters, a virtual circular area is located, and this area covers most of the discriminative areas of the palm information. Through multiple scans, the size and shape of this area are adjusted, and the ROI is refined to make it more conform to the actual shape, that is, the area of the palm biometric information is located; 3) Extract the final ROI: Use the seed filling method to collect all the points in the area of the palm biometric information and extract the ROI.

[0040] Embodiment 4 As Figure 2As shown in the figure, an embodiment of the present invention provides a palmprint and palm vein recognition device combining ultrasound and infrared, which includes the palmprint and palm vein recognition method combining ultrasound and infrared in the previous Embodiment 1, 2 or 3, and also includes a hardware part and a software algorithm part; wherein, the hardware part includes an optical module and an ultrasound module; the software algorithm part includes a palm detection and positioning module, a feature extraction module, a bidirectional feature fusion module and a backend classifier module.

[0041] Among them, the optical module is responsible for obtaining palm vein images, and the ultrasound module is responsible for obtaining ultrasonic palmprint images. The obtained palm vein images and ultrasonic palmprint images are sent to the software algorithm part for processing. After the palm detection and positioning module detects a palm, it enters the feature extraction module and the bidirectional feature fusion module for feature recognition, and finally obtains the final classification result through the backend classifier module.

[0042] Specifically, it also includes a core circuit board which is a computing and communication module, including a computing chip, a communication chip, a power interface, transistors, resistors, capacitors, etc., and is responsible for performing functions such as processing signals, controlling circuits, storing data, and computing recognition. The communication between the hardware parts and the computing of the software algorithm together achieve palmprint and palm vein recognition.

[0043] Embodiment 5 On the basis of Embodiment 4, the optical module refers to a module that performs imaging, acquisition or sensing through optical technology, including several optical modules, mainly composed of a camera, an infrared lamp, an indicator light, a CMOS, etc., and works simultaneously with the ultrasound module to obtain palm vein images through the infrared lamp and the camera; the ultrasound module is composed of an ultrasound array. When an object is detected approaching, the entire module is awakened, ultrasonic waves are emitted and received, and ultrasonic palmprint images are obtained through signal processing algorithms.

[0044] Specifically, the camera is used to capture image or video data, and can be an ordinary color camera or an infrared camera, and a suitable type is selected according to specific application requirements. The infrared lamp is used to provide an infrared light source to irradiate the palm, so that the veins of the palm are visible. The indicator light is used to indicate the working state of the device and the position of the camera, such as shooting, connection or abnormal state, etc., to improve the user experience and facilitate user operation.

[0045] Specifically, CMOS (Complementary Metal-Oxide-Semiconductor) is an integrated circuit chip technology used for the image sensor of the optical module. The CMOS image sensor can convert the captured optical signal into an electrical signal, and realize the perception and acquisition of the optical signal through the photoelectric conversion function. It integrates a series of image processing functions, such as white balance, automatic exposure, denoising, etc., and can perform real-time processing while image acquisition to improve image quality and accuracy.

[0046] Through the collaborative work of these components, the optical module can achieve imaging, acquisition, and analysis of the target object or scene, providing support for applications such as identification, monitoring, and measurement. In palmprint and palm vein recognition technology, the optical module is mainly used to capture palm image data, extract palm vein information, and provide data support for subsequent feature extraction and recognition.

[0047] In particular, an ultrasonic array, that is, an ultrasonic transducer array, is an array structure composed of several ultrasonic transducers arranged. An ultrasonic transducer is a device that can convert electrical energy into acoustic energy or vice versa, and is used to transmit and receive ultrasonic signals. Each ultrasonic transducer generates or receives ultrasonic pulses. By controlling the working time, intensity, and phase of each transducer, a specific beam shape and direction can be formed to achieve different scanning methods and imaging modes, thereby realizing the positioning and imaging of the target area. Since multiple transducers work simultaneously, fast imaging and real-time monitoring can be achieved. In addition, by controlling the density and layout of the transducers in the array, the spatial resolution of imaging can be improved.

[0048] Specifically, the main functions and principles of the ultrasonic module are as follows: 1) Detect the approach of an object and wake up the entire module. Keep one transducer in the ultrasonic transducer array working normally, and the rest of the module components are in a dormant state until an object is detected approaching. Then, wake up the entire module including components such as the entire ultrasonic transducer array and the optical module. The principle is that when an object moves within a certain area, it will cause changes in air density and reflection conditions, which will lead to corresponding changes in the ultrasonic echo. Therefore, by analyzing the ultrasonic echo, the presence and movement of the object can be determined. 2) Detect the distance of the palm. The principle of ultrasonic ranging is that, since the speed of ultrasonic waves propagating in the air is fixed, the distance between the object and the sensor is measured by sending and receiving ultrasonic signals. The emitted ultrasonic signal propagates in the air. When the ultrasonic wave encounters the surface of an object, part of it will be reflected back. The sensor can calculate the time it takes for the ultrasonic signal to travel from the sensor to the object surface and back based on the time interval between sending and receiving the ultrasonic wave. According to the propagation speed of sound waves in the air (about 343 m / s), the distance between the object and the sensor can be calculated through the simple formula distance = speed * time. 3) Liveness detection. Ultrasonic technology can detect the blood flow in the palm. By using this information to judge whether the palm information is real, it improves the security of liveness detection. Since blood movement will cause slight changes in tissue structure, these changes will be reflected in the ultrasonic return signal. By processing and analyzing the received ultrasonic signal, the slight fluctuations caused by blood flow can be captured. By detecting the blood flow information in the palm, the system can verify whether the person being detected is a real live body. Because a false palm model usually does not have a normal blood flow pattern, and ultrasonic technology can detect these differences. 4) Collect palm images. After being awakened, the ultrasonic module emits ultrasonic signals, which will propagate along the palm surface and inside the skin in a certain way. Since the skin and the tissue structure under the skin are different, the propagation speed of ultrasonic waves in different tissues is also different, which causes the ultrasonic waves to reflect and refract when passing through the palmprint and the skin. At the same time, the ultrasonic module receives the reflected and refracted ultrasonic signals. The received ultrasonic signals contain information about different tissue structures, such as palmprint lines (ridge line and valley line information) and blood vessels under the skin. The received ultrasonic signals are transmitted to the signal processing unit for signal processing and analysis. By processing the characteristics such as the amplitude and frequency of the ultrasonic signal, the texture of the palmprint image and even the details such as the blood vessel distribution under the skin can be extracted to form an ultrasonic palmprint image similar to an optical palmprint image.

[0049] Embodiment 6 Based on Embodiment 4 or 5, the ultrasonic module and the optical module need to be installed at positions where data can be directly collected, and there must be no obstruction (it cannot block the camera or the ultrasonic array).

[0050] As Figure 3 shown below, the following are several possible arrangements: The optical module is located on one side of the device, and the ultrasonic module is located on the other side of the device; The optical module is located at the bottom of the device, and the ultrasonic module is located at the top of the device; The ultrasonic array in the ultrasonic module surrounds the optical module.

[0051] Embodiment 7 Based on Embodiment 4, 5 or 6, the palm detection and positioning module performs ROI extraction to identify the regions containing palm biometric information in the original image, obtaining two ROIs, namely the ROI of the palm vein image and the ROI of the ultrasonic palmprint image. ROI extraction refers to identifying and extracting the regions containing palm biometric information from the obtained original images (ultrasonic palmprint images, palm vein images), and these information are the key information in the recognition module. The ROI extraction process helps to improve the accuracy and efficiency of the recognition system while reducing resource consumption.

[0052] Specifically, the palm detection and positioning module includes a palm state detection module and an ROI positioning module. The palm state is detected when the user places the palm, and after verifying that the state is normal, the ROI images of the palmprint and palm vein are extracted through the ROI positioning module.

[0053] The palm state detection is performed by the images collected by multiple sensors (infrared, RGB camera or ultrasound) in the system to detect the position of the user's palm and ensure that the palm is correctly placed on the recognition device. If no palm is detected, the palm is incomplete, or the palm position is incorrect during the palm detection stage, the system can give corresponding prompt information (text, voice, image) to guide the user to re-collect. The prompt information includes asking the user to straighten the palm, adjust the palm to an appropriate distance, and ensure that the palm is moderately relaxed, etc., to improve the reliability and accuracy of the collected data quality. After the state is normal, the higher-quality biometric information collected enters the ROI positioning process.

[0054] Generally, there are two ideas for ROI extraction, namely based on traditional methods and based on deep learning, which can be flexibly selected according to actual needs. Traditional ROI extraction methods include: 1) Contour extraction: Use the edge detection operator - Canny operator - to perform convolution operations on the image to find the edge pixels in the image. The Canny operator is a multi-stage algorithm that includes steps such as Gaussian filtering, gradient calculation, non-maximum suppression, and double-threshold processing, and finally obtains an edge image. From the edge image obtained by the edge detection operator, the contour of the palm is extracted according to a certain threshold segmentation algorithm, and some additional steps are used to remove small linear objects or spots generated due to threshold segmentation errors. Commonly used contour extraction algorithms include methods such as the findContours function and contour approximation; 2) Locate the ROI: According to the extracted contour information, geometric shape parameters of the palm can be calculated, such as area, perimeter, centroid coordinates, etc. Based on these parameters, a virtual circular area is located, and this area covers most of the discriminative areas of the palm information. Through multiple scans, the size and shape of this area are adjusted, and the ROI is refined to make it more conform to the actual shape, that is, the area of the palm biometric information is located; 3) Extract the final ROI: Use the seed filling method to collect all the points in the area of the palm biometric information and extract the ROI.

[0055] Adopting a deep learning method for ROI extraction is to use a localization network and a spatial transformation network to output the image ROI. Among them, the localization network is responsible for detecting the key landmarks in the hand image and transforming the image according to these landmarks to correct the elastic deformation and non-affine transformation of the hand image. The transformed ROI image is sent to the subsequent module for feature extraction and recognition. In particular, the deep learning method includes: 1) Contour extraction: Use the edge detection operator - Canny operator - to perform convolution operations on the image to find the edge pixels in the image. The Canny operator is a multi-stage algorithm that includes steps such as Gaussian filtering, gradient calculation, non-maximum suppression, and double-threshold processing, and finally obtains an edge image. From the edge image obtained by the edge detection operator, the contour of the palm is extracted according to a certain threshold segmentation algorithm, and some additional steps are used to remove small linear objects or spots generated due to threshold segmentation errors. Commonly used contour extraction algorithms include methods such as the findContours function and contour approximation; 2) Localization network: It includes a feature extraction network and a fully connected regression network, which are used to output the normalized coordinates of the palm, and these coordinates are used to define the position and shape of the palmprint ROI. Among them, the hyperparameters of the feature extraction network can be adjusted and the backbone network can be selected according to actual needs, such as ResNet50 or VGG-16, etc. The extracted features are connected to a fully connected regression network, and the hyperparameters of the network can also be flexibly adjusted. Usually, the hidden layer is followed by a ReLU activation function and a Dropout layer to prevent overfitting.

[0056] 3) Spatial transformation network: It includes a grid generator and a bilinear sampler, and can perform effective spatial image transformation operations, such as affine transformation, projective transformation, or thin plate spline transformation, etc. Among them, the grid generator creates a deformed sampling network according to the normalized landmark coordinates output by the localization network. The bilinear sampler receives the deformed sampling network and the original image, and converts the original image into a hand ROI image with a regular grid through sampling.

[0057] The feature extraction module inputs the ROI of the palm vein image into Feature Encoder 1 and the ROI of the ultrasonic palmprint image into Feature Encoder 2, and extracts palm vein features and palmprint features respectively.

[0058] Specifically, Feature Encoder 1 and Feature Encoder 2 can use different feature extraction strategies for feature extraction, such as DCNN (dilated convolution), MobileNets (depthwise separable convolution), ResNet (residual network) + FPN (feature pyramid network), or structures like U-Net. Additional skip connections can also be added in feature extraction networks at different depths to enhance the network's ability to capture image details, such as residual connections and dense connections. According to actual application requirements, the feature extraction network can flexibly adjust the pooling layer, batch normalization layer, and convolution layer to achieve better results, such as changing the pooling or normalization strategy, using the RELU activation function and its variants (LeakyReLU, PReLU), adjusting the convolution kernel size, and changing the convolution kernel form (dilated convolution, deformable convolution).

[0059] The bidirectional feature fusion module synchronously performs feature fusion during the feature extraction process. On the one hand, it inputs the palm vein features and palmprint features extracted from different layers into the feature fusion network for fusion to obtain fused features. On the other hand, it feeds back the obtained fused features into the feature extraction process to optimize the extraction of palm vein features and palmprint features. This feedback mechanism helps to optimize the performance of feature extraction, enabling the feature extraction process to be adjusted and optimized according to the feedback information of the fused features, thereby better realizing the extraction of palmprint and palm vein features. The extracted palmprint features, palm vein features, and fused features are fused to obtain multi-scale features, further fully integrating the palmprint features and palm vein features, improving the feature expression ability and model performance. This design can effectively reduce the computational and memory overhead, and at the same time also helps to better utilize the correlation between different features.

[0060] In particular, the feature fusion network can adopt the structure of a deep neural network, including convolutional layers, pooling layers, fully connected layers, etc., for effectively integrating and fusing these features. The feature fusion network can be fused in several ways: element-wise addition or concatenation, adaptive pooling, attention mechanism, etc. Element-wise addition or concatenation: Upsample or perform a convolution operation on the high-level feature map to make its size the same as that of the low-level feature map, and then add or concatenate them element-wise to achieve the fusion of features at different scales. Adaptive Pooling: Perform an adaptive pooling operation on the feature maps at different scales, compress them to the same size, and then fuse the compressed feature maps. Adaptive pooling can retain important information in the feature map, helping to improve the representational ability of the fused features. Attention Mechanism: Introducing an attention mechanism can help the network dynamically learn the importance between features at different scales and perform feature fusion according to the importance, such as introducing attention modules like SENet, STN, or CBAM. By introducing an attention mechanism, features of different modalities, channels, and spatial positions in the input are adaptively fused. In this way, the network can pay more attention to important feature regions, thereby better utilizing the correlation between palmprint information and palm vein information and improving the perception ability of the target.

[0061] The fused features are input into the classifier in the backend classifier module for the final recognition and classification process.

[0062] In particular, the classifier includes convolutional layers, normalization layers, pooling layers, fully connected layers, etc., and structures such as residual connections or dense connections can be used internally. Appropriate structures and layer types can be selected according to specific tasks and resource limitations to build an efficient and accurate model. The last layer of the network is usually a fully connected layer (Fully Connected Layer). These layers are used to integrate and summarize the multi-scale features extracted by the network before, and finally output the classification result. In the fully connected layer, the network unfolds the multi-scale feature maps into a vector through multiple fully connected operations, and then performs a non-linear transformation through an activation function to map the high-dimensional features to the dimension of the classification label, and finally outputs the classification result.

[0063] Embodiment 8 The embodiment of the present invention also provides a palmprint and palm vein recognition device combining ultrasound and infrared, including the palmprint and palm vein recognition device combining ultrasound and infrared as described above.

[0064] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field according to the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should fall within the protection scope determined by the claims.

Claims

1. A palm print and palm vein recognition method combining ultrasound and infrared, characterized in that: The method comprises the following steps: Step 1, collecting original images, including palm vein images and ultrasonic palm print images; Step 2, extracting ROI, identifying the area of ​​palm biometric information contained in the original image, and obtaining two ROIs, namely, the ROI of the palm vein image and the ROI of the ultrasonic palm print image; Step 3, inputting the ROI of the palm vein image into the feature encoder 1, and inputting the ROI of the ultrasonic palm print image into the feature encoder 2, and extracting palm vein features and palm print features respectively; Step 4: During the feature extraction process, bidirectional feature fusion is performed synchronously. The palm vein features and palm print features extracted from different layers are input into the feature fusion network for fusion to obtain fusion features. The obtained fusion features are fed back into the feature extraction process to optimize the extraction of the palm vein features and the palm print features, and finally obtain multi-scale features. Step 5: Input the multi-scale features into a classifier for classification and identification, and output the classification results; Wherein, the ROI extraction in step 2 adopts a deep learning method; The deep learning method includes: 1) using the edge detection Canny operator to perform a convolution operation on the original image to find the edge pixels in the original image, and then extracting the contour information of the palm according to the threshold segmentation algorithm; 2) using a positioning network to obtain the normalized coordinates of the palm and detect the key landmarks of the palm, wherein the normalized coordinates are used to define the position and shape of the ROI; 3) based on the key landmarks, using a spatial transformation network to transform the original image and extract the ROI.

2. The method for palm print and palm vein recognition by combining ultrasound and infrared as claimed in claim 1, characterized in that: The ROI extraction in step 2 may also adopt a traditional ROI extraction method; The traditional ROI extraction method includes: 1) using the edge detection Canny operator to perform a convolution operation on the original image to find the edge pixels in the original image, and then extracting the contour information of the palm according to the threshold segmentation algorithm; 2) calculating the geometric shape parameters of the palm and locating the area of ​​the palm biometric information; 3) using the seed filling method to collect all points in the area of ​​the palm biometric information to extract the ROI.

3. The method for palm print and palm vein recognition by combining ultrasound and infrared as claimed in claim 1, characterized in that: The feature extraction strategies adopted by the feature encoder 1 and the feature encoder 2 in step 3 are dilated convolution, depthwise separable convolution, residual network + feature pyramid network, or U-Net.

4. The method for palm print and palm vein recognition by combining ultrasound and infrared as claimed in claim 1, characterized in that: The fusion method adopted by the feature fusion network in step 4 includes: element-by-element addition or splicing, adaptive pooling, or attention mechanism.

5. The method for palm print and palm vein recognition combining ultrasound and infrared as claimed in claim 1, characterized in that: The classifier in step 5 includes a convolution layer, a normalization layer, a pooling layer and a fully connected layer, wherein the fully connected layer is the last layer. The multi-scale features are expanded into a vector through multi-layer fully connected operations, and then nonlinear transformation is performed through an activation function to map the high-dimensional features to the dimension of the classification label, and finally the classification result is output.

6. An ultrasonic and infrared combined palm print and palm vein recognition device, characterized in that: The method for palm print and palm vein recognition by combining ultrasound and infrared as described in any one of claims 1 to 5 also includes a hardware part and a software algorithm part; The hardware part includes an optical module and an ultrasonic module; the software algorithm part includes a palm detection and positioning module, a feature extraction module, a bidirectional feature fusion module and a back-end classifier module; Among them, the optical module is responsible for obtaining the palm vein image, and the ultrasonic module is responsible for obtaining the ultrasonic palm print image. The obtained palm vein image and the ultrasonic palm print image are sent to the software algorithm part for processing. After the palm detection and positioning module detects the presence of a palm, it enters the feature extraction module and the bidirectional feature fusion module for feature recognition, and finally obtains the final classification result through the back-end classifier module.

7. The ultrasonic and infrared combined palm print and palm vein recognition device as claimed in claim 6, characterized in that: The optical module includes several optical modules, including a camera, an infrared lamp, an indicator light, and a CMOS; the ultrasonic module is composed of an ultrasonic array, and the ultrasonic array is an array structure composed of several ultrasonic transducers arranged in an array.

8. The ultrasonic and infrared combined palm print and palm vein recognition device as claimed in claim 7, characterized in that: The optical module and the ultrasonic module are arranged in such a manner that the optical module is located on one side of the device, and the ultrasonic module is located on the other side of the device.

9. The ultrasonic and infrared combined palm print and palm vein recognition device as claimed in claim 7, characterized in that: The optical module and the ultrasonic module are arranged in the following manner: the optical module is located at the bottom of the device, and the ultrasonic module is located at the top of the device; or the ultrasonic array in the ultrasonic module surrounds the optical module.

10. An ultrasonic and infrared combined palm print and palm vein recognition device, characterized in that: The invention comprises the ultrasonic and infrared combined palm print and palm vein recognition device as described in claim 6.

Images