Non-contact multi-modal fusion biological recognition system, method and device

Through the non-contact multimodal biometric system, the light is adjusted using binocular cameras and CMOS technology, combined with neural networks and feature pyramid networks, multimodal recognition of QR codes, palm lines and palm veins is realized, and ambient light is solved The problem of affecting the separation of equipment design and algorithms improves identification accuracy and security.

CN119007252APending Publication Date: 2024-11-22SHENZHEN METRO GROUP +2
View PDF 0 Cites 9 Cited by

Patent Information

Application Number
CN202411068021.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, QR code recognition is susceptible to ambient light and cannot realize multimodal biometric recognition of QR code, palm lines and palm veins. The equipment design is separated from algorithm design, affecting the recognition accuracy.

Method used

Using contactless multimodal biometric system, including induction module, optical imaging module and computing and communication module, the light is adjusted using binocular camera and CMOS technology, combined with lightweight neural network and feature pyramid network to achieve image quality evaluation and Feedback adjustment, integrating equipment design and algorithm design, and realizing multimodal recognition of QR codes, palm lines and palm veins.

Benefits of technology

It improves image quality and anti-light interference capability, realizes multimodal integrated biometrics of QR codes, palm lines and palm veins, reduces the rate of misidentification, and has low power consumption, high security and non-contact characteristics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007252A_ABST
    Figure CN119007252A_ABST
Patent Text Reader

Abstract

The invention discloses a non-contact multi-mode fusion biological recognition system, method and device, and relates to the field of palm print and palm vein recognition and two-dimensional code and biological information recognition, the system comprises a sensing module, an optical imaging module, a calculation and communication module and an interaction module; the sensing module can sense and detect the object distance in real time, and when the object distance is within a set range, the light source is automatically started to start to collect images; the optical imaging module is connected with the sensing module, adopts a binocular camera and a CMOS (Complementary Metal Oxide Semiconductor) technology, is responsible for guiding light to a photosensitive element, and can adjust an incident angle and a focal length of the light; the calculation and communication module is connected with the optical imaging module, comprises a front-end detection module, a near-infrared light module, an optical compensation module, a two-dimensional code recognition module and a palm print and palm vein multi-mode recognition module, and is responsible for performing classification detection and quality evaluation on received images, recognizing the images and generating a recognition result; and the interaction module receives the identification result and realizes interaction with the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of palm print and vein recognition, QR code and biometric information recognition, and in particular to a contactless multimodal fusion biometric recognition system, method and device. Background Art

[0002] QR code recognition is a common identification method that has been widely used in daily life, and can be used for mobile payments, personal identity verification, and other aspects. However, in actual use, QR code recognition is easily affected by factors such as lighting, angle, and blur, resulting in reduced recognition accuracy.

[0003] With the rapid development of information technology, people have a more urgent need for security and convenience. Biometric technology is gradually becoming part of our daily lives, with applications in key areas such as financial payments, intelligent security, and information security. Palmprint recognition and palm vein recognition are both biometric technologies. Palmprint recognition utilizes the unique texture features of the human palm, such as wrinkles, lines, and protrusions. Palmprint images are captured by a camera, and specialized algorithms are used to extract and analyze the palmprint features. These captured palmprint features are then compared with existing palmprint templates to identify individuals. However, palmprints are superficial features of the palm, making them easily obtained and forged by criminals.

[0004] Palm vein recognition is a biometric technology that uses the unique characteristics of palm veins for identity verification. It exploits the difference in near-infrared light absorption between deoxyhemoglobin in veins and other physiological tissues (skin, fat). This causes the subcutaneous veins to appear darker when illuminated by near-infrared light, thereby capturing venous information. Palm vein recognition requires a camera to capture palm vein images under near-infrared light. The palm vein image is then pre-processed, and vein features are extracted using a palm vein feature extraction algorithm. These features are then compared with a database of palm vein data to identify the individual. Palm vein recognition technology is highly resistant to interference and can accurately identify users in complex environments, such as at night, during heavy rain, in strong sunlight, in extreme cold, and in extreme heat. This technology not only addresses the insecurity and inconvenience of traditional authentication methods, which rely on media such as cards, keys, and mobile phone Bluetooth, but also addresses the poor performance of facial and fingerprint recognition technologies in complex environments. Palm vein recognition is a natural liveness recognition technology that is difficult to imitate and cannot be stolen. It avoids security issues caused by fake face attacks, fingerprint covers, and fake hands. In addition, palm vein recognition does not require contact with the device, is more hygienic, and has good user acceptance.

[0005] In their Chinese invention patent application, "A Multimodal Biometric Recognition Method for Integrating Palm Prints and Palm Veins" (Application No. CN202311099662.4), Fan Dandan et al. perform weighted fusion based on the adaptive matching scores of the palm print and palm vein, i.e., fusion at the matching score level. In their Chinese invention patent application, "A Method for Fusion of Palm Vein and Palm Print Features from a Single Near-Infrared Palm Image" (Application No. CN202311503122.8), Chen Jianchang employs different image enhancement algorithms to obtain palm vein and palm print images. He then employs a dual feature extraction network to extract features separately before further extracting fused features. However, the image enhancement methods employed are all traditional image processing methods, such as low-pass, blunting, and edge operator detection. Chen Junan and others proposed a new non-contact palm print and palm vein synchronous automatic collection device in their Chinese invention patent application "A non-contact palm print and palm vein synchronous automatic collection device and method" (application number CN202310268956.9), which solves the problem that non-contact palm print and palm vein collection devices cannot automatically and synchronously collect clear palm prints and palm veins. However, this solution focuses on hardware design and does not involve algorithm innovation.

[0006] The existing technical solutions still have the following defects: 1. Existing identification devices can often only realize single QR code recognition or palm print and palm vein recognition, and there is no multi-modal biometric recognition device that can realize QR code, palm print and palm vein recognition at the same time.

[0007] 2. A major issue with existing QR code recognition technology is that it is affected by ambient light (such as strong sunlight, sunlight, and the light source of the recognition device itself). This can affect the quality of the QR code image captured, thereby affecting recognition accuracy. Traditional QR code location methods typically use image processing operations to identify the three characteristic regions of the QR code to locate the code. These methods then use relevant QR code scanning algorithms for detection, placing high demands on the QR code resolution and viewing angle.

[0008] 3. Existing palm print and palm vein recognition technologies usually separate device design from algorithm design, and the palm print and palm vein images may come from different databases and different acquisition devices, which affects the recognition accuracy of palm print and palm vein.

[0009] Therefore, technicians in this field are committed to developing a new biometric identification method, equipment and device to deeply integrate QR code recognition, palm print and palm vein recognition to solve the above-mentioned defects in existing technical solutions, thereby improving the accuracy and security of identification. Summary of the Invention

[0010] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to avoid the influence of ambient light on image acquisition quality, realize multimodal biometric recognition, and integrate device design and algorithm design in palm print and palm vein recognition technology.

[0011] To achieve the above objectives, the present invention provides a contactless multimodal fusion biometric recognition system, comprising a sensing module, an optical imaging module, a computing and communication module, and an interaction module; The sensing module can sense and detect the distance of an object in real time. When the distance of the object is within a set range, the light source will be automatically activated to start collecting images. The optical imaging module is connected to the sensing module and uses a binocular camera and CMOS technology to guide light to the photosensitive element and adjust the incident angle and focal length of the light to ensure the clarity and quality of the image; The computing and communication module is connected to the optical imaging module and includes a front-end detection module, a near-infrared light module, an optical compensation module, a QR code recognition module, and a palm print and palm vein multimodal recognition module. The computing and communication module is responsible for classifying, detecting, and evaluating the quality of the received images to ensure that the image quality meets the standards, and for recognizing the images to generate recognition results. The interaction module is connected to the calculation and communication module, receives the recognition result and implements interaction with the user.

[0012] Furthermore, the front-end detection module includes a lightweight neural network classification network and an image quality assessment module; The lightweight neural network classification network is used to detect whether the image is a QR code or a palm; if the detection result is a QR code, the image quality assessment module is used to perform image quality assessment, and the optical imaging module is continuously feedback-adjusted until the image quality meets the standard, thereby obtaining a QR code image; if the detection result is a palm, two images are captured, one for a palm print image under visible light and the other for a palm vein image under infrared light, and the image quality assessment module is used to perform image quality assessment, and the optical imaging module is continuously feedback-adjusted until the image quality meets the standard, thereby obtaining a palm print image and a palm vein image.

[0013] Furthermore, the loss function selected by the lightweight neural network classification network is a binary cross entropy loss function, and the calculation formula is: in, Indicates the true label of the sample, which can be 0 or 1, 0 represents a QR code, and 1 represents a palm; represents the probability that the lightweight neural network classification network predicts that the sample is a positive example, and the value is between [0, 1]; when When , the sample is a positive example, the loss function is , that is, it is hoped that the probability that the lightweight neural network classification network predicts a positive example is as large as possible; when When , the sample is a negative example, the loss function is , that is, it is hoped that the probability that the lightweight neural network classification network predicts a negative example is as large as possible; The output layer of the lightweight neural network classification network uses The activation function converts the output into the probability distribution of each category and makes the sum of the output values ​​​​1. The category with the highest probability is directly taken as the final prediction result.

[0014] Furthermore, the image quality assessment module detects the illumination uniformity and contrast of the image, evaluates the brightness distribution, clarity and brightness of the image based on the illumination uniformity and contrast, and adjusts the illumination intensity of the optical compensation module or the near-infrared light module according to a preset threshold.

[0015] Furthermore, the two-dimensional code recognition module is connected to the front-end detection module to obtain the two-dimensional code image, and recognizes the two-dimensional code image through a two-dimensional code recognition network to obtain the recognition result; The backbone network of the QR code recognition network is a feature pyramid network, which includes several convolutional layers to extract features of the input QR code image at different scales. The network also includes the following components: multi-scale feature extraction, downsampling and upsampling, feature fusion, and a fully connected layer. A spatial transformation network is introduced between the two-dimensional code image and the feature pyramid network, wherein the spatial transformation network can adaptively perform spatial transformation and alignment on the two-dimensional code image, and includes: a local network, a grid generator, and a sampler; A CBAM is provided in the feature pyramid network to help the feature pyramid network dynamically learn the relevance and importance of features between high-level feature maps and low-level feature maps, and weight the features according to the importance, thereby achieving cross-scale feature fusion.

[0016] Furthermore, the palm print and palm vein multimodal recognition module is connected to the front-end detection module to obtain the palm print image and the palm vein image, and recognizes the palm print image and the palm vein image through the palm print and palm vein multimodal recognition network to obtain the recognition result; The palm print and palm vein multimodal recognition network includes a target detection-based ROI extraction module, a multi-scale feature encoder, a multi-scale feature fusion module, feature fusion and a back-end classifier; The target detection-based ROI extraction module obtains the palm print and palm vein regions of interest and determines the positions and shapes of the palm prints and palm veins; The multi-scale feature encoder is connected to the target detection-based ROI extraction module, and is a feature extraction module constructed based on depthwise separable convolution and inverse residual structure, including a multi-scale feature encoder 1 and a multi-scale feature encoder 2. The palm vein image is input into the multi-scale feature encoder 1 to extract palm vein features of different scales and levels, and the palm print image is input into the multi-scale feature encoder 2 to extract palm print features of different scales and levels. The multi-scale feature fusion module is connected to the multi-scale feature encoder to fuse the palm vein features of different scales and levels extracted by the multi-scale feature encoder 1, and also fuse the palm print features of different scales and levels extracted by the multi-scale feature encoder 2; The feature fusion is connected to the multi-scale feature fusion module to fuse the palm vein feature and the palm print feature; The back-end classifier is connected to the feature fusion, including a convolution layer, a normalization layer, a pooling layer, and a fully connected layer, and uses a residual connection or an Inception module to generate the recognition result.

[0017] The present invention also provides a non-contact multimodal fusion biometric recognition method, which automatically activates a light source to collect images by real-time sensing and detecting the distance to an object. A binocular camera and CMOS technology are used to guide light to a photosensitive element, and the incident angle and focal length of the light can be adjusted to ensure the clarity and quality of the image. The method includes the following steps: Step 1: Classify and detect the image to determine whether it is a QR code or a palm. If the detection result is a QR code, perform image quality assessment and continuously perform feedback adjustment until the image quality meets the standard, thereby obtaining a QR code image. If the detection result is a palm, capture the image twice: once for the palm print image under visible light and once for the palm vein image under infrared light. Image quality assessment is also performed, and continuous feedback adjustment is performed until the image quality meets the standard, thereby obtaining a palm print image and a palm vein image. Step 2: Recognize the QR code image through a QR code recognition network and obtain the recognition result; Step 3: Recognize the palm print image and the palm vein image through a palm print and palm vein multimodal recognition network and obtain the recognition result; Step 4: Receive the recognition results generated in steps 2 and 3, and implement interaction with the user.

[0018] Furthermore, in step 2, the backbone network of the QR code recognition network is a feature pyramid network, which includes several convolutional layers to extract features of different scales on the input QR code image, and also includes the following components: multi-scale feature extraction, downsampling and upsampling, feature fusion, and a fully connected layer; A spatial transformation network is introduced between the two-dimensional code image and the feature pyramid network, wherein the spatial transformation network can adaptively perform spatial transformation and alignment on the two-dimensional code image, and includes: a local network, a grid generator, and a sampler; A CBAM is provided in the feature pyramid network to help the feature pyramid network dynamically learn the relevance and importance of features between high-level feature maps and low-level feature maps, and weight the features according to the importance, thereby achieving cross-scale feature fusion.

[0019] Furthermore, in step 3, the palm print and palm vein multimodal recognition network includes a target detection-based ROI extraction module, a multi-scale feature encoder, a multi-scale feature fusion module, feature fusion, and a back-end classifier; The target detection-based ROI extraction module obtains the palm print and palm vein regions of interest and determines the positions and shapes of the palm prints and palm veins; The multi-scale feature encoder is connected to the target detection-based ROI extraction module, and is a feature extraction module constructed based on depthwise separable convolution and inverse residual structure, including a multi-scale feature encoder 1 and a multi-scale feature encoder 2. The palm vein image is input into the multi-scale feature encoder 1 to extract palm vein features of different scales and levels, and the palm print image is input into the multi-scale feature encoder 2 to extract palm print features of different scales and levels. The multi-scale feature fusion module is connected to the multi-scale feature encoder to fuse the palm vein features of different scales and levels extracted by the multi-scale feature encoder 1, and also fuse the palm print features of different scales and levels extracted by the multi-scale feature encoder 2; The feature fusion is connected to the multi-scale feature fusion module to fuse the palm vein feature and the palm print feature; The back-end classifier is connected to the feature fusion, including a convolution layer, a normalization layer, a pooling layer, and a fully connected layer, and uses a residual connection or an Inception module to generate the recognition result.

[0020] The present invention also provides a non-contact multimodal fusion biometric identification device, including the non-contact multimodal fusion biometric identification system as described above, or the non-contact multimodal fusion biometric identification method as described above.

[0021] The present invention provides a non-contact multimodal fusion biometric recognition system, method, and device that have at least the following technical effects: 1. The technical solution provided by the present invention adopts feedback-based automatic illumination adjustment and optimized recognition algorithm to improve image quality and resistance to light interference; 2. The technical solution provided by this invention realizes a multi-modal biometric authentication device that integrates QR codes, palm prints, and palm veins, with low power consumption, high security, and contactless features. The relevant chips and components are all domestically produced, unaffected by the international environment, reducing technical risks brought by the external environment; 3. The technical solution provided by the present invention combines the spatial transformer network (STN) and the feature pyramid network (FPN) to realize QR code recognition, improving recognition accuracy and adaptability; 4. The technical solution provided by this invention proposes a multimodal recognition network, including a multi-scale feature encoder, feature fusion, and a back-end classifier, to achieve accurate palm print and palm vein recognition. It has significant advantages in recognition rate, speed, and stability, with the error rate reduced to 1 in 10 million.

[0022] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 1 is a schematic diagram of a system structure of a preferred embodiment of the present invention; Figure 2 yes Figure 1 System relationship connection diagram of the illustrated embodiment; Figure 3 yes Figure 1 A schematic diagram of the workflow of the front-end detection module of the illustrated embodiment; Figure 4 yes Figure 1 A schematic diagram of the workflow of the QR code recognition network of the illustrated embodiment; Figure 5 yes Figure 1 A schematic diagram of the workflow of the palm print and palm vein multimodal recognition network of the illustrated embodiment. DETAILED DESCRIPTION

[0024] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0025] Existing technologies and equipment can often only achieve single QR code recognition or palm print and palm vein recognition, and do not have multi-modal biometric recognition devices that can simultaneously realize QR code, palm print and palm vein recognition. Existing QR code recognition technology is easily affected by ambient light (such as strong light, sunlight, and the light source of the recognition device itself), which affects the recognition accuracy. Traditional QR code positioning methods usually use related image processing operations to mark and find the three characteristic areas of the QR code to locate the QR code, and then use related QR code scanning algorithms to detect it, which has high requirements for QR code resolution and angle. Existing palm print and palm vein recognition technology usually separates device design from algorithm design, and the palm print and palm vein images may come from different databases and different acquisition devices, which affects the recognition accuracy of palm print and palm vein.

[0026] Compared with existing technologies and devices, the embodiments of the present invention provide the following technical solutions: 1. This embodiment of the present invention designs a multimodal biometric recognition device that simultaneously captures, registers, and recognizes QR codes, palm prints, and palm veins. This device uses the same device to simultaneously capture both palm print and palm vein images, reducing costs and improving the quality of the captured images. Furthermore, both biometrics can be captured from the same palm, allowing users to complete both capture and recognition in a single operation. This convenient and fast device features high integration, high recognition accuracy, low power consumption, high security, and a good user experience. It can be applied in areas such as rail transit, financial payments, and access control and attendance.

[0027] 2. This embodiment of the present invention designs an end-to-end palm print and vein recognition network, primarily comprising a pixel-level fusion module, a feature encoder, feature-level fusion, and a back-end classifier. This network performs image fusion at both the pixel and feature levels, sharing network parameters between feature encoders. By inputting palm print and vein information into the network, the network directly outputs the final recognition result, achieving high accuracy and robustness.

[0028] 3. The embodiment of the present invention designs a QR code illumination compensation module, which detects the ambient light intensity in real time during the QR code image acquisition process, adaptively compensates for light of different intensities according to the environment, automatically adjusts parameters such as exposure time and sensitivity, eliminates the impact of strong light or shadows on the QR code image, and accumulates illumination compensation experience through methods such as deep learning to improve the intelligence level of illumination compensation.

[0029] 4. The embodiments of the present invention use deep learning for QR code detection and recognition. Deep learning models can be trained with large amounts of data to learn more complex and abstract features, improving the accuracy of QR code detection and recognition. Deep learning models have strong generalization capabilities and can adapt to QR code detection and recognition in different scenarios and lighting conditions, providing greater applicability.

[0030] Example 1 The embodiment of the present invention provides a non-contact multi-modal fusion biometric recognition system, including a sensing module, an optical imaging module, a computing and communication module, and an interaction module. Figure 1 shown.

[0031] The sensing module is mainly based on TOF (optical time of flight) sensor, which can sense and detect the distance of objects in real time. When the distance of the object is within the set range (7cm-14cm), the light source and camera will be automatically started to start collecting images.

[0032] The optical imaging module, connected to the sensing module, utilizes a binocular camera and CMOS (Complementary Metal-Oxide-Semiconductor) technology. It directs light onto the photosensitive element and can adjust the incident angle and focal length to ensure image clarity and quality. Compared to traditional CCD (Charge-Coupled Device) technology, CMOS technology offers advantages such as low power consumption, low cost, and high integration, making it an increasingly popular choice for camera components. The optical imaging module also includes supporting optical components and lenses. These components, including lenses, filters, and reflectors, direct light onto the photosensitive element and adjust the incident angle and focal length to ensure image clarity and quality. Under normal light, it primarily captures palm print information, while under near-infrared light, it primarily captures palm vein information.

[0033] The computing and communication module is connected to the optical imaging module, including the front-end detection module, near-infrared light module, optical compensation module, QR code recognition module, and palm print and vein multimodal recognition module. It is responsible for classifying, detecting, and evaluating the quality of the received images to ensure that the image quality meets the standards, and identifying the images to generate recognition results.

[0034] From the perspective of hardware configuration, the computing and communication module includes a controller, computing module, storage module, communication module (such as WIFI / Bluetooth), etc., which are used for image storage, calculation, recognition and communication.

[0035] The controller is responsible for controlling the entire module's operation and coordinating communication and data transmission between its various components. The computing module processes image data, performs QR code recognition, and extracts and identifies palm print and vein features. The storage module stores QR code information, palm print and vein feature data, and other relevant data. The communication module communicates with external devices or cloud servers, transmitting data and receiving commands, enabling data exchange and networking.

[0036] Specifically, the near-infrared light module utilizes multiple near-infrared LEDs to form a near-infrared LED light strip. Taking into account the shape of the palm area, a circular LED array surrounding the camera is used to better meet actual system requirements. During the device hardware design, the light source and imaging components were rationally arranged to minimize the impact of uneven lighting on image quality.

[0037] Specifically, the optical compensation module consists of a circular array of LED strips, which can be evenly lit at intervals of 2, 3, 4, 6, or 12 depending on the ambient light intensity. This design can further reduce power consumption and computation time.

[0038] The interaction module connects to the computing and communication modules, receives recognition results, and enables interaction with the user. Specifically, it can be equipped with a touchscreen display interface to display the recognition result (successful / unsuccessful). Furthermore, users can use the touchscreen to perform operations such as inputting information, selecting functions, and viewing instructions. A voice interaction function can also be provided to guide the user through the operation process through voice prompts, such as "Please place your hand in the scanning area," "Your hand is too far away," and "Recognition successful," thereby improving user convenience.

[0039] The positions of the components of the contactless multimodal fusion biometric identification system provided by the embodiment of the present invention and the connection relationship between them are as follows: Figure 2 As shown: 1. The sensor module and optical imaging module are both located on the same camera circuit board and connected to the core circuit board via flexible cables. "Flex cables" or "flex ribbon cables" are used to connect the camera module (optical imaging module), touch screen display, and circuit board, respectively. Their flexible and thin design makes them ideal for connections within confined spaces and can adapt to bending and folding of the device, facilitating practical deployment.

[0040] 2. The touch screen interface displays the recognition result (successful / unsuccessful) and is also connected to the core circuit board through a flexible cable.

[0041] 3. The core circuit board is the computing and communication module, including computing chips, communication chips, power interfaces, transistors, resistors, capacitors, etc. It is responsible for performing functions such as signal processing, circuit control, data storage, computing and identification, and jointly realizes QR code recognition and palm print and vein recognition.

[0042] Example 2 Based on Example 1, the front-end detection module in the computing and communication module includes a lightweight neural network classification network and an image quality assessment module, such as Figure 3 shown.

[0043] A lightweight neural network classification network detects whether the image is a QR code or a palm. If the detection result is a QR code, the image quality assessment module is used to evaluate the image quality, and the optical imaging module is continuously adjusted for feedback until the image quality meets the requirements, resulting in a QR code image. If the detection result is a palm, two images are captured: one for the palm print under visible light and the other for the palm vein under infrared light. The image quality is also evaluated using the image quality assessment module, and the optical imaging module is continuously adjusted for feedback until the image quality meets the requirements, resulting in both palm print and palm vein images. This stage requires fast response and efficient completion of the image capture and detection tasks to ensure recognition accuracy and efficiency.

[0044] The lightweight neural network classification network boasts a concise and efficient structure, offering fast inference speed and a small model size, meeting the need for rapid response. It employs a series of optimized and streamlined design strategies, including but not limited to depthwise separable convolutions, lightweight network blocks, and fully connected layers. These design strategies significantly reduce the number of parameters and computational complexity while maintaining high recognition accuracy, ensuring efficient operation in resource-constrained environments.

[0045] The loss function selected by the lightweight neural network classification network is a binary cross entropy loss function (CrossEntropy Error Fuction), which is calculated as follows: in, Indicates the true label of the sample, which can be 0 or 1, 0 represents a QR code, and 1 represents a palm; Indicates the probability that the lightweight neural network classification network predicts that the sample is a positive example, and the value is between [0, 1]; when When the sample is a positive example, the loss function is , that is, we hope that the probability of the lightweight neural network classification network predicting a positive example is as large as possible; when When the sample is a negative example, the loss function is , that is, we hope that the probability that the lightweight neural network classification network predicts a negative example is as large as possible.

[0046] The cross entropy loss function encourages the model to output probabilities close to the true labels, thereby improving the model's prediction accuracy. During training, the average cross entropy loss on the entire training set is usually minimized to optimize the model parameters. The model output layer uses activation function, Convert the model output into the probability distribution of each category so that the sum of the output values ​​is 1, and directly take the category with the highest probability as the final prediction result. The combination can directly output the classification result without the need for additional threshold setting or decision boundary determination.

[0047] The image quality detection module checks the quality of captured images, including illumination uniformity and contrast. Based on this uniformity and contrast, it evaluates the image's brightness distribution, clarity, and brightness, and adjusts the illumination intensity of the optical compensation module or near-infrared light module. The illumination intensity can be increased or decreased to improve the image's illumination uniformity and contrast, thereby improving the image's brightness distribution, clarity, and brightness. If the image is generally dark or details are unclear based on the assessment of brightness distribution and clarity, increasing the illumination intensity can be considered to improve brightness and clarity. A threshold (empirical value) can be set based on the image's average brightness and clarity score. When the brightness or clarity falls below the threshold, the illumination intensity is increased.

[0048] Furthermore, the front-end detection module incorporates real-time feedback and a dynamic adjustment strategy. Based on real-time image quality metrics and recognition results, it dynamically adjusts network parameters and lighting intensity to achieve optimal image quality. This real-time feedback and dynamic adjustment strategy enables the front-end detection module to adapt to varying lighting conditions and environmental changes, improving system stability and robustness.

[0049] In particular, the training process of the lightweight neural network classification network is as follows: 1) Data preparation: Collect and annotate image datasets containing QR codes and palm images. 2) Network Design: Designing a lightweight neural network classification network structure, including selecting appropriate modules such as depthwise separable convolution and lightweight network blocks, and initially determining network parameters such as the number of layers and channels; 3) Model training: Using the prepared dataset and designed network structure, select an appropriate loss function, such as the cross-entropy loss function, and optimize the network parameters by performing a backpropagation algorithm on the training set to enable the network to accurately classify images. 4) Parameter tuning: Optimize network performance and avoid overfitting or underfitting by adjusting hyperparameters such as the number of network layers, number of network channels, learning rate, and regularization terms.

[0050] Example 3 On the basis of Examples 1 and 2, the QR code recognition module is connected to the front-end detection module to obtain a QR code image, and recognizes the QR code image through the QR code recognition network to obtain a recognition result.

[0051] Specifically, the QR code recognition module involves collecting a QR code dataset. This dataset includes QR codes in various conditions (reflective, backlit, in different colors, at different tilt angles, etc.), along with their corresponding annotations. During the data collection process, data augmentation techniques such as rotation, scaling, and noise addition can be used to expand the dataset and enhance the model's generalization capabilities.

[0052] The QR code recognition network in the embodiment of the present invention uses the feature pyramid network as the backbone network, and introduces the spatial transformer network (STN) and CBAM (Channel Attention and Spatial Attention Module) to improve the recognition network. Figure 4 shown.

[0053] The feature pyramid network can extract features at different scales, adapting to QR codes of varying sizes and densities, effectively improving the accuracy and robustness of QR code recognition. The feature pyramid structure typically consists of multiple convolutional layers, each of which extracts features at different scales from the input QR code image.

[0054] The feature pyramid structure includes the following key components: multi-scale feature extraction, downsampling and upsampling, feature fusion, and fully connected layers.

[0055] Multi-scale feature extraction: The feature pyramid structure extracts features at different scales through convolution layers, dilated convolution, or separable convolution. It can capture the characteristic information of target objects at different sizes and densities, and improve the model's ability to recognize targets of different scales.

[0056] Downsampling and upsampling: In the feature pyramid structure, downsampling and upsampling operations are often used to extract features at different scales. Downsampling can reduce the size of the feature map and improve computational efficiency, while upsampling can restore the size of the feature map and retain more detailed information.

[0057] Feature fusion: In a feature pyramid structure, features extracted at different scales are fused to obtain a more global and richer representation, improving the model's representational capabilities and robustness to the target. Features extracted at different scales can be fused using several methods: element-wise addition or concatenation, adaptive pooling, and attention mechanisms. Element-wise addition or concatenation: High-level feature maps are upsampled or convolved to the same size as lower-level feature maps, and then element-wise addition or concatenation is performed to fuse features at different scales. Adaptive pooling: Adaptive pooling is performed on feature maps at different scales, compressing them to the same size and then fusing the compressed feature maps. Adaptive pooling preserves important information in the feature maps, helping to improve the representational capabilities of the fused features. Attention mechanisms: Introducing attention mechanisms can help the network dynamically learn the importance of features at different scales and fuse features based on this importance, such as in the CBAM architecture. This attention mechanism allows the network to focus on important feature regions, improving object perception.

[0058] Fully Connected Layer: The final layer of a feature pyramid network is typically a fully connected layer. These layers integrate and aggregate the multi-scale features previously extracted by the network, ultimately outputting the classification result. In a fully connected layer, the network uses multiple layers of fully connected operations to expand the multi-scale feature maps into a single vector. This is then transformed nonlinearly using an activation function, ultimately outputting the classification result. Fully connected layers are typically used to map high-dimensional features to the dimensions of the class label to achieve classification.

[0059] A spatial transformer network is a network module that can learn to perform geometric transformations on images. In a QR code recognition network, the spatial transformer network can be used to rotate, scale, or translate the input image to better align the position and orientation of the QR code, thereby improving recognition accuracy. The spatial transformer network does not require key point calibration and can adaptively transform and align the QR code (including translation, scaling, rotation, and other geometric transformations). This reduces the impact of geometric transformations such as rotation, translation, scaling, and distortion on tasks such as classification and positioning, making the network more robust and generalizable, thereby improving the network's performance in tasks such as object recognition and target detection.

[0060] The spatial transformation network primarily consists of a localization network, a grid generator, and a sampler. The localization network is used to regress the transformation parameters θ. Its input is a feature image, which then passes through a series of hidden network layers (fully connected or convolutional networks, followed by a regression layer) to output the spatial transformation parameters. θ can take various forms, such as implementing 2D affine transformations, transmission transformations (Proj), and thin plate spline transformations (TPS). The size of 𝜃 depends on the type of transformation. The grid generator constructs a sampling grid based on the predicted transformation parameters. This is the output obtained by sampling and transforming a set of points in the input image. The sampler uses both the sampling grid and the input feature map as input to produce an output, resulting in the transformed feature map.

[0061] CBAM is set up in the feature pyramid network to help the feature pyramid network dynamically learn the relevance and importance of features between high-level feature maps and low-level feature maps, and weight features according to importance, thereby achieving cross-scale feature fusion. CBAM combines channel attention and spatial attention to improve the model's ability to represent image features. Channel attention is used to adjust the importance between different channels, while spatial attention is used to adjust the importance of different spatial positions. By introducing CBAM, the model's attention to important features in the image can be enhanced, thereby improving the performance of classification and detection tasks. In an embodiment of the present invention, CBAM can be used for feature fusion between high-level features and low-level features (optional), helping the network to dynamically learn the relevance and importance of features between high-level feature maps and low-level features, and weight features according to importance, thereby achieving cross-scale feature fusion and improving the network's perception of complex scenes.

[0062] The training steps of the QR code recognition network are as follows: (1) Divide the dataset into training and test sets, usually using cross-validation or holdout method; (2) Select a suitable loss function, such as the cross entropy loss function, use the training set to train the deep learning model, and continuously adjust the model parameters through the back propagation algorithm to make the model gradually converge; (3) Verify the performance of the model through the validation set, and adjust the number of network layers, number of network channels, and hyperparameters (such as learning rate, batch size, etc.) to optimize the model performance.

[0063] The use phase of the QR code recognition network involves deploying the trained model into a module for actual QR code recognition tasks. To continuously optimize the network, QR code data can be collected during use and used for online model learning and optimization.

[0064] Example 4 Based on Examples 1 to 3, the palm print and palm vein multimodal recognition module is connected to the front-end detection module to obtain palm print images and palm vein images, and the palm print and palm vein multimodal recognition network is used to recognize the palm print and palm vein images and obtain recognition results.

[0065] Specifically, the palm print and palm vein multimodal recognition module includes the collection of a palm print and palm vein database. Palm print and palm vein data are collected to establish a comprehensive database containing a large amount of real palm print and palm vein image data for system training, validation, and testing. When collecting palm print and palm vein data, it is necessary to ensure data diversity and representativeness, including data from different populations of different ages, genders, and ethnicities. Furthermore, factors such as varying lighting conditions, angles, and hand postures must be considered to ensure accurate recognition in all situations. To protect user privacy, data collection strictly complies with relevant laws and regulations, and necessary security measures are implemented to ensure data security and privacy. Furthermore, user consent must be obtained to clarify the purpose and scope of data use.

[0066] The palm print and palm vein multimodal recognition network includes a ROI extraction module based on target detection, a multi-scale feature encoder, a multi-scale feature fusion module, feature fusion and a back-end classifier, such as Figure 5 shown.

[0067] For the target detection-based ROI extraction module, obtaining the region of interest (ROI) of the palm print and palm vein is a key step. By obtaining the ROI, the location and shape of the palm vein can be determined, and invalid information and interference factors can be removed, thereby improving recognition accuracy and efficiency.

[0068] Palm vein ROI extraction methods can be based on different techniques and algorithms. The following are some common methods: (1) Key point-based determination: By extracting key points, such as feature points and edge points, from the palm vein image, the position and shape of the ROI are determined based on these key points. Common key point extraction algorithms include SIFT (Scale Invariant Feature Transform) and SURF (Speeded Robust Features).

[0069] (2) Corner detection: Corners are important feature points in an image and can be used to determine the position and shape of a ROI. Common corner detection algorithms include Harris corner detection and Shi-Tomasi corner detection.

[0070] (3) Machine learning method: By training machine learning models, such as support vector machines (SVM) and convolutional neural networks (CNN), the palm vein ROI is identified and intercepted. The machine learning method can learn the characteristics of the palm vein based on the existing labeled data, thereby accurately determining the ROI.

[0071] In an embodiment of the present invention, a ROI extraction module based on SSD target detection is trained. The SSD (Single Shot MultiBox Detector) model, by using multi-scale feature maps and an anchor frame mechanism, can simultaneously perform bounding box regression and classification of the target in a single forward propagation. It has high detection speed and accuracy, and can automatically detect and extract the palm position. Compared with traditional methods, the palm ROI extraction method based on SSD target detection has the advantages of automation, efficiency, and robustness. First, the model learns the feature representation of the palm vein image, including texture, shape and other information, so that it can accurately identify the location of the palm vein. Second, through the design of the anchor frame and the bounding box regression technology, the model can accurately locate the position of the palm and extract the accurate ROI area.

[0072] In particular, some key features and components of the SSD model are: (1) Multi-scale feature maps: The SSD model uses multiple feature maps of different scales to detect objects of different sizes. These feature maps come from convolutional layers of different depths and can capture the features of objects at different scales.

[0073] (2) Anchor Boxes: The SSD model generates multiple predefined anchor boxes at each feature map unit position for detecting objects of different shapes and scales. By matching with these anchor boxes, the position and size of the object can be determined.

[0074] (3) Bounding Box Regression: The SSD model predicts the offset of each anchor box relative to the true target bounding box through regression, thereby accurately locating the target position.

[0075] (4) Classification Head: The SSD model classifies the target at each anchor box position and predicts the target category probability. This allows for both target classification and detection.

[0076] (5) Loss function: The SSD model uses a multi-task loss function, including confidence loss, classification loss, and bounding box regression loss, to simultaneously optimize the accuracy of classification and positioning.

[0077] The multi-scale feature encoder is connected to the ROI extraction module based on target detection. It is a feature extraction module built based on depthwise separable convolution and inverted residual block. Depthwise separable convolution divides the convolution operation into two steps: depthwise convolution and pointwise convolution. It improves computational efficiency by reducing the number of parameters, thereby reducing the complexity of the model.

[0078] The multi-scale feature encoder extracts features from the input feature map using multiple convolution kernels of different scales, thereby capturing information at different scales. This encoder can effectively improve the model's ability to understand the input data, thereby enhancing the model's performance and generalization capabilities.

[0079] Specifically, the multi-scale feature encoder includes multi-scale feature encoder 1 and multi-scale feature encoder 2. The palm vein image is input into multi-scale feature encoder 1 to extract palm vein features at different scales and levels. The palm print image is input into multi-scale feature encoder 2 to extract palm print features at different scales and levels, including line and texture information. Multi-scale feature encoder 1 and multi-scale feature encoder 2 have the same structure.

[0080] The multi-scale feature fusion module connects to the multi-scale feature encoder and fuses the detailed features of large-scale low-level feature maps with the semantic information of small-scale high-level feature maps, ultimately generating low-level feature maps with richer semantic representations. Specifically, it fuses the palm vein features of different scales and levels extracted by multi-scale feature encoder 1, and also fuses the palm print features of different scales and levels extracted by multi-scale feature encoder 2.

[0081] The specific process is as follows: First, a 1×1 convolution is applied to each feature map to reduce the channel dimension, making the channel dimension consistent across all feature maps. Next, the size of Bottleneck1's feature layer is set to the base size. For feature maps smaller than the base size, upsampling is used to resize them to the base size, making all feature maps have the same scale in the spatial dimension. These feature layers are then fused using concatenation, and batch normalization is added to prevent overfitting. Finally, a 1×1 convolution is applied to reduce the number of channels to generate the final feature map.

[0082] Feature fusion connects the multi-scale feature fusion module to fuse palm vein features and palm print features to improve recognition performance.

[0083] There are three main ways to fuse palm print and palm vein images: pixel level, feature level and decision (score) level.

[0084] Pixel-level image fusion performs fusion processing directly at the pixel level. The advantage is that it retains the complete information of the original image and has a better fusion effect. The disadvantage is that it requires high-quality original image data and has a high processing overhead.

[0085] Feature-level image fusion first preprocesses the image and extracts features, and then fuses them at the feature level. The advantage is that the amount of calculation is small and the equipment requirements are relatively low. The disadvantage is that some original information will be lost during the feature extraction process, and the fusion effect may be slightly worse than the pixel level.

[0086] Decision-level image fusion: Based on feature extraction and classification, fusion is performed according to specific decision rules. Advantages include low computational effort, low equipment requirements, and high reliability. Disadvantages include significant information loss, and the fusion effect may not be as good as the previous two methods.

[0087] The present invention uses feature-level image fusion based on the attention mechanism. When designing an attention mechanism module to fuse the features extracted by two encoders, the following steps can be followed: (1) Calculate similarity: First, calculate the similarity between the features extracted by the two encoders. You can use a similarity calculation method such as cosine similarity or dot product calculation. (2) Calculate attention weight: Based on the value calculated by similarity, the softmax function can be used to convert the similarity into attention weight. The attention weight represents the importance of the features extracted by the two encoders when fusion.

[0088] (3) Feature fusion: Finally, the attention weights are used to fuse the features extracted by the two encoders. Features can be fused by weighted averaging. Through feature-level image fusion based on the attention mechanism, the palm print and palm vein multimodal recognition network can dynamically fuse the features extracted by the two encoders and adjust the weight of feature fusion according to the similarity between them, thereby better utilizing the correlation between palm print information and palm vein information.

[0089] The backend classifier connects feature fusion, including convolutional layers, normalization layers, pooling layers, and fully connected layers. Residual connections or Inception modules can also be used. The appropriate structure and layer type can be selected based on the specific task and resource constraints to build an efficient and accurate model.

[0090] Training the palm print and vein multimodal recognition network: The network is trained and optimized using a collected palm print and vein dataset to improve recognition accuracy and robustness. Supervised learning is employed, with network parameters continuously adjusted via a backpropagation algorithm. Adaptive optimizers such as Adam and RMSprop are used to fine-tune the learning rate and accelerate convergence.

[0091] Pruning optimization of palm print and palm vein multimodal recognition network: Pruning is a technique for optimizing neural network models, which can reduce the size of the model and improve the inference speed by removing unnecessary connections or nodes.

[0092] The main process of pruning optimization is as follows: (1) Train an original palm print and palm vein multimodal recognition network model with good performance. The original model has a large number of network parameters and a slow inference speed; (2) Determine the importance of parameters in the original model; (3) Remove less important parameters. Reasonable setting of these parameters can control the extent of pruning and avoid excessive pruning that leads to performance degradation; (4) Fine-tune on the training set to avoid performance degradation due to changes in network structure; (5) Determine whether the model size, inference speed, effect, etc. meet the requirements. If not, continue pruning.

[0093] Repeat the above steps until the required model performance and inference speed are achieved.

[0094] Example 5 The present invention also provides a non-contact, multimodal fusion biometric identification method. Based on any of the non-contact, multimodal fusion biometric identification systems provided in Examples 1 to 4, the method first automatically activates a light source to capture an image by real-time sensing and detecting the distance to an object. A binocular camera and CMOS technology are used to direct light onto a photosensitive element, and the incident angle and focal length of the light can be adjusted to ensure image clarity and quality. The method includes the following steps: Step 1: Classify and detect the image to see if it is a QR code or a palm. If the detection result is a QR code, perform image quality assessment and make continuous feedback adjustments until the image quality meets the standard, thereby obtaining a QR code image. If the detection result is a palm, capture two images: one for the palm print image under visible light and the other for the palm vein image under infrared light. Image quality assessment is also performed, and continuous feedback adjustments are made until the image quality meets the standard, thereby obtaining a palm print image and a palm vein image. Step 2: Recognize the QR code image through the QR code recognition network and obtain the recognition result; Step 3: Recognize the palm print image and palm vein image through the palm print and palm vein multimodal recognition network and obtain the recognition result; Step 4: Receive the recognition results generated in steps 2 and 3, and implement interaction with the user.

[0095] In particular, in step 2, the backbone network of the QR code recognition network is a feature pyramid network, which includes several convolutional layers to extract features of different scales from the input QR code image. It also includes the following components: multi-scale feature extraction, downsampling and upsampling, feature fusion, and fully connected layers. A spatial transformation network is introduced between the QR code image and the feature pyramid network. The spatial transformation network can adaptively transform and align the QR code image, including: a local network, a grid generator, and a sampler. CBAM is set up in the feature pyramid network to help the feature pyramid network dynamically learn the correlation and importance of features between high-level feature maps and low-level feature maps, and weight the features according to the importance, thereby achieving cross-scale feature fusion.

[0096] In particular, in step 3, the palm print and palm vein multimodal recognition network includes a target detection-based ROI extraction module, a multi-scale feature encoder, a multi-scale feature fusion module, feature fusion, and a back-end classifier; The ROI extraction module based on target detection obtains the region of interest of the palm print and palm vein, and determines the position and shape of the palm print and palm vein; The multi-scale feature encoder is connected to the ROI extraction module based on target detection. It is a feature extraction module built based on depthwise separable convolution and inverse residual structure. It includes multi-scale feature encoder 1 and multi-scale feature encoder 2. The palm vein image is input into the multi-scale feature encoder 1 to extract palm vein features of different scales and levels. The palm print image is input into the multi-scale feature encoder 2 to extract palm print features of different scales and levels. The multi-scale feature fusion module is connected to the multi-scale feature encoder to fuse the palm vein features of different scales and levels extracted by the multi-scale feature encoder 1, and also fuse the palm print features of different scales and levels extracted by the multi-scale feature encoder 2; Feature fusion connects the multi-scale feature fusion module to fuse palm vein features and palm print features; The back-end classifier connects feature fusion, including convolutional layers, normalization layers, pooling layers, and fully connected layers, and uses residual connections or Inception modules to generate recognition results.

[0097] Example 6 An embodiment of the present invention provides a non-contact multi-modal fusion biometric recognition device, including any one of the non-contact multi-modal fusion biometric recognition systems provided in Embodiments 1 to 4.

[0098] Example 7 An embodiment of the present invention provides a non-contact multi-modal fusion biometric identification device, including the non-contact multi-modal fusion biometric identification method provided in Example 5.

[0099] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible without inventive effort by those skilled in the art. Therefore, any technical solution that can be derived by one skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A contactless multi-modal fusion biometric recognition system. It is characterized in that The system includes a sensing module, an optical imaging module, a computing and communication module, and an interaction module; The sensing module can sense and detect the distance of an object in real time. When the distance of the object is within a set range, the light source will be automatically activated to start collecting images. The optical imaging module is connected to the sensing module, and uses a binocular camera and CMOS technology to guide light to the photosensitive element, and can adjust the incident angle and focal length of the light to ensure the clarity and quality of the image; The computing and communication module is connected to the optical imaging module, including a front-end detection module, a near-infrared light module, an optical compensation module, a two-dimensional code recognition module, and a palm print and palm vein multimodal recognition module, and is responsible for classifying, detecting and evaluating the quality of the received image to ensure that the image quality meets the standard, and identifying the image to generate a recognition result; The interaction module is connected to the calculation and communication module, receives the recognition result and implements interaction with the user.

2. The contactless multi-modal fusion biometric recognition system as claimed in claim 1, It is characterized in that The front-end detection module includes a lightweight neural network classification network and an image quality assessment module; The lightweight neural network classification network is used to detect whether the image is a QR code or a palm; if the detection result is a QR code, the image quality assessment module is used to perform image quality assessment, and the optical imaging module is continuously feedback-adjusted until the image quality meets the standard to obtain a QR code image; if the detection result is a palm, two images are taken, one for a palm print image under visible light and the other for a palm vein image under infrared light, and the image quality assessment module is used to perform image quality assessment, and the optical imaging module is continuously feedback-adjusted until the image quality meets the standard to obtain a palm print image and a palm vein image.

3. The contactless multi-modal fusion biometric recognition system as claimed in claim 2, It is characterized in that The loss function selected by the lightweight neural network classification network is a binary cross entropy loss function, and the calculation formula is: in, Indicates the true label of the sample, which can be 0 or 1, 0 represents a QR code, and 1 represents a palm; represents the probability that the lightweight neural network classification network predicts that the sample is a positive example, and the value is between [0, 1]; when When , the sample is a positive example, the loss function is , that is, it is hoped that the probability that the lightweight neural network classification network predicts a positive example is as large as possible; when When , the sample is a negative example, the loss function is , that is, it is hoped that the probability that the lightweight neural network classification network predicts a negative example is as large as possible; The output layer of the lightweight neural network classification network uses The activation function converts the output into the probability distribution of each category and makes the sum of the output values ​​​​1. The category with the highest probability is directly taken as the final prediction result.

4. The contactless multi-modal fusion biometric recognition system as claimed in claim 2, It is characterized in that The image quality assessment module detects the illumination uniformity and contrast of the image, assesses the brightness distribution, clarity and brightness of the image based on the illumination uniformity and contrast, and adjusts the illumination intensity of the optical compensation module or the near-infrared light module based on a preset threshold.

5. The contactless multi-modal fusion biometric recognition system as claimed in claim 1, It is characterized in that The two-dimensional code recognition module is connected to the front-end detection module to obtain the two-dimensional code image, and recognizes the two-dimensional code image through the two-dimensional code recognition network to obtain the recognition result; The backbone network of the two-dimensional code recognition network is a feature pyramid network, which includes several convolutional layers, performs feature extraction of different scales on the input two-dimensional code image, and also includes the following components: multi-scale feature extraction, downsampling and upsampling, feature fusion, and fully connected layers; A spatial transformation network is introduced between the two-dimensional code image and the feature pyramid network, and the spatial transformation network can adaptively perform spatial transformation and alignment on the two-dimensional code image, including: a local network, a grid generator and a sampler; A CBAM is provided in the feature pyramid network to help the feature pyramid network to dynamically learn the relevance and importance of features between high-level feature maps and low-level feature maps, and to weight the features according to the importance, thereby achieving cross-scale feature fusion.

6. The contactless multi-modal fusion biometric recognition system as claimed in claim 1, It is characterized in that The palm print and palm vein multimodal recognition module is connected to the front-end detection module to obtain the palm print image and the palm vein image, and recognizes the palm print image and the palm vein image through a palm print and palm vein multimodal recognition network to obtain the recognition result; The palm print and palm vein multimodal recognition network includes a target detection-based ROI extraction module, a multi-scale feature encoder, a multi-scale feature fusion module, feature fusion and a back-end classifier; The target detection-based ROI extraction module obtains the palm print and palm vein regions of interest, and determines the positions and shapes of the palm prints and palm veins; The multi-scale feature encoder is connected to the ROI extraction module based on target detection, and is a feature extraction module constructed based on deep separable convolution and inverse residual structure, including a multi-scale feature encoder 1 and a multi-scale feature encoder 2. The palm vein image is input into the multi-scale feature encoder 1 to extract palm vein features of different scales and levels, and the palm print image is input into the multi-scale feature encoder 2 to extract palm print features of different scales and levels. The multi-scale feature fusion module is connected to the multi-scale feature encoder, and fuses the palm vein features of different scales and levels extracted by the multi-scale feature encoder 1, and also fuses the palm print features of different scales and levels extracted by the multi-scale feature encoder 2; The feature fusion is connected to the multi-scale feature fusion module to fuse the palm vein feature and the palm print feature; The back-end classifier is connected to the feature fusion, including a convolution layer, a normalization layer, a pooling layer, and a fully connected layer, and uses a residual connection or an Inception module to generate the recognition result.

7. A non-contact multi-modal fusion biometric recognition method that automatically starts the light source to collect images by real-time sensing and detecting the distance of the object. It uses a binocular camera and CMOS technology to guide the light to the photosensitive element and can adjust the incident angle and focal length of the light to ensure the clarity and quality of the image. It is characterized in that The method comprises the following steps: Step 1: classify and detect the image to determine whether it is a QR code or a palm; if the detection result is a QR code, then perform image quality assessment, and perform continuous feedback adjustment until the image quality meets the standard to obtain a QR code image; if the detection result is a palm, then take two images, one for a palm print image under visible light, and another for a palm vein image under infrared light, and also perform image quality assessment, and perform continuous feedback adjustment until the image quality meets the standard to obtain a palm print image and a palm vein image; Step 2: Recognize the two-dimensional code image through a two-dimensional code recognition network and obtain the recognition result; Step 3, recognizing the palm print image and the palm vein image through a palm print and palm vein multimodal recognition network and obtaining the recognition result; Step 4: Receive the recognition results generated in step 2 and step 3, and implement interaction with the user.

8. The non-contact multi-modal fusion biometric identification method as claimed in claim 7, It is characterized in that In step 2, the backbone network of the two-dimensional code recognition network is a feature pyramid network, which includes several convolutional layers, performs feature extraction of different scales on the input two-dimensional code image, and also includes the following components: multi-scale feature extraction, downsampling and upsampling, feature fusion, and a fully connected layer; A spatial transformation network is introduced between the two-dimensional code image and the feature pyramid network, and the spatial transformation network can adaptively perform spatial transformation and alignment on the two-dimensional code image, including: a local network, a grid generator and a sampler; A CBAM is provided in the feature pyramid network to help the feature pyramid network to dynamically learn the relevance and importance of features between high-level feature maps and low-level feature maps, and to weight the features according to the importance, thereby achieving cross-scale feature fusion.

9. The non-contact multi-modal fusion biometric identification method as claimed in claim 7, It is characterized in that In step 3, the palm print and palm vein multimodal recognition network includes a ROI extraction module based on target detection, a multi-scale feature encoder, a multi-scale feature fusion module, feature fusion and a back-end classifier; The target detection-based ROI extraction module obtains the palm print and palm vein regions of interest, and determines the positions and shapes of the palm prints and palm veins; The multi-scale feature encoder is connected to the ROI extraction module based on target detection, and is a feature extraction module constructed based on deep separable convolution and inverse residual structure, including a multi-scale feature encoder 1 and a multi-scale feature encoder 2. The palm vein image is input into the multi-scale feature encoder 1 to extract palm vein features of different scales and levels, and the palm print image is input into the multi-scale feature encoder 2 to extract palm print features of different scales and levels. The multi-scale feature fusion module is connected to the multi-scale feature encoder, and fuses the palm vein features of different scales and levels extracted by the multi-scale feature encoder 1, and also fuses the palm print features of different scales and levels extracted by the multi-scale feature encoder 2; The feature fusion is connected to the multi-scale feature fusion module to fuse the palm vein feature and the palm print feature; The back-end classifier is connected to the feature fusion, including a convolution layer, a normalization layer, a pooling layer, and a fully connected layer, and uses a residual connection or an Inception module to generate the recognition result.

10. A non-contact multi-modal fusion biometric recognition device, It is characterized in that It includes the non-contact multi-modal fusion biometric recognition system as described in any one of claims 1 to 6, or the non-contact multi-modal fusion biometric recognition method as described in any one of claims 7 to 9.

Citation Information

Cited By

  • Ultrasonic and infrared combined palm print and palm vein recognition method, equipment and device

    CN120220196A

  • An ultrasonic, infrared combined palm print and palm vein recognition method, device and apparatus

    CN120220196B

  • Biological recognition method and system based on monocular depth guided multi-modal fusion

    CN120299100A

  • Light supplement control method and device for multi-mode module, storage medium and computer equipment

    CN120877339A

  • Multi-modal biological recognition method based on artificial intelligence

    CN120997915A