Palm print and palm vein fusion method, device and electronic device

Abstract

The present application discloses a palm print palm vein fusion method, device and electronic device, belonging to the field of biometric identification technology. The method comprises: inputting a palm print RGB image and a palm vein IR image into a palm print palm vein fusion model, fusing the palm print RGB image and the palm vein IR image, and obtaining a palm print palm vein fusion feature map; the palm print palm vein fusion model is trained based on palm print RGB image samples and palm vein IR image samples. The present application fuses the palm print RGB image and the palm vein IR image through a palm print palm vein fusion model to generate a palm print palm vein fusion feature map, and performs biometric identification based on the palm print palm vein fusion feature map, thereby improving the accuracy and stability of biometric identification.

Description

Technical Field

[0001] The present application belongs to the field of biometric identification technology, and in particular relates to a palm print and palm vein fusion method, device and electronic equipment. Background Art

[0002] Palm print and palm vein information are of great value in biometrics. Palm prints are captured through RGB images to show the surface texture of the palm, while palm veins are captured through infrared (IR) images. Both are unique and stable biometric features suitable for identity authentication and security identification. Traditional single-modality methods have limitations. RGB images are easily affected by lighting and angles, while IR images have few details and are greatly affected by temperature. Fusion recognition methods in related technologies usually assume that RGB and IR images are geometrically aligned, but in reality they are only weakly aligned due to asynchrony of sensors, fields of view, etc. The diversity of RGB and IR features in color and texture also brings challenges to fusion. It is necessary to propose a palm print and palm vein fusion method that can solve the above problems. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a palm print and palm vein fusion method, device and electronic device, which fuses the palm print RGB image and the palm vein IR image through a palm print and palm vein fusion model to generate a palm print and palm vein fusion feature map, and performs biometric identification based on the palm print and palm vein fusion feature map, thereby improving the accuracy and stability of biometric identification.

[0004] In a first aspect, the present application provides a palm print and palm vein fusion method, the method comprising:

[0005] Inputting the palmprint RGB image and the palm vein IR image into the palmprint-palm vein fusion model, fusing the palmprint RGB image and the palm vein IR image, and obtaining a palmprint-palm vein fusion feature map;

[0006] The palm print and palm vein fusion model is trained based on palm print RGB image samples and palm vein IR image samples.

[0007] In the above technical solution, the palm print RGB image and the palm vein IR image are fused through the palm print and palm vein fusion model to generate a palm print and palm vein fusion feature map containing both palm print and palm vein features. Compared with a single palm print or palm vein image, biometric identification based on the palm print and palm vein fusion feature map can improve the accuracy of biometric identification. The palm print and palm vein fusion feature map can also be used for identification under external interference conditions, thereby improving the stability and robustness of biometric identification.

[0008] According to an embodiment of the present application, the step of inputting the palm print RGB image and the palm vein IR image into a palm print and palm vein fusion model, fusing the palm print RGB image and the palm vein IR image, and obtaining a palm print and palm vein fusion feature map includes:

[0009] Obtain palmprint RGB image and palm vein IR image;

[0010] Extracting features from the palmprint RGB image and the palm vein IR image respectively to obtain palmprint features and palm vein features;

[0011] Extracting features of the palm print feature and the palm vein feature respectively to obtain specific palm print features and specific palm vein features;

[0012] Extracting common features from the palm print features and the palm vein features to obtain common palm print features and common palm vein features;

[0013] Performing offset prediction on the common palm print feature and the common palm vein feature to determine a transformation matrix;

[0014] Performing feature alignment on the specific palmprint feature and the specific palm vein feature according to the transformation matrix;

[0015] The specific palm print feature and the specific palm vein feature after feature alignment are fused to obtain a palm print and palm vein fusion feature map.

[0016] In the above technical scheme, by acquiring a palmprint RGB image and a palm vein IR image, and extracting palmprint features and palm vein features respectively, extracting specific palmprint features and specific palm vein features from the palmprint features and palm vein features respectively, extracting the palmprint features and palm vein features together to obtain common palmprint features and common palm vein features, using the common palmprint features and common palm vein features to perform offset prediction to determine a transformation matrix, aligning the specific palmprint features and the specific palm vein features according to the transformation matrix, and fusion to obtain a palmprint-palm vein fusion feature map, achieving spatial consistency and effective combination of palmprints and palm veins, performing biometric identification based on the palmprint-palm vein fusion feature map, the accuracy of biometric identification can be improved, the palmprint-palm vein fusion feature map can also be used for identification under external interference conditions, and the stability and robustness of biometric identification are improved.

[0017] According to one embodiment of the present application, the palm print and palm vein fusion model includes: an RGB image encoder, an IR image encoder, a specific encoder, a common encoder, an offset prediction module, a feature alignment module and a feature fusion module.

[0018] The RGB image encoder is used to extract features from the palmprint RGB image to determine palmprint features;

[0019] The IR image encoder is used to extract features from the palm vein IR image to determine palm vein features;

[0020] The specific encoder is used to extract features from the palm print features to determine specific palm print features, and / or to extract features from the palm vein features to determine specific palm vein features;

[0021] The public encoder is used to extract the public features of the palm print features and the palm vein features, and determine the public palm print features and the public palm vein features;

[0022] The offset prediction module is used to perform offset prediction on the common palm print feature and the common palm vein feature to determine a transformation matrix;

[0023] The feature alignment module is used to perform feature alignment on the specific palm print feature and the specific palm vein feature according to the transformation matrix;

[0024] The feature fusion module is used to fuse the specific palm print feature and the specific palm vein feature after feature alignment to obtain a palm print and palm vein fusion feature map.

[0025] In the above technical scheme, the palm print and palm vein fusion model includes an RGB image encoder, an IR image encoder, a specific encoder, a public encoder, an offset prediction module, a feature alignment module and a feature fusion module. The RGB image encoder is used to extract features from the palm print RGB image and the palm vein IR image, and then determine the specific palm print features and the specific palm vein features through the specific encoder. The public encoder is used to determine the public palm print features and the public palm vein features. Through the offset prediction module, the specific palm print features and the specific palm vein features are offset predicted to determine the transformation matrix, and through the transformation matrix, the feature alignment of the specific palm print features and the specific palm vein features is achieved through the feature alignment module. The feature fusion module fuses the aligned feature palm print features and the specific palm vein features to obtain a palm print and palm vein fusion feature map, which realizes the spatial consistency and effective combination of palm prints and palm veins. Biometrics are identified based on the palm print and palm vein fusion feature map, which can improve the accuracy of biometrics. The palm print and palm vein fusion feature map can also be used for identification under external interference conditions, which improves the stability and robustness of biometrics.

[0026] According to an embodiment of the present application, the offset prediction module is used to determine a first transformation matrix based on the common palm print feature and the common palm vein feature, wherein the first transformation matrix is ​​used to represent a spatial transformation relationship between the specific palm print feature and the specific palm vein feature;

[0027] The feature alignment module is used to transform the specific palmprint feature according to the first transformation matrix to obtain an aligned palmprint feature, wherein the aligned palmprint feature and the specific palm vein feature have the same spatial position;

[0028] The feature fusion module is used to fuse the aligned palm print features and the specific palm vein features to obtain the palm print and palm vein fusion feature map.

[0029] In the above technical solution, the offset prediction module is used to determine the first transformation matrix according to the common palm print features and the common palm vein features, the first transformation matrix describes the spatial transformation relationship between the specific palm print features and the specific palm vein features, the feature alignment module is used to transform the specific palm print features according to the first transformation matrix to obtain the aligned palm print features, that is, the specific palm print features after the alignment processing, the feature fusion module is used to fuse the aligned palm print features and the specific palm vein features to generate a palm print palm vein fusion feature map, the palm print palm vein fusion feature map realizes the effective combination of palm prints and palm veins, and provides a high-precision and strong robust feature representation for biometrics.

[0030] According to an embodiment of the present application, the offset prediction module is used to determine a second transformation matrix based on the common palm print feature and the common palm vein feature, wherein the second transformation matrix is ​​used to represent the spatial transformation relationship between the specific palm vein feature and the specific palm print feature;

[0031] The feature alignment module is used to transform the specific palm vein feature according to the second transformation matrix to obtain an aligned palm vein feature, wherein the aligned palm vein feature and the specific palm print feature have the same spatial position;

[0032] The feature fusion module is further used to fuse the aligned palm vein feature and the specific palm print feature to obtain the palm print and palm vein fusion feature map.

[0033] In the above technical solution, the offset prediction module is used to determine the second transformation matrix according to the common palm print features and the common palm vein features, the second transformation matrix describes the spatial transformation relationship between the specific palm vein features and the specific palm print features, the feature alignment module is used to transform the specific palm vein features according to the second transformation matrix to obtain the aligned palm vein features, that is, the specific palm vein features after alignment processing, the feature fusion module is used to fuse the aligned palm vein features with the palm print features to generate a palm print and palm vein fusion feature map, the palm print and palm vein fusion feature map realizes the effective combination of palm prints and palm veins, and provides a high-precision and strong robust feature representation for biometrics.

[0034] According to one embodiment of the present application, the palm print and palm vein fusion model is trained by the following steps:

[0035] Obtain palmprint RGB image samples and palm vein IR image samples;

[0036] By means of the RGB image encoder, feature extraction is performed on the palmprint RGB image sample to determine a palmprint feature sample, and by means of the IR image encoder, feature extraction is performed on the palm vein IR image sample to determine a palm vein feature sample;

[0037] By means of the specific encoder, feature extraction is performed on the palmprint feature sample to determine a specific palmprint feature sample, and feature extraction is performed on the palm vein feature sample to determine a specific palm vein feature sample;

[0038] By using the public encoder, extracting public features from the palm print feature samples and the palm vein feature samples to determine a public palm print feature sample and a public palm vein feature sample;

[0039] By means of the offset prediction module, the common palm print feature samples and the common palm vein feature samples are subjected to offset prediction to determine transformation matrix samples;

[0040] By means of the feature alignment module, the specific palmprint feature sample and the specific palm vein feature sample are feature aligned according to the transformation matrix sample;

[0041] By means of the feature fusion module, the specific palmprint feature sample and the specific palm vein feature sample after feature alignment are fused to obtain the palmprint and palm vein fusion feature map sample;

[0042] A specific encoder loss is determined according to the specific palm print feature samples and the specific palm vein feature samples, a common encoder loss is determined according to the common palm print feature samples and the common palm vein feature samples, a first loss is determined according to the palm print and palm vein fusion feature map samples, and parameters of the palm print and palm vein fusion model are optimized according to the specific encoder loss, the common encoder loss and the first loss.

[0043] In the above technical solution, the RGB image encoder and the IR image encoder are used to extract features from the palm print RGB image samples and the palm vein IR image samples respectively to obtain palm print feature samples and palm vein feature samples; the specific encoder is used to extract features from the palm print feature samples and the palm vein feature samples respectively to obtain specific palm print feature samples and specific palm vein feature samples; the public encoder is used to extract public features from the palm print feature samples and the palm vein feature samples to obtain public palm print feature samples and public palm vein feature samples; the offset prediction module is used to perform offset prediction on the public palm print feature samples and the public palm vein feature samples. The palm print and palm vein fusion feature map is trained through the above steps, so that the model can obtain the palm print and palm vein fusion feature map according to the palm print RGB image and the palm vein IR image. The palm print and palm vein fusion feature map realizes the effective combination of palm print and palm vein, and provides a high-precision and strong robust feature representation for biometrics.

[0044] According to an embodiment of the present application, determining a specific encoder loss according to the specific palm print feature sample and the specific palm vein feature sample includes:

[0045] The specific palmprint feature sample and the specific palm vein feature sample are input into a specific encoder loss function to determine a specific encoder loss, wherein the specific encoder loss function is:

[0046]

[0047] L s For the specific encoder loss, is the specific palmprint feature sample, is the specific palm vein feature sample.

[0048] In the above technical solution, specific palmprint feature samples and specific palm vein feature samples are input into a specific encoder loss function to determine the specific encoder loss. The specific encoder loss function quantifies the similarity between the specific palmprint feature samples and the specific palm vein feature samples by calculating the dot product of the specific palmprint feature samples and dividing the dot product by the product of their norms, thereby providing loss information for the training of the specific encoder, achieving effective training of the specific encoder, and further achieving effective training of the palmprint-palm vein fusion model, so that the model can obtain a palmprint-palm vein fusion feature map based on the palmprint RGB image and the palm vein IR image.

[0049] According to an embodiment of the present application, determining a common encoder loss according to the common palm print feature sample and the common palm vein feature sample includes:

[0050] The public palmprint feature sample and the public palm vein feature sample are input into a public encoder loss function to determine a public encoder loss, wherein the public encoder loss function is:

[0051]

[0052] L c is the common encoder loss, is the public palmprint feature sample, is the common palm vein feature sample.

[0053] In the above technical solution, the public palm print feature sample and the public palm vein feature sample are input into the public encoder loss function to determine the public encoder loss. The similarity between the public palm print feature sample and the public palm vein feature sample is quantified by calculating the dot product of the public palm print feature sample and the public palm vein feature sample and dividing it by the product of their respective norms. The public encoder loss function is obtained by subtracting the similarity value from 1. The public encoder loss function provides loss information for the training of the public encoder, realizes the effective training of the public encoder, and further realizes the effective training of the palm print and palm vein fusion model, so that the model can obtain the palm print and palm vein fusion feature map according to the palm print RGB image and the palm vein IR image.

[0054] According to an embodiment of the present application, the optimizing the parameters of the palm print and palm vein fusion model according to the specific encoder loss, the common encoder loss and the first loss includes:

[0055] Inputting the specific encoder loss, the common encoder loss and the first loss into the total loss function of the palmprint and palm vein fusion model to determine the total loss of the palmprint and palm vein fusion model;

[0056] The parameters of the palm print and palm vein fusion model are optimized based on the total loss of the palm print and palm vein fusion model; wherein the total loss function of the palm print and palm vein fusion model is:

[0057] L total =αL s +βL c +λL r

[0058] L total is the total loss of the palm print and palm vein fusion model, L s is the loss of the specific encoder, L c is the common encoder loss, L ris the first loss, α is the weight of the specific encoder loss, β is the weight of the public encoder loss, and λ is the weight of the first loss;

[0059] The first loss is a triplet loss or an additive angle interval loss determined based on the palm print and palm vein fusion feature map samples.

[0060] In the above technical scheme, the specific encoder loss, the common encoder loss and the first loss are weightedly combined into the total loss function of the palmprint and palm vein fusion model, and the model parameters are optimized accordingly, so as to achieve effective training of the palmprint and palm vein fusion model, so that the model can obtain the palmprint and palm vein fusion feature map according to the palmprint RGB image and the palm vein IR image.

[0061] In a second aspect, the present application provides a palm print and palm vein fusion device, the device comprising:

[0062] Inputting the palmprint RGB image and the palm vein IR image into the palmprint-palm vein fusion model, fusing the palmprint RGB image and the palm vein IR image, and obtaining a palmprint-palm vein fusion feature map;

[0063] The palm print and palm vein fusion model is trained based on palm print RGB image samples and palm vein IR image samples.

[0064] In the above technical solution, the palm print and palm vein fusion device is used to fuse the palm print RGB image and the palm vein IR image through the palm print and palm vein fusion model to generate a palm print and palm vein fusion feature map containing both palm print and palm vein features. Compared with a single palm print or palm vein image, biometric identification based on the palm print and palm vein fusion feature map can improve the accuracy of biometric identification. The palm print and palm vein fusion feature map can also be used for identification under external interference conditions, thereby improving the stability and robustness of biometric identification.

[0065] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the palm print and palm vein fusion method as described in the first aspect above is implemented.

[0066] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the palm print and palm vein fusion method as described in the first aspect above is implemented.

[0067] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the palm print and palm vein fusion method as described in the first aspect above.

[0068] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the palm print and palm vein fusion method as described in the first aspect above.

[0069] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0071] Figure 1 This is one of the flowcharts of the palm print and palm vein fusion method provided in some embodiments of the present application;

[0072] Figure 2 This is the second flow chart of the palm print and palm vein fusion method provided in some embodiments of the present application;

[0073] Figure 3 is a schematic diagram of the structure of a palm print and palm vein fusion model provided in some embodiments of the present application;

[0074] Figure 4 is a schematic diagram of the structure of a palm print and palm vein fusion device provided in some embodiments of the present application;

[0075] Figure 5 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application.

[0076] Description of reference numerals:

[0077] 200: Palm print and palm vein fusion model; 201: RGB image encoder; 202: IR image encoder;

[0078] 203: specific encoder; 204: common encoder; 205: offset prediction module;

[0079] 206: feature alignment module; 207: feature fusion module; 300: palm print and palm vein fusion device;

[0080] 301: fusion unit; 400: electronic device; 401: processor; 402: memory. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0082] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0083] The palm print and palm vein fusion method, device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0084] The palm print and palm vein fusion method provided in the embodiment of the present application can be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the palm print and palm vein fusion method. The electronic device mentioned in the embodiment of the present application includes but is not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The palm print and palm vein fusion method provided in the embodiment of the present application is described below by taking the electronic device as an example of the execution subject.

[0085] Figure 1 FIG. 1 is one of the flowcharts of the palm print and palm vein fusion method provided in some embodiments of the present application. Figure 1 As shown, the present application provides a palm print and palm vein fusion method, and the palm print and palm vein fusion method includes: step 100.

[0086] Step 100: inputting the palmprint RGB image and the palm vein IR image into a palmprint-palm vein fusion model, fusing the palmprint RGB image and the palm vein IR image, and obtaining a palmprint-palm vein fusion feature map;

[0087] The palm print and palm vein fusion model is trained based on palm print RGB image samples and palm vein IR image samples.

[0088] It can be understood that palm prints refer to the texture on the surface of the palm, including the main lines, wrinkles, and tiny details, which are unique due to individual differences and are widely used as a feature in biometric technology to verify or identify personal identity.

[0089] Palm veins refer to the veins in the vascular network inside the palm. Due to the absorption characteristics of blood to light of specific wavelengths, palm veins can be clearly displayed through specific imaging techniques (such as infrared imaging). The distribution of palm veins is also unique due to individual differences and is used as another feature in biometric technology.

[0090] RGB refers to red, green, and blue, representing the three basic color channels in color images. Palmprint RGB image is a color image of the palmprint, which is represented in RGB color space and contains information such as the texture of the palm surface (i.e., palmprint information).

[0091] IR refers to infrared. A palm vein IR image is an image of the vein distribution inside the palm taken using infrared imaging technology, which contains information such as the distribution of veins (i.e., palm vein information).

[0092] The palm print RGB image and palm vein IR image are input into the palm print palm vein fusion model to obtain the palm print palm vein fusion feature map, which includes palm print features and palm vein features and can represent palm print information and palm vein information. In application scenarios such as biometrics and identity authentication, the palm print palm vein fusion feature map has higher accuracy and stability compared with a single palm print RGB image or palm vein IR image.

[0093] It can be understood that the palm print and palm vein fusion feature map has strong robustness when used for biometrics. For example, when the lighting conditions are weak and the palm print RGB image acquisition effect is poor, biometrics can be performed through the palm vein information included in the palm print and palm vein fusion feature map.

[0094] In the above technical solution, the palm print RGB image and the palm vein IR image are fused through the palm print and palm vein fusion model to generate a palm print and palm vein fusion feature map containing both palm print and palm vein features. Compared with a single palm print or palm vein image, biometric identification based on the palm print and palm vein fusion feature map can improve the accuracy of biometric identification. The palm print and palm vein fusion feature map can also be used for identification under external interference conditions, thereby improving the stability and robustness of biometric identification.

[0095] Figure 2 This is the second flow chart of the palm print and palm vein fusion method provided in some embodiments of the present application. Figure 2As shown, in some embodiments, the palm print RGB image and the palm vein IR image are input into the palm print and palm vein fusion model, and the palm print RGB image and the palm vein IR image are fused to obtain a palm print and palm vein fusion feature map, including: step 110, step 120, step 130, step 140, step 150, step 160 and step 170.

[0096] Step 110, obtaining a palm print RGB image and a palm vein IR image;

[0097] Step 120, performing feature extraction on the palmprint RGB image and the palm vein IR image respectively to obtain palmprint features and palm vein features;

[0098] It can be understood that the palm print features are extracted from the palm print RGB image and used to represent the palm print information in the palm print RGB image, including main lines, wrinkles, etc.; the palm vein features are extracted from the palm vein IR image and used to represent the palm vein information in the palm vein IR image, including vein distribution, etc.

[0099] Step 130, extracting the palm print feature and the palm vein feature respectively to obtain a specific palm print feature and a specific palm vein feature;

[0100] Optionally, a specific palmprint feature is a high-level feature obtained by further extracting the palmprint feature, which can more deeply represent the unique texture features such as fine lines and wrinkles on the palm surface, and has higher uniqueness and discrimination than the palmprint feature.

[0101] Optionally, the specific palm vein feature is a high-level feature obtained by further extracting the palm vein feature, including unique vein features such as the detailed distribution of veins inside the palm and subtle differences in vein structure, and has higher uniqueness and discrimination than the palm vein feature.

[0102] Step 140: extracting common features from the palm print features and the palm vein features to obtain common palm print features and common palm vein features;

[0103] Optionally, the public palmprint feature is extracted from the palmprint feature and the palm vein feature, and is a feature in the palmprint feature that is common to the palm vein feature and does not depend on a specific modality (such as a palmprint RGB image or a palm vein IR image). For example, the overall outline of the palm is a common feature in both the palmprint feature and the palm vein feature, and can be extracted as part of the public palmprint feature through the public encoder.

[0104] Optionally, the common palm vein feature is extracted from the palm print feature and the palm vein feature, which is a feature shared by the palm vein feature and the palm print feature and does not depend on a specific modality, such as the general direction and distribution density of the vein, which may also be reflected in the palm print feature.

[0105] Step 150: performing offset prediction on the common palm print features and the common palm vein features to determine a transformation matrix;

[0106] It is understandable that due to different acquisition conditions, there may be differences in spatial position between the palm print RGB image and the palm vein IR image, which will lead to differences between the palm print features and the palm vein features, thus affecting the subsequent feature fusion. Therefore, before feature fusion, the palm print features and the palm vein features need to be aligned to achieve consistent spatial positions. As common features between the two modalities, the common palm print features and the common palm vein features provide a common benchmark for alignment and help eliminate differences.

[0107] The offset prediction is performed based on the common palm print features and the common palm vein features to determine the transformation matrix. The transformation matrix describes the spatial transformation relationship between the palm print features and the palm vein features, including transformations such as rotation, translation, and scaling.

[0108] Specific palmprint features are high-level features obtained by further extracting palmprint features, and specific palm vein features are also high-level features obtained by further extracting palm vein features. Therefore, the transformation matrix can also be used to describe the spatial transformation relationship between specific palmprint features and specific palm vein features. Using the transformation matrix, specific palmprint features or specific palm vein features can be spatially adjusted to achieve consistency in spatial position between the two, thereby improving the accuracy of palmprint and palm vein feature fusion.

[0109] Step 160: aligning the specific palm print feature and the specific palm vein feature according to the transformation matrix;

[0110] Optionally, the transformation matrix is ​​a first transformation matrix, which describes a spatial transformation relationship between a specific palm print feature and a specific palm vein feature. According to the first transformation matrix, the specific palm print feature can be adjusted to be consistent with the specific palm vein feature in spatial position.

[0111] Optionally, the transformation matrix is ​​a second transformation matrix, which describes the spatial transformation relationship between specific palm vein features and specific palm print features. According to the second transformation matrix, the specific palm vein features can be adjusted to be consistent with the specific palm print features in spatial position.

[0112] Step 170: Fusing the specific palm print feature and the specific palm vein feature after feature alignment to obtain a palm print and palm vein fusion feature map.

[0113] The aligned specific palmprint features and specific palm vein features are connected in a specific dimension to obtain the palmprint and palm vein fusion features.

[0114] Optionally, the palm print and palm vein fusion features are further processed by a convolutional network to generate a final palm print and palm vein fusion feature map, so as to more comprehensively represent the comprehensive biometric features of the palm.

[0115] In the above technical scheme, by acquiring a palmprint RGB image and a palm vein IR image, and extracting palmprint features and palm vein features respectively, extracting specific palmprint features and specific palm vein features from the palmprint features and palm vein features respectively, extracting the palmprint features and palm vein features together to obtain common palmprint features and common palm vein features, using the common palmprint features and common palm vein features to perform offset prediction to determine a transformation matrix, aligning the specific palmprint features and the specific palm vein features according to the transformation matrix, and fusion to obtain a palmprint-palm vein fusion feature map, achieving spatial consistency and effective combination of palmprints and palm veins, performing biometric identification based on the palmprint-palm vein fusion feature map, the accuracy of biometric identification can be improved, the palmprint-palm vein fusion feature map can also be used for identification under external interference conditions, and the stability and robustness of biometric identification are improved.

[0116] Figure 3 Schematic diagram of the structure of the palm print and palm vein fusion model provided in some embodiments of the present application. Figure 3 As shown, in some embodiments, the palm print and palm vein fusion model 200 includes: an RGB image encoder 201, an IR image encoder 202, a specific encoder 203, a common encoder 204, an offset prediction module 205, a feature alignment module 206 and a feature fusion module 207.

[0117] The RGB image encoder 201 is used to extract features from the palmprint RGB image to determine palmprint features;

[0118] Optionally, the RGB image encoder 201 is a convolutional neural network, comprising a plurality of convolutional layers, a pooling layer and an activation function. In the convolutional layer, the palmprint RGB image is scanned by a filter, and the texture features of different scales and directions of the palm surface can be effectively captured. The pooling layer in the convolutional neural network is used to reduce the dimension of the texture features and retain key palmprint information, thereby reducing the complexity of subsequent calculations. Such a design enables the RGB image encoder 201 to efficiently process the palmprint RGB image, extract the palmprint information such as the main lines, wrinkles and tiny details of the palm surface, and generate a high-dimensional feature vector, i.e., the palmprint feature.

[0119] The IR image encoder 202 is used to extract features from the palm vein IR image to determine palm vein features;

[0120] Optionally, the IR image encoder 202 adopts a structure similar to that of the RGB image encoder 201 to extract palm vein information such as the distribution and shape characteristics of the palm subcutaneous veins from the palm vein IR image and generate a high-dimensional feature vector, namely, a palm vein feature.

[0121] The specific encoder 203 is used to extract features from the palm print features to determine specific palm print features, and / or to extract features from the palm vein features to determine specific palm vein features;

[0122] Optionally, the specific encoder 203 is constructed by stacking multiple convolution blocks, and can extract specific features of different modalities layer by layer from local to global. For the palm print RGB image, the specific encoder 203 extracts more detailed palm print information such as the texture and shape of the palm surface based on the palm print features to generate specific palm print features, while for the palm vein IR image, the specific encoder 203 extracts more detailed palm vein information such as the distribution, direction and density of the veins based on the palm vein features to generate specific palm vein features. In the above manner, the two features are effectively deeply extracted, so that the features corresponding to different modalities (specific palm print features or specific palm vein features) can maintain their uniqueness as much as possible.

[0123] The public encoder 204 is used to extract the public features of the palm print features and the palm vein features, and determine the public palm print features and the public palm vein features;

[0124] Optionally, the public encoder 204 adopts a structure similar to the specific encoder 203. Under the premise of maintaining the difference between different modalities, it extracts the commonalities between the corresponding features (palmprint features and palm vein features) of the two, determines the public palmprint features and public palm vein features, and provides a basis for subsequent feature alignment and fusion.

[0125] The offset prediction module 205 is used to perform offset prediction on the common palm print feature and the common palm vein feature to determine a transformation matrix;

[0126] Optionally, the offset prediction module 205 calculates the difference in spatial position between the palmprint RGB image and the palm vein IR image based on the common palmprint features and the common palm vein features through a regression network, and predicts a transformation matrix including a rotation angle, a translation vector and a scaling factor.

[0127] Optionally, the transformation matrix is ​​expressed as:

[0128]

[0129] φ 2×3 is the transformation matrix, θ is the rotation angle between the specific palm print feature and the specific palm vein feature, (t x ,t y) is the translation vector between the specific palm print feature and the specific palm vein feature, (s x ,s y ) is the scaling factor between the specific palm print feature and the specific palm vein feature.

[0130] The feature alignment module 206 is used to perform feature alignment on the specific palm print feature and the specific palm vein feature according to the transformation matrix;

[0131] Optionally, the feature alignment module 206 includes a grid generator and a sampler, and performs feature alignment through a differentiable sampling mechanism.

[0132] In some embodiments, for a specific palm print feature Transformation is performed, and the grid generator is based on the transformation matrix φ 2×3 , calculate the target coordinates of the specific palmprint feature, the target coordinates indicate the spatial position of the specific palmprint feature after adjustment, the sampler in the module uses the bilinear interpolation algorithm to interpolate based on the target coordinates to determine the specific palmprint feature after alignment, that is, the aligned palmprint feature

[0133]

[0134] Alignment is the alignment algorithm used.

[0135] In some embodiments, for a specific palm vein feature Transformation is performed, and the grid generator is based on the transformation matrix φ 2×3 , calculate the target coordinates of the specific palm vein feature, the target coordinates indicate the spatial position of the specific palm vein feature after adjustment, the sampler in the module uses the bilinear interpolation algorithm to interpolate based on the target coordinates to determine the specific palm vein feature after alignment, that is, the aligned palm vein feature

[0136]

[0137] Alignment is the alignment algorithm used.

[0138] The feature fusion module 207 is used to fuse the specific palm print feature and the specific palm vein feature after feature alignment to obtain a palm print and palm vein fusion feature map.

[0139] Optionally, the feature fusion module 207 uses a splicing method to combine the aligned specific palm print features and specific palm vein features in a specific dimension to generate a palm print and palm vein fusion feature. Subsequently, the palm print and palm vein fusion feature is further extracted and compressed through a convolutional network to generate a compact and distinctive palm print and palm vein fusion feature map. The palm print and palm vein fusion feature map includes palm print information and palm vein information, providing a high-precision and robust feature representation for biometrics.

[0140] In some embodiments, the offset prediction module 205 is used to determine a first transformation matrix based on the common palm print feature and the common palm vein feature, wherein the first transformation matrix is ​​used to represent the spatial transformation relationship between the specific palm print feature and the specific palm vein feature;

[0141] The feature alignment module 206 is used to transform the specific palmprint feature according to the first transformation matrix to obtain an aligned palmprint feature, wherein the aligned palmprint feature and the specific palm vein feature have the same spatial position;

[0142] The feature fusion module 207 is used to fuse the aligned palm print features and the specific palm vein features to obtain the palm print and palm vein fusion feature map.

[0143] The offset prediction module is used to determine the first transformation matrix based on the common palm print features and the common palm vein features. The first transformation matrix describes the spatial transformation relationship between the specific palm print features and the specific palm vein features, that is, it indicates how to transform the palm print features to the same spatial position as the palm vein features. The feature alignment module is used to transform the specific palm print features according to the first transformation matrix to obtain the aligned palm print features, that is, the specific palm print features after the alignment process. The feature fusion module is used to fuse the aligned palm print features with the specific palm vein features to generate a palm print palm vein fusion feature map. The palm print palm vein fusion feature map realizes the effective combination of palm prints and palm veins, and provides a high-precision and strong robust feature representation for biometrics.

[0144] In some embodiments, the offset prediction module 205 is used to determine a second transformation matrix based on the common palm print feature and the common palm vein feature, wherein the second transformation matrix is ​​used to represent the spatial transformation relationship between the specific palm vein feature and the specific palm print feature;

[0145] The feature alignment module 206 is used to transform the specific palm vein feature according to the second transformation matrix to obtain an aligned palm vein feature, wherein the aligned palm vein feature and the specific palm print feature have the same spatial position;

[0146] The feature fusion module 207 is further used to fuse the aligned palm vein feature and the specific palm print feature to obtain the palm print and palm vein fusion feature map.

[0147] The offset prediction module is used to determine the second transformation matrix based on the common palm print features and the common palm vein features. The second transformation matrix describes the spatial transformation relationship between the specific palm vein features and the specific palm print features, that is, it indicates how to transform the palm vein features to the same spatial position as the palm print features. The feature alignment module is used to transform the specific palm vein features according to the second transformation matrix to obtain the aligned palm vein features, that is, the specific palm vein features after alignment processing. The feature fusion module is used to fuse the aligned palm vein features with the palm print features to generate a palm print and palm vein fusion feature map. The palm print and palm vein fusion feature map realizes the effective combination of palm prints and palm veins, and provides a high-precision and strong robust feature representation for biometrics.

[0148] In the above technical scheme, the palm print and palm vein fusion model includes an RGB image encoder, an IR image encoder, a specific encoder, a public encoder, an offset prediction module, a feature alignment module and a feature fusion module. The RGB image encoder is used to extract features from the palm print RGB image and the palm vein IR image, and then determine the specific palm print features and the specific palm vein features through the specific encoder. The public encoder is used to determine the public palm print features and the public palm vein features. The offset prediction module performs offset prediction on the specific palm print features and the specific palm vein features to determine the transformation matrix. The feature alignment module is used to achieve feature alignment of the specific palm print features and the specific palm vein features through the transformation matrix. The feature fusion module fuses the aligned feature palm print features and the specific palm vein features to obtain a palm print and palm vein fusion feature map, thereby achieving spatial consistency and effective combination of palm prints and palm veins. Biometrics can be identified based on the palm print and palm vein fusion feature map, which can improve the accuracy of biometrics. The palm print and palm vein fusion feature map can also be used for identification under external interference conditions, thereby improving the stability and robustness of biometrics.

[0149] According to one embodiment of the present application, the palm print and palm vein fusion model is trained by the following steps:

[0150] Obtain palmprint RGB image samples and palm vein IR image samples;

[0151] By means of the RGB image encoder, feature extraction is performed on the palmprint RGB image sample to determine a palmprint feature sample, and by means of the IR image encoder, feature extraction is performed on the palm vein IR image sample to determine a palm vein feature sample;

[0152] By means of the specific encoder, feature extraction is performed on the palmprint feature sample to determine a specific palmprint feature sample, and feature extraction is performed on the palm vein feature sample to determine a specific palm vein feature sample;

[0153] By using the public encoder, extracting public features from the palm print feature samples and the palm vein feature samples to determine a public palm print feature sample and a public palm vein feature sample;

[0154] By means of the offset prediction module, the common palm print feature samples and the common palm vein feature samples are subjected to offset prediction to determine transformation matrix samples;

[0155] By means of the feature alignment module, the specific palmprint feature sample and the specific palm vein feature sample are feature aligned according to the transformation matrix sample;

[0156] By means of the feature fusion module, the specific palmprint feature sample and the specific palm vein feature sample after feature alignment are fused to obtain the palmprint and palm vein fusion feature map sample;

[0157] A specific encoder loss is determined according to the specific palm print feature samples and the specific palm vein feature samples, a common encoder loss is determined according to the common palm print feature samples and the common palm vein feature samples, a first loss is determined according to the palm print and palm vein fusion feature map samples, and parameters of the palm print and palm vein fusion model are optimized according to the specific encoder loss, the common encoder loss and the first loss.

[0158] It can be understood that the palm print RGB image samples and palm vein IR image samples are the basic data for training the model, and contain diverse palm images so that the palm print and palm vein fusion model can learn the characteristics of different palms.

[0159] In some embodiments, the offset prediction module is used to determine a first transformation matrix sample based on the common palm print feature sample and the common palm vein feature sample, wherein the first transformation matrix sample is used to represent the spatial transformation relationship between the specific palm print feature sample and the specific palm vein feature sample;

[0160] The feature alignment module is used to transform the specific palmprint feature samples according to the first transformation matrix samples to obtain aligned palmprint feature samples, wherein the aligned palmprint feature samples and the specific palm vein feature samples have the same spatial position;

[0161] The feature fusion module is used to fuse the aligned palmprint feature samples and the specific palm vein feature samples to obtain the palmprint and palm vein fusion feature map samples.

[0162] In some embodiments, the offset prediction module is used to determine a second transformation matrix sample based on the common palm print feature sample and the common palm vein feature sample, wherein the second transformation matrix sample is used to represent the spatial transformation relationship between the specific palm vein feature sample and the specific palm print feature sample;

[0163] The feature alignment module is used to transform the specific palm vein feature sample according to the second transformation matrix sample to obtain an aligned palm vein feature sample, wherein the aligned palm vein feature sample and the specific palm print feature sample have the same spatial position;

[0164] The feature fusion module is further used to fuse the aligned palm vein feature samples and the specific palm print feature samples to obtain the palm print and palm vein fusion feature map sample.

[0165] In some embodiments, determining the specific encoder loss according to the specific palm print feature sample and the specific palm vein feature sample comprises:

[0166] The specific palmprint feature sample and the specific palm vein feature sample are input into a specific encoder loss function to determine a specific encoder loss, wherein the specific encoder loss function is:

[0167]

[0168] L s For the specific encoder loss, is the specific palmprint feature sample, is the specific palm vein feature sample.

[0169] The specific encoder loss function quantifies the difference between the specific palmprint feature sample and the specific palm vein feature sample by calculating the dot product of the specific palmprint feature sample and the specific palm vein feature sample and dividing it by the product of their respective norms. The dot product is defined as the sum of the products of the corresponding elements of the two features, which reflects the directional similarity of the two features in the multidimensional space. The norm is the length or modulus of the two features.

[0170] In the above technical solution, specific palmprint feature samples and specific palm vein feature samples are input into a specific encoder loss function to determine the specific encoder loss. The specific encoder loss function quantifies the similarity between the specific palmprint feature samples and the specific palm vein feature samples by calculating the dot product of the specific palmprint feature samples and dividing the dot product by the product of their norms, thereby providing loss information for the training of the specific encoder, achieving effective training of the specific encoder, and further achieving effective training of the palmprint-palm vein fusion model, so that the model can obtain a palmprint-palm vein fusion feature map based on the palmprint RGB image and the palm vein IR image.

[0171] In some embodiments, determining the public encoder loss according to the public palm print feature sample and the public palm vein feature sample comprises:

[0172] The public palmprint feature sample and the public palm vein feature sample are input into a public encoder loss function to determine a public encoder loss, wherein the public encoder loss function is:

[0173]

[0174] L c is the common encoder loss, is the public palmprint feature sample, is the common palm vein feature sample.

[0175] The public encoder loss function quantifies the difference between the public palm print feature samples and the public palm vein feature samples by calculating the dot product of the public palm print feature samples and the public palm vein feature samples, and divides it by the product of their respective norms, and subtracts the difference value from 1 to calculate the similarity.

[0176] In the above technical solution, the public palm print feature sample and the public palm vein feature sample are input into the public encoder loss function to determine the public encoder loss. The similarity between the public palm print feature sample and the public palm vein feature sample is quantified by calculating the dot product of the public palm print feature sample and the public palm vein feature sample and dividing it by the product of their respective norms. The public encoder loss function is obtained by subtracting the similarity value from 1. The public encoder loss function provides loss information for the training of the public encoder, realizes the effective training of the public encoder, and further realizes the effective training of the palm print and palm vein fusion model, so that the model can obtain the palm print and palm vein fusion feature map according to the palm print RGB image and the palm vein IR image.

[0177] In some embodiments, the optimizing the parameters of the palm print and palm vein fusion model according to the specific encoder loss, the common encoder loss and the first loss comprises:

[0178] Inputting the specific encoder loss, the common encoder loss and the first loss into the total loss function of the palmprint and palm vein fusion model to determine the total loss of the palmprint and palm vein fusion model;

[0179] The parameters of the palm print and palm vein fusion model are optimized based on the total loss of the palm print and palm vein fusion model; wherein the total loss function of the palm print and palm vein fusion model is:

[0180] L total =αL s +βL c +λL r

[0181] L total is the total loss of the palm print and palm vein fusion model, L s is the loss of the specific encoder, L c is the common encoder loss, L r is the first loss, α is the weight of the specific encoder loss, β is the weight of the public encoder loss, and λ is the weight of the first loss;

[0182] The first loss is a triplet loss or an additive angle interval loss determined based on the palm print and palm vein fusion feature map samples.

[0183] Optionally, the first loss is determined based on a palmprint and palm vein fusion feature map sample and a corresponding label. The palmprint and palm vein fusion feature map sample includes palmprint features and palm vein features, and the corresponding label represents the true identity of the individual to whom these features belong. The first loss can be a triplet loss (Triplet Loss) or an additive angular interval loss (ArcFaceloss), which is used to enhance the model's ability to distinguish palmprint and palm vein features.

[0184] It is understandable that the total loss of the palm print and palm vein fusion model is L total represents the overall loss of the model during training, the specific encoder loss L s Represents the difference between a specific palm print feature sample and a specific palm vein feature sample, and the public encoder L c The loss represents the similarity between the public palmprint feature samples and the public palm vein feature samples. The first loss L r , determined based on the palmprint and palm vein fusion feature map samples and corresponding labels, it is used to enhance the model's ability to distinguish palmprint and palm vein features. The weight coefficients α, β, and λ represent the relative importance of each loss function in the total loss.

[0185] In the above technical scheme, the specific encoder loss, the common encoder loss and the first loss are weightedly combined into the total loss function of the palmprint and palm vein fusion model, and the model parameters are optimized accordingly, so as to achieve effective training of the palmprint and palm vein fusion model, so that the model can obtain the palmprint and palm vein fusion feature map according to the palmprint RGB image and the palm vein IR image.

[0186] In the above technical solution, the RGB image encoder and the IR image encoder are used to extract features from the palm print RGB image samples and the palm vein IR image samples respectively to obtain palm print feature samples and palm vein feature samples; the specific encoder is used to extract features from the palm print feature samples and the palm vein feature samples respectively to obtain specific palm print feature samples and specific palm vein feature samples; the public encoder is used to extract public features from the palm print feature samples and the palm vein feature samples to obtain public palm print feature samples and public palm vein feature samples; the offset prediction module is used to perform offset prediction on the public palm print feature samples and the public palm vein feature samples. The palm print and palm vein fusion feature map is trained through the above steps, so that the model can obtain the palm print and palm vein fusion feature map according to the palm print RGB image and the palm vein IR image. The palm print and palm vein fusion feature map realizes the effective combination of palm print and palm vein, and provides a high-precision and strong robust feature representation for biometrics.

[0187] The palm print and palm vein fusion method provided in the embodiment of the present application can be executed by a palm print and palm vein fusion device. In the embodiment of the present application, the palm print and palm vein fusion method performed by the palm print and palm vein fusion device is taken as an example to illustrate the palm print and palm vein fusion device provided in the embodiment of the present application.

[0188] Figure 4 Schematic diagram of the structure of the palm print and palm vein fusion device provided in some embodiments of the present application. Figure 4 As shown, the palm print and palm vein fusion device 300 includes:

[0189] A fusion unit 301 is used to input the palm print RGB image and the palm vein IR image into a palm print and palm vein fusion model, fuse the palm print RGB image and the palm vein IR image, and obtain a palm print and palm vein fusion feature map;

[0190] The palm print and palm vein fusion model is trained based on palm print RGB image samples and palm vein IR image samples.

[0191] Optionally, the fusion unit 301 is used for:

[0192] Obtain palmprint RGB image and palm vein IR image;

[0193] Extracting features from the palmprint RGB image and the palm vein IR image respectively to obtain palmprint features and palm vein features;

[0194] Extracting features of the palm print feature and the palm vein feature respectively to obtain specific palm print features and specific palm vein features;

[0195] Extracting common features from the palm print features and the palm vein features to obtain common palm print features and common palm vein features;

[0196] Performing offset prediction on the common palm print feature and the common palm vein feature to determine a transformation matrix;

[0197] Performing feature alignment on the specific palmprint feature and the specific palm vein feature according to the transformation matrix;

[0198] The specific palm print feature and the specific palm vein feature after feature alignment are fused to obtain a palm print and palm vein fusion feature map.

[0199] Optionally, the palm print and palm vein fusion model includes: an RGB image encoder, an IR image encoder, a specific encoder, a common encoder, an offset prediction module, a feature alignment module and a feature fusion module.

[0200] The RGB image encoder is used to extract features from the palmprint RGB image to determine palmprint features;

[0201] The IR image encoder is used to extract features from the palm vein IR image to determine palm vein features;

[0202] The specific encoder is used to extract features from the palm print features to determine specific palm print features, and / or to extract features from the palm vein features to determine specific palm vein features;

[0203] The public encoder is used to extract the public features of the palm print features and the palm vein features, and determine the public palm print features and the public palm vein features;

[0204] The offset prediction module is used to perform offset prediction on the common palm print feature and the common palm vein feature to determine a transformation matrix;

[0205] The feature alignment module is used to perform feature alignment on the specific palm print feature and the specific palm vein feature according to the transformation matrix;

[0206] The feature fusion module is used to fuse the specific palm print feature and the specific palm vein feature after feature alignment to obtain a palm print and palm vein fusion feature map.

[0207] Optionally, the offset prediction module is used to determine a first transformation matrix according to the common palm print feature and the common palm vein feature, wherein the first transformation matrix is ​​used to represent a spatial transformation relationship between the specific palm print feature and the specific palm vein feature;

[0208] The feature alignment module is used to transform the specific palmprint feature according to the first transformation matrix to obtain an aligned palmprint feature, wherein the aligned palmprint feature and the specific palm vein feature have the same spatial position;

[0209] The feature fusion module is used to fuse the aligned palm print features and the specific palm vein features to obtain the palm print and palm vein fusion feature map.

[0210] Optionally, the offset prediction module is used to determine a second transformation matrix according to the common palm print feature and the common palm vein feature, wherein the second transformation matrix is ​​used to represent a spatial transformation relationship between the specific palm vein feature and the specific palm print feature;

[0211] The feature alignment module is used to transform the specific palm vein feature according to the second transformation matrix to obtain an aligned palm vein feature, wherein the aligned palm vein feature and the specific palm print feature have the same spatial position;

[0212] The feature fusion module is further used to fuse the aligned palm vein feature and the specific palm print feature to obtain the palm print and palm vein fusion feature map.

[0213] Optionally, the palm print and palm vein fusion model is trained by the following steps:

[0214] Obtain palmprint RGB image samples and palm vein IR image samples;

[0215] By means of the RGB image encoder, feature extraction is performed on the palmprint RGB image sample to determine a palmprint feature sample, and by means of the IR image encoder, feature extraction is performed on the palm vein IR image sample to determine a palm vein feature sample;

[0216] By means of the specific encoder, feature extraction is performed on the palmprint feature sample to determine a specific palmprint feature sample, and feature extraction is performed on the palm vein feature sample to determine a specific palm vein feature sample;

[0217] By using the public encoder, extracting public features from the palm print feature samples and the palm vein feature samples to determine a public palm print feature sample and a public palm vein feature sample;

[0218] By means of the offset prediction module, the common palm print feature samples and the common palm vein feature samples are subjected to offset prediction to determine transformation matrix samples;

[0219] By means of the feature alignment module, the specific palmprint feature sample and the specific palm vein feature sample are feature aligned according to the transformation matrix sample;

[0220] By means of the feature fusion module, the specific palmprint feature sample and the specific palm vein feature sample after feature alignment are fused to obtain the palmprint and palm vein fusion feature map sample;

[0221] A specific encoder loss is determined according to the specific palm print feature samples and the specific palm vein feature samples, a common encoder loss is determined according to the common palm print feature samples and the common palm vein feature samples, a first loss is determined according to the palm print and palm vein fusion feature map samples, and parameters of the palm print and palm vein fusion model are optimized according to the specific encoder loss, the common encoder loss and the first loss.

[0222] Optionally, determining a specific encoder loss according to the specific palm print feature sample and the specific palm vein feature sample includes:

[0223] The specific palmprint feature sample and the specific palm vein feature sample are input into a specific encoder loss function to determine a specific encoder loss, wherein the specific encoder loss function is:

[0224]

[0225] L s For the specific encoder loss, is the specific palmprint feature sample, is the specific palm vein feature sample.

[0226] Optionally, determining a public encoder loss according to the public palm print feature sample and the public palm vein feature sample includes:

[0227] The public palmprint feature sample and the public palm vein feature sample are input into a public encoder loss function to determine a public encoder loss, wherein the public encoder loss function is:

[0228]

[0229] L c is the common encoder loss, is the public palmprint feature sample, is the common palm vein feature sample.

[0230] Optionally, optimizing the parameters of the palm print and palm vein fusion model according to the specific encoder loss, the common encoder loss and the first loss includes:

[0231] Inputting the specific encoder loss, the common encoder loss and the first loss into the total loss function of the palmprint and palm vein fusion model to determine the total loss of the palmprint and palm vein fusion model;

[0232] The parameters of the palm print and palm vein fusion model are optimized based on the total loss of the palm print and palm vein fusion model; wherein the total loss function of the palm print and palm vein fusion model is:

[0233] L total =αL s +βL c +λL r

[0234] L total is the total loss of the palm print and palm vein fusion model, L s is the loss of the specific encoder, L c is the common encoder loss, L r is the first loss, α is the weight of the specific encoder loss, β is the weight of the public encoder loss, and λ is the weight of the first loss;

[0235] The first loss is a triplet loss or an additive angle interval loss determined based on the palm print and palm vein fusion feature map samples.

[0236] In the above technical solution, the palm print and palm vein fusion device is used to fuse the palm print RGB image and the palm vein IR image through the palm print and palm vein fusion model to generate a palm print and palm vein fusion feature map containing both palm print and palm vein features. Compared with a single palm print or palm vein image, biometric identification based on the palm print and palm vein fusion feature map can improve the accuracy of biometric identification. The palm print and palm vein fusion feature map can also be used for identification under external interference conditions, thereby improving the stability and robustness of biometric identification.

[0237] The palm print and palm vein fusion device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0238] The palm print and palm vein fusion device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0239] The palm print and palm vein fusion device provided in the embodiment of the present application can realize each process realized by the above-mentioned palm print and palm vein fusion method embodiment, and will not be described again here to avoid repetition.

[0240] In some embodiments, Figure 5 As shown, the embodiment of the present application also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the above-mentioned palm print and palm vein fusion method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0241] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0242] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned palm print and palm vein fusion method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0243] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0244] The embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned palm print and palm vein fusion method when executed by a processor.

[0245] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0246] The embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned palm print and palm vein fusion method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0247] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0248] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0249] For example, the palm print and palm vein fusion device can be used in financial institutions, public transportation, and education scenarios. Financial institutions include but are not limited to banks, securities companies, or insurance institutions, public transportation includes but is not limited to subway stations, high-speed rail stations, and airports, and education scenarios include: examination rooms and teaching sign-in. In the above scenarios, before obtaining data through the palm print and palm vein fusion device, it has been fully authorized by all parties.

[0250] Of course, the palm print and palm vein fusion device can also be used in private scenarios, such as company attendance. When used in private scenarios, user authorization must be obtained before obtaining data through the palm print and palm vein fusion device.

[0251] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0252] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0253] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0254] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0255] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A palm print and palm vein fusion method, characterized in that: include: Inputting the palmprint RGB image and the palm vein IR image into the palmprint-palm vein fusion model, fusing the palmprint RGB image and the palm vein IR image, and obtaining a palmprint-palm vein fusion feature map; The palm print and palm vein fusion model is trained based on palm print RGB image samples and palm vein IR image samples.

2. The palm print and palm vein fusion method according to claim 1, characterized in that: The step of inputting the palm print RGB image and the palm vein IR image into the palm print and palm vein fusion model, fusing the palm print RGB image and the palm vein IR image, and obtaining a palm print and palm vein fusion feature map includes: Obtain palmprint RGB image and palm vein IR image; Extracting features from the palmprint RGB image and the palm vein IR image respectively to obtain palmprint features and palm vein features; Extracting features of the palm print feature and the palm vein feature respectively to obtain specific palm print features and specific palm vein features; Extracting common features from the palm print features and the palm vein features to obtain common palm print features and common palm vein features; Performing offset prediction on the common palm print feature and the common palm vein feature to determine a transformation matrix; According to the transformation matrix, aligning the specific palm print feature and the specific palm vein feature; The specific palm print feature and the specific palm vein feature after feature alignment are fused to obtain a palm print and palm vein fusion feature map.

3. The palm print and palm vein fusion method according to claim 1 or 2, characterized in that: The palm print and palm vein fusion model includes: an RGB image encoder, an IR image encoder, a specific encoder, a common encoder, an offset prediction module, a feature alignment module and a feature fusion module. The RGB image encoder is used to extract features from the palmprint RGB image to determine palmprint features; The IR image encoder is used to extract features from the palm vein IR image to determine palm vein features; The specific encoder is used to extract features from the palm print features to determine specific palm print features, and / or to extract features from the palm vein features to determine specific palm vein features; The public encoder is used to extract the public features of the palm print features and the palm vein features, and determine the public palm print features and the public palm vein features; The offset prediction module is used to perform offset prediction on the common palm print feature and the common palm vein feature to determine a transformation matrix; The feature alignment module is used to perform feature alignment on the specific palm print feature and the specific palm vein feature according to the transformation matrix; The feature fusion module is used to fuse the specific palm print feature and the specific palm vein feature after feature alignment to obtain a palm print and palm vein fusion feature map.

4. The palm print and palm vein fusion method according to claim 3, characterized in that: The offset prediction module is used to determine a first transformation matrix according to the common palm print feature and the common palm vein feature, wherein the first transformation matrix is ​​used to represent the spatial transformation relationship between the specific palm print feature and the specific palm vein feature; The feature alignment module is used to transform the specific palmprint feature according to the first transformation matrix to obtain an aligned palmprint feature, wherein the aligned palmprint feature and the specific palm vein feature have the same spatial position; The feature fusion module is used to fuse the aligned palm print features and the specific palm vein features to obtain the palm print and palm vein fusion feature map.

5. The palm print and palm vein fusion method according to claim 3, characterized in that: The offset prediction module is used to determine a second transformation matrix according to the common palm print feature and the common palm vein feature, wherein the second transformation matrix is ​​used to represent the spatial transformation relationship between the specific palm vein feature and the specific palm print feature; The feature alignment module is used to transform the specific palm vein feature according to the second transformation matrix to obtain an aligned palm vein feature, wherein the aligned palm vein feature and the specific palm print feature have the same spatial position; The feature fusion module is further used to fuse the aligned palm vein feature and the specific palm print feature to obtain the palm print and palm vein fusion feature map.

6. The palm print and palm vein fusion method according to claim 3, characterized in that: The palm print and palm vein fusion model is trained by the following steps: Obtain palmprint RGB image samples and palm vein IR image samples; By means of the RGB image encoder, feature extraction is performed on the palmprint RGB image sample to determine a palmprint feature sample, and by means of the IR image encoder, feature extraction is performed on the palm vein IR image sample to determine a palm vein feature sample; By means of the specific encoder, feature extraction is performed on the palmprint feature sample to determine a specific palmprint feature sample, and feature extraction is performed on the palm vein feature sample to determine a specific palm vein feature sample; By using the public encoder, extracting public features from the palm print feature samples and the palm vein feature samples to determine a public palm print feature sample and a public palm vein feature sample; By means of the offset prediction module, the common palm print feature samples and the common palm vein feature samples are subjected to offset prediction to determine transformation matrix samples; By means of the feature alignment module, the specific palmprint feature sample and the specific palm vein feature sample are feature aligned according to the transformation matrix sample; By means of the feature fusion module, the specific palmprint feature sample and the specific palm vein feature sample after feature alignment are fused to obtain the palmprint and palm vein fusion feature map sample; A specific encoder loss is determined according to the specific palm print feature samples and the specific palm vein feature samples, a common encoder loss is determined according to the common palm print feature samples and the common palm vein feature samples, a first loss is determined according to the palm print and palm vein fusion feature map samples, and parameters of the palm print and palm vein fusion model are optimized according to the specific encoder loss, the common encoder loss and the first loss.

7. The palm print and palm vein fusion method according to claim 6, characterized in that: The determining of the specific encoder loss according to the specific palm print feature sample and the specific palm vein feature sample comprises: The specific palmprint feature sample and the specific palm vein feature sample are input into a specific encoder loss function to determine a specific encoder loss, wherein the specific encoder loss function is: L s For the specific encoder loss, is the specific palmprint feature sample, is the specific palm vein feature sample.

8. The palm print and palm vein fusion method according to claim 6, characterized in that: The determining of the public encoder loss according to the public palm print feature sample and the public palm vein feature sample comprises: The public palmprint feature sample and the public palm vein feature sample are input into a public encoder loss function to determine a public encoder loss, wherein the public encoder loss function is: L c is the common encoder loss, is the public palmprint feature sample, is the common palm vein feature sample.

9. The palm print and palm vein fusion method according to any one of claims 6 to 8, characterized in that: The step of optimizing the parameters of the palm print and palm vein fusion model according to the specific encoder loss, the common encoder loss and the first loss includes: Inputting the specific encoder loss, the common encoder loss and the first loss into the total loss function of the palmprint and palm vein fusion model to determine the total loss of the palmprint and palm vein fusion model; Optimizing the parameters of the palmprint and palm vein fusion model based on the total loss of the palmprint and palm vein fusion model; Among them, the total loss function of the palm print and palm vein fusion model is: L total =αL s +βL c +λL r L total is the total loss of the palm print and palm vein fusion model, L s is the loss of the specific encoder, L c is the common encoder loss, L r is the first loss, α is the weight of the specific encoder loss, β is the weight of the public encoder loss, and λ is the weight of the first loss; The first loss is a triplet loss or an additive angle interval loss determined based on the palm print and palm vein fusion feature map samples.

10. A palm print and palm vein fusion device, characterized in that: include: A fusion unit, used for inputting the palm print RGB image and the palm vein IR image into a palm print and palm vein fusion model, fusing the palm print RGB image and the palm vein IR image, and obtaining a palm print and palm vein fusion feature map; The palm print and palm vein fusion model is trained based on palm print RGB image samples and palm vein IR image samples.

Images