Identity recognition device and method based on multi-modal hand veins

Abstract

The invention belongs to the technical field of identity recognition, and particularly discloses an identity recognition device and method based on multi-mode hand veins. The device comprises a vein image acquisition device and a computer device, and an improved UNet network and a multi-mode fusion network CVit are deployed on the computer device. The vein image acquisition equipment is multi-mode acquisition equipment and can acquire hand back and finger vein images at the same time, in addition, the invention further provides a self-adaptive dimming algorithm, proper illumination intensity is adjusted according to different palm thicknesses, and the problem that veins are not clear due to the palm thicknesses is solved. The ROI of any vein image can be extracted by the improved UNet network, so that the development cost is greatly reduced. According to the multi-modal fusion network CVit, VGG16 is used as a backbone network, features between two modals are fully extracted, the fusion part uses an attention mechanism in improved Transform to extract features associated with the two modals, the two sets of features are fused, and the effectiveness and robustness of vein recognition are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of identity recognition, and relates to an identity recognition device and method based on multi-modal hand veins. Background Art

[0002] In recent years, due to the convenience and uniqueness of biometrics, good results have been achieved in the field of identity recognition and authentication. Biometrics include face, iris, ear, fingerprint, palm print, back of hand vein, finger vein, gait, etc. There are a large number of vascular tissues distributed in the skin of the palm and back of the hand, which constitute the vein characteristics of the hand. Generally speaking, the hand vein characteristics remain relatively unchanged after a person's puberty and are unique to a subject, making them difficult to steal and copy. Compared with other biometrics, veins have the advantages of easy collection, high stability, being inside the palm and not easily affected by external wear and tear, and being able to perform liveness detection. However, a single modality is easy to obtain and forge, and has poor security, so hand vein multimodal identity authentication is a more secure authentication method. In recent years, many multimodal recognition studies have appeared.

[0003] At present, there are two main methods for obtaining vein images according to the light source: far-infrared imaging and near-infrared imaging. Fig.16 As shown in (a) and (b) in the figure. Near-infrared imaging is the most mainstream method at present. It uses the characteristics of hemoglobin in the blood that has a strong ability to absorb near-infrared light. Blood vessels absorb infrared light and appear dark, while other tissues have a weak ability to absorb infrared light and appear light. Far-infrared imaging is more sensitive to environmental conditions, while near-infrared imaging has better robustness and can maintain relatively stable imaging quality in most cases.

[0004] According to the propagation path of light, it can be divided into three types: reflection imaging, transmission imaging and mixed mode imaging. Mixed mode imaging is a combination of reflection mode and transmission mode. Fig.17 As shown in (a) and (b), reflection imaging uses the reflection of light. The light source and the camera are on the same side, and the camera receives the light reflected after being absorbed by the hand. Transmission imaging uses the transmission of light. The light source and the camera are on both sides of the hand. The light source emits light, passes through the hand, and the camera captures the light after being absorbed by the hand.

[0005] Based on this, the current vein collection device is mainly composed of a light source, a diffuser, a lens, a filter, a computing board, a power supply, and a shell. There are three types of vein collection methods: transmission, reflection, and dual-mode. The current research is basically a single-mode collection device for finger veins, and multi-modal collection methods such as back of hand veins and finger veins have not yet appeared.

[0006] Patent document 1 discloses an end-to-end finger trimodal feature fusion classification device and method. The patent document collects trimodal finger images, constructs feature extraction models based on convolutional neural networks for finger vein images, fingerprint images and knuckle print images, respectively, to extract trimodal finger features, inputs all extracted features into the finger trimodal feature fusion model for fusion, obtains trimodal finger fusion features, and outputs feature vectors of trimodal finger fusion features based on a fully connected layer; classification is achieved based on the distance measurement results between the feature vectors and each vector in a preset feature vector set.

[0007] The vein acquisition device used in this method does not have an LED dimming design, so the quality of the vein images collected for palms of different thicknesses is different. In the image segmentation task, the network will learn the characteristics of the segmentation target and then classify each pixel to achieve image segmentation. A high-quality vein image needs to contain as much vein information as possible, and the segmented vein area will be larger. The vein acquisition device of Patent Document 1 cannot guarantee the quality of the above vein image. In addition, the recognition network used in Patent Document 1 only convolves each modality, ignoring the mutual characteristics between different modalities.

[0008] The veins on the back of the hand and the veins on the fingers are distributed on different sides of the hand and are not easy to capture under visible light. The two can constitute multimodal recognition, which can greatly reduce the risk of theft. Both are vein features, which can greatly reduce the development cost in the process of data processing and two-modal fusion. However, the current limitation of multimodal recognition of hand veins is that the data sets of different modalities are not collected from the same subject, but are composed of single-modal data sets of different subjects. Collecting multimodal data from the same subject is more valuable for research, because the inter-class similarity of multimodal data from the same subject is small, and the intra-class similarity is large, which can improve the effectiveness and robustness of personal identity recognition and is more in line with real-life needs.

[0009] References

[0010] Patent document 1 Chinese invention patent application, publication number: CN115223210A, publication date: 2022.10.21. Summary of the invention

[0011] The purpose of the present invention is to propose an identity recognition device based on multi-modal hand veins. The vein image acquisition device in the device can simultaneously acquire infrared images of the back of the hand veins and the finger veins. Compared with the single-modal vein acquisition device, the acquisition efficiency is significantly improved. In addition, through the proposed adaptive brightness adjustment algorithm, it is helpful to solve the problem of unclear vein images collected due to palm thickness, improve the quality of vein image acquisition, and ensure the accuracy of vein identity recognition.

[0012] In order to achieve the above object, the present invention adopts the following technical scheme:

[0013] An identity recognition device based on multimodal hand veins includes a vein image acquisition device and a computer device; the vein image acquisition device includes a housing, a camera, a host computer, a light source board and a light source core board;

[0014] An opening is provided in the middle of the front side of the shell for inserting a hand;

[0015] There are two cameras, defined as the first camera and the second camera;

[0016] The first camera is arranged at the top of the front side of the housing, and the lens of the first camera is directed downward to the back of the hand; the second camera is arranged at the bottom of the right side of the housing, and the lens of the second camera is directed upward to the finger position;

[0017] There are four light source panels, which are defined as a back-of-hand reflection light source panel, a back-of-hand transmission light source panel, a finger reflection light source panel, and a finger transmission light source panel; the four light source panels are arranged in the housing as follows:

[0018] The hand back reflection light source plate is located below the first camera, and a filter is installed at the lens of the first camera;

[0019] The finger reflection light source plate is located above the second camera, and a filter is installed at the lens of the second camera;

[0020] The hand back transmission light source panel is located in the middle of the front side of the housing, and the installation height of the hand back transmission light source panel is lower than the height of the opening, so that when the hand is fully inserted into the housing, the hand back transmission light source panel is located below the hand;

[0021] The finger transmission light source plate is aligned with the first camera at the upper and lower positions;

[0022] The finger-transmitting light source board is located in the middle of the rear side of the housing, and the installation height of the finger-transmitting light source board is higher than the height of the opening, so that when the hand is fully inserted into the housing, the finger-transmitting light source board is located above the hand;

[0023] The finger transmits the light source plate and aligns it with the second camera at the upper and lower positions;

[0024] The four light source boards are connected to the light source core board through lines respectively, and the light source core board is connected to the host computer through lines;

[0025] The first camera and the second camera are connected to the host computer through lines.

[0026] In addition, based on the above-mentioned multi-modal hand vein-based identity recognition device, the present invention also proposes a multi-modal hand vein-based identity recognition method, which adopts the following technical solutions:

[0027] A method for identifying a person based on multimodal hand veins comprises the following steps:

[0028] Step 1. Place the palm to be collected on the vein image acquisition device and adjust the light. After two steps of coarse adjustment and fine adjustment, set the optimal LED brightness, and collect the back of the hand and finger modal data at the same time;

[0029] Step 2. First, the vein dataset collected in step 1 is sent to the improved VU-Net network for vein segmentation. Then the smallest rectangle containing all vein information is found and the vertex information of the smallest rectangle is recorded. Finally, the vertex position of the smallest rectangle is found on the original image and the rectangular area is cropped to obtain the vein ROI image.

[0030] Step 3. Build a CVit multimodal network with a dual-branch structure;

[0031] The built CVit multimodal network includes feature extraction network, fusion network, feature reconstruction network and classification network;

[0032] The fusion network includes a first branch and a second branch structure;

[0033] The SENet channel attention mechanism in the first branch helps the network learn the correlation between the features of each modality more effectively, and the Cross-Attention mutual attention mechanism in the second branch learns the correlation between the two modalities;

[0034] The signal processing flow in the CVit multimodal network is as follows:

[0035] The hand back vein image and finger vein image are input into the feature extraction network to obtain the hand back vein feature map f DHV And finger vein feature map f FV ;

[0036] The feature map output by the feature extraction network is input into the first branch of the fusion network. The processing flow of the first branch is as follows:

[0037] The input feature map is subjected to global average pooling to obtain a one-dimensional vector S1″ of dimension 1×1×512, and then S1″ is input into a fully connected layer of dimension 128 to obtain a vector S2″ of dimension 1×1×512, and then passed through a 512-dimensional fully connected layer to obtain a vector S3″ of dimension 1×1×512. Finally, S3″ is multiplied by the input feature to obtain the output feature map;

[0038] Hand vein characteristics DHV After the feature extraction of the first branch, the feature map f is obtained DHV_SE , finger vein feature map f FV After the feature extraction of the first branch, the feature map f is obtained FV_SE ;

[0039] The feature map output by the feature extraction network is input into the second branch of the fusion network. The processing flow of the second branch is as follows:

[0040] f DHV and f FV Divide the space into one-dimensional vectors, and get a vector f of size 49×512 DHV_token1 and f FV_token1 , the vector f DHV_token1 and f FV_token1 Input them into a Transformer network respectively, and swap the Q in the two Transformer networks to generate a vector f of dimension 49×512 DHV_token2 and f FV_token2 , and then restore the vector to 7×7×512 size and add it to f DHV and f FV Add them together to get f DHV_ViT and f FV_ViT ;

[0041] The feature map f DHV_SE 、f FV_SE 、f DHV_ViT 、f FV_ViT Concatenate to get the feature map F fusion , and then through the feature reconstruction network, further features are extracted, and finally through two fully connected layers to obtain the identity recognition result;

[0042] Among them, the VU-Net network and the CVit multimodal network are both deployed on computer devices.

[0043] The present invention has the following advantages:

[0044] As described above, the present invention relates to an identity recognition device and method based on multimodal hand veins. The identity recognition device includes a vein image acquisition device and a computer device, and an improved UNet network and a multimodal fusion network CVit are deployed on the computer device. The vein image acquisition device in the present invention is a multimodal acquisition device, which fills the gap of multimodal acquisition devices for back veins and finger veins. The multimodal acquisition device can acquire two modes at the same time. Compared with the traditional single-modal acquisition device, the multimodal acquisition device significantly improves the acquisition efficiency. In addition, based on the above-mentioned vein image acquisition device, the present invention further proposes an adaptive dimming algorithm. Compared with a fixed light source, the adaptive dimming algorithm proposed in the present invention will select a suitable light intensity according to the thickness of the palm, solving the problem of unclear veins due to the thickness of the palm. The VU-Net algorithm proposed in the present invention uses an improved UNet network to extract the characteristics of the veins, classifies each pixel of the image, thereby identifying the range of the vein image, and then extracts the ROI, which can extract the ROI area of ​​any vein image, greatly reducing the development cost. The multimodal fusion network CVit proposed in this invention uses VGG16 as the backbone network in the feature extraction part to fully extract the features between the two modalities. The fusion part fuses the two sets of features of the two modalities. On the basis of retaining the features of the finger vein and the back of the hand vein, the Cross-Attention mechanism in the Transformer is used to extract the features associated with the two modalities. The fusion of the two sets of features improves the expression ability of the network, makes it better adaptable to classification tasks, and improves the effectiveness and robustness of vein recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of an identity recognition device based on multi-modal hand veins in an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of a vein image acquisition device in an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the hand back vein transmission dimming algorithm in an embodiment of the present invention;

[0048] Figure 4 Schematic diagram of the finger vein transmission dimming algorithm in an embodiment of the present invention;

[0049] Figure 5 Schematic diagram of the hand back vein reflection dimming algorithm in an embodiment of the present invention;

[0050] Figure 6 is a flow chart of an identity recognition method based on multi-modal hand veins in an embodiment of the present invention;

[0051] Figure 7 It is a structural diagram of the improved VU-Net network in an embodiment of the present invention;

[0052] Figure 8 It is a structural diagram of SENet in an embodiment of the present invention;

[0053] Fig. 9 It is a structural diagram of DeSENet in an embodiment of the present invention;

[0054] Fig.10 Schematic diagram of vein ROI extraction steps in an embodiment of the present invention;

[0055] Fig.11 Schematic diagram of the structure of a multimodal fusion network in an embodiment of the present invention;

[0056] Fig.12 A schematic diagram of a feature extraction network in an embodiment of the present invention;

[0057] Fig.13 This is a schematic diagram of the first branch of the fusion network in an embodiment of the present invention;

[0058] Fig.14 This is a schematic diagram of the second branch of the fusion network in an embodiment of the present invention;

[0059] Fig.15 This is a schematic diagram of a feature reconstruction network in an embodiment of the present invention;

[0060] Fig.16 Schematic diagram of vein images obtained by two methods of dividing according to light sources; Fig.16 (a) is a vein image obtained by far-infrared imaging, and (b) is a vein image obtained by near-infrared imaging;

[0061] Fig.17 A schematic diagram of a method for obtaining a vein image according to the propagation path of light; wherein Fig.17 (a) is a method of obtaining a vein image using the reflection imaging principle, and (b) is a method of obtaining a vein image using the transmission imaging principle;

[0062] Fig.18 This is the distribution diagram of LEDs in the hand back vein transmission light source panel;

[0063] Fig.19 This is the distribution diagram of LEDs of the finger vein transmission light source panel;

[0064] Fig. 20 This is the distribution diagram of LEDs on the finger reflection light source board and the back of the hand reflection light source board.

[0065] Among them, 1-shell, 2-host computer, 3-opening, 4-first camera, 5-second camera, 6-back of hand reflection light source board, 7-back of hand transmission light source board, 8-finger reflection light source board, 9-finger transmission light source board;

[0066] 10-first optical filter, 11-second optical filter, 12-light source core board, 13-power supply. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0068] Example 1

[0069] This embodiment describes an identity recognition device based on multimodal hand veins. The vein image acquisition device provided by the device is a convenient and efficient multimodal acquisition device that can simultaneously acquire infrared images of the back veins and finger veins, making up for the lack of multimodal devices for hand veins. Compared with traditional single-modal devices to acquire multimodal data sets, it greatly reduces the time cost. In addition, the blood in the human body changes all the time, and the similarity of the two vein data acquired at the same time is the greatest, which is also impossible for single-modal acquisition devices. In addition, based on the above-mentioned multimodal acquisition device, the present invention also proposes an adaptive brightness adjustment algorithm based on VU-net, which solves the problem that the images acquired by palms of different thicknesses facing fixed light intensity will be too bright or too dark, and can provide appropriate light intensity for palms of different thicknesses.

[0070] like Figure 1 As shown, the multi-modal hand vein-based identity recognition device in this embodiment includes two parts: a vein image acquisition device and a computer device. The vein image acquisition device is used to acquire vein image data.

[0071] The data set collection part sets different LED light intensities according to the thickness of the human palm, collects vein data through the CCD camera, and provides training data for the network. In the recognition process, it can also be used as an input source for recognition data.

[0072] The VU-Net network and the recognition network, namely the multimodal fusion network CVit, are deployed on the computer device.

[0073] The multimodal ROI extraction algorithm based on the improved VU-Net network is helpful to solve the problem of different ROI extraction methods between different modalities; the multimodal fusion network CVit fuses the dorsal hand veins with the finger veins, thereby extracting the correlation information between the dorsal hand veins and the finger veins, which can effectively improve the accuracy of identity recognition.

[0074] Specifically, the recognition network uses the VGG16 network for feature extraction, and the feature fusion part fuses the dual-branch structure of SENet and the mutual attention mechanism. After fusion, the feature reconstruction network is used to reduce the number of parameters in the fully connected layer. The data set collected by the data acquisition part is used for training, and the best model is deployed in the recognition task. The multimodal vein image to be registered is feature extracted and the features are saved in the local database. During recognition, the same feature extraction operation is performed on the image obtained from the data acquisition part, and the best matching feature is selected in the database for category output.

[0075] like Figure 2 As shown, the vein image acquisition device includes a housing 1, a camera, a host computer 2, a light source board and a light source core board 12, and a power supply 13. An opening 3 for a hand to be inserted is provided in the middle of the front side of the housing 1. Figure 2 The middle right side of the .

[0076] There are two cameras, which are defined as a first camera 4 and a second camera 5.

[0077] The first camera 4 is arranged at the top front position in the shell, with the lens of the first camera facing downward to the back of the hand; the second camera 5 is arranged at the bottom right position in the shell, with the lens of the second camera facing upward to the finger position.

[0078] The back of the hand position and the finger position here refer to the position of the back of the hand and the fingers when the palm is fully inserted into the housing 1.

[0079] There are four light source panels, which are defined as a back-of-hand reflection light source panel 6, a back-of-hand transmission light source panel 7, a finger reflection light source panel 8, and a finger transmission light source panel 9; the four light source panels are arranged in the housing as follows:

[0080] The hand back reflecting light source plate 6 is located below the first camera, and a first filter 10 is installed at the lens of the first camera 4 .

[0081] The finger reflecting light source plate 8 is located above the second camera 5 , and a second filter 11 is installed at the lens of the second camera 5 .

[0082] The back-of-hand transmission light source panel 7 is located in the middle of the front side of the shell, and the installation height of the back-of-hand transmission light source panel 7 is lower than the height of the opening 3, so that when the hand is fully inserted into the shell, the back-of-hand transmission light source panel 7 is located below the hand.

[0083] The finger-transmitting light source plate 7 is aligned with the first camera 4 at the upper and lower positions.

[0084] The finger transmission light source board 9 is located in the middle of the rear side of the shell, and the installation height of the finger transmission light source board 9 is higher than the height of the opening 3, so that when the hand is fully inserted into the shell, the finger transmission light source board 9 is located above the hand.

[0085] The finger transmits the light source plate 9 and is aligned with the second camera 5 at the upper and lower positions.

[0086] The four light source boards are connected to the light source core board 12 through lines respectively, and the light source core board is connected to the host computer 2 through lines.

[0087] The first camera 4 and the second camera 5 are connected to the host computer 2 through a line. The first camera 4 and the second camera 5 are both CCD cameras, and 850nm filters (ie, the first filter 10 and the second filter 11) are installed in front of the lens.

[0088] The vein image acquisition device is equipped with an adaptive dimming algorithm for vein acquisition. During the acquisition process, dimming is performed first and then the image is acquired. By deploying the host computer 2 to select the acquisition method, i.e., projection acquisition or reflection acquisition, the host computer 2 sends instructions to the light source core board 12 through the serial port. The light source core board 12 sets the LED light intensity according to the adaptive dimming algorithm. Each time the brightness is set, the host computer will record the number of vein pixels of this brightness. After the dimming algorithm is completed, the host computer selects the light intensity with the largest number of vein pixels. After the dimming is completed, the acquisition instruction is sent through the host computer, and the vein image can be captured at the same time.

[0089] Transmission and reflection are two mainstream methods in the field of venous imaging, each with its own advantages. Transmission has the advantages of fast acquisition speed and can be completed without contact with the hands. The veins acquired by the transmission acquisition method are clearer.

[0090] Specifically, the adaptive dimming algorithm program includes a hand back vein transmission dimming algorithm program, a finger vein transmission dimming algorithm program, a hand back vein reflection dimming algorithm program and a finger vein reflection dimming algorithm program.

[0091] like Figure 3 As shown, when the hand back vein transmission dimming algorithm program is executed, the processing is performed according to the following steps:

[0092] Step I.1. Coarse adjustment phase.

[0093] Initialize the brightness matrix. At this stage, the brightness of the back-of-hand transmitted light source panel is adjusted as a whole.

[0094] In order to adjust the brightness value of each LED light on the back of the hand transmission light source board, it is preset according to the structure of the hand. The back of the hand transmission light source board has a 6×6 LED array, and the 36 LEDs are divided into an area according to each four adjacent LEDs, and a total of 9 areas are divided. The brightness values ​​of the four LEDs in the same area are the same, and a 3×3 LED brightness matrix Brt is preset;

[0095] Where Brt[i][j] is the brightness of region [i][j]:

[0096]

[0097] Among them, x is a variable, and when dimming is performed, the value of x ranges from 30 to 250 with a step size of 5; among them, i and j represent the i+1th row and j+1th column areas of the LED array on the back-of-hand transmission light source board, respectively, and i, j = 0, 1, 2.

[0098] Step I.2. Set the traversal range of x.

[0099] Set the initial value of x to 30, traverse to 250 and end with a step length of 5.

[0100] In step I.3.x, within the traversal range, use the VU-Net algorithm to calculate the number of vein pixels N for each x value, store the N obtained for each x and the LED brightness matrix Brt in a list, and add 5 to the value of x.

[0101] Step I.4.x traversal ends;

[0102] Take out the LED brightness matrix Brt corresponding to the maximum N value in the list as the optimal brightness matrix in the coarse adjustment stage.

[0103] Step I.5. Fine-tuning phase.

[0104] In this stage, the brightness of the nine areas is adjusted in turn; the brightness threshold of the i-th row and j-th column area of ​​the LED light source board is set to {Brt[i][j]-3,Brt[i][j]+3}, where Brt is the optimal brightness matrix obtained in step I.4.

[0105] Step I.6. First, adjust the LED brightness of the first area [0,0] within the brightness threshold to obtain the optimal brightness, and then calculate the optimal brightness of the remaining eight areas in turn to complete the fine-tuning process.

[0106] Step I.7. Adjust the nine areas in sequence to obtain the final brightness matrix Brt of the light source panel.

[0107] like Figure 4 As shown, when the finger vein transmission dimming algorithm program is executed, the processing is performed according to the following steps:

[0108] When the finger vein transmission dimming algorithm program is executed, the processing is carried out according to the following steps:

[0109] Step II.1. Coarse adjustment phase.

[0110] Initialize the brightness matrix. At this stage, the brightness of the light source panel is adjusted simultaneously.

[0111] The brightness value of each LED light on the finger transmission light source board is preset according to the structure of the hand. The finger transmission light source board has a 1×7 LED array. Each LED is used as a region, and a 1×7 LED brightness matrix Brt is preset, where Brt[i][j] is the brightness of region [i][j]:

[0112] Brt=[x-9,x-6,x-3,x,x+3,x+6,x+9];

[0113] Among them, x ranges from 30 to 250 with a step size of 5; i and j represent the i+1th row and j+1th column areas of the LED array on the back of the hand transmission light source board, respectively, i=0, j=0,1,2,3,4,5,6.

[0114] Step II.2. Set the traversal range of x.

[0115] Set the initial value of x to 10, traverse to 240 and end with a step size of 10.

[0116] Step II.3.x: In the traversal range, use the VU-Net algorithm to calculate the number of vein pixels N for each x value, store the N and LED brightness matrix Brt obtained for each x in a list, and add 10 to the value of x.

[0117] Step II.4.x traversal ends.

[0118] The LED brightness matrix Brt corresponding to the maximum N value in the list is taken out as the optimal brightness value in the coarse adjustment stage.

[0119] Step II.5. Fine-tuning phase.

[0120] In this stage, the brightness of the LED lights in the 7 areas is adjusted in turn; the brightness threshold of the LED light source board is set to [Brt[i][j]-5,Brt[i][j]+5], where Brt is the optimal brightness matrix obtained in step II.4.

[0121] Step II.6. Traverse the brightness of the LEDs in the first area within the threshold to obtain the optimal brightness, and then calculate the optimal brightness of the remaining six LEDs in turn to complete the fine-tuning process.

[0122] Step II.7. Take the matrix with the largest number of vein pixels as the final brightness matrix Brt in the fine-tuning stage.

[0123] like Figure 5 As shown, when the hand back vein reflection dimming algorithm and the finger vein reflection dimming algorithm programs are executed, there is no need to control the LEDs individually, only their overall brightness needs to be adjusted.

[0124] The LED arrangement on the hand back reflection light source board and the finger reflection light source board is the same, that is, a group of 2×2 LED arrays are respectively arranged on the opposite sides of the corresponding light source board, and the middle of the light source board is vacant, such as Fig. 20 shown.

[0125] The hand vein reflection dimming algorithm and finger vein reflection dimming algorithm are specifically processed as follows:

[0126] Step III.1. Rough adjustment stage;

[0127] Set the traversal range of x, set the initial value of x to 10, traverse to 255 and end, with a step length of 10;

[0128] Step III.2.x: In the traversal range, use the VU-Net algorithm to calculate the number of vein pixels N for each x value, store the N and LED brightness matrix Brt obtained for each x into a list, and add 10 to the value of x;

[0129] Step III.3.x traversal ends;

[0130] Take out the LED brightness Brt corresponding to the maximum N value in the list as the optimal brightness value in the coarse adjustment stage;

[0131] Step III.4. Fine-tuning phase;

[0132] Set the brightness threshold of the LED light source board to [Brt-5,Brt+5]; where Brt is the optimal brightness obtained in step III.4;

[0133] Step III.5. Traverse the brightness of the LED within the threshold value to obtain the optimal brightness and complete the fine-tuning process.

[0134] The present invention designs a multimodal acquisition device for hand back veins and finger veins, which can perform two acquisition modes, namely transmission and reflection, using an adaptive dimming method. The transmission and reflection acquisition modes correspond to the above-mentioned transmission and reflection adaptive dimming.

[0135] The way of collecting with the camera after transmission dimming is the transmission collection method. At this time, the reflected LED is not bright. The way of collecting with the camera after reflection dimming is the reflection dimming method. At this time, the transmitted LED is not bright.

[0136] The present invention adopts adaptive dimming, which can adaptively adjust the LED brightness of each area according to the collected back of the hand, so as to capture as much vein information as possible, and the captured vein information is more comprehensive. In the process of identity authentication, more vein features can be extracted, which will be easier to distinguish from other categories, which is conducive to increasing the recognition accuracy.

[0137] Example 2

[0138] The second embodiment describes an identity recognition method based on multi-modal hand veins. The identity recognition method based on multi-modal hand veins is based on the identity recognition device based on multi-modal hand veins in the first embodiment.

[0139] like Figure 6 As shown, the identity recognition method based on multimodal hand veins includes the following steps:

[0140] Step 1. Place the palm to be collected on the vein image acquisition device, select the acquisition method and perform adaptive dimming. Set the optimal LED brightness through coarse adjustment and fine adjustment, and collect the back of the hand and finger modal data at the same time.

[0141] For the specific adjustment process, reference may be made to the adaptive dimming process in the above-mentioned embodiment 1.

[0142] Step 2. First, the vein dataset collected in step 1 is sent to the VU-Net network for vein segmentation. Then, the minimum rectangle containing all vein information is found, and the vertex information of the minimum rectangle is recorded: a(x1, y1), b(x2, y2), c(x3, y3), d(x4, y4). Finally, the vertex position of the minimum rectangle is found on the original image, and the rectangular area is cropped to obtain the vein ROI image.

[0143] like Figure 7 As shown, in order to obtain the number of vein features (i.e., used in the adaptive dimming algorithm in the above embodiment 1) and obtain the ROI of the image during image preprocessing, the present invention designs a VU-Net algorithm. The VU-Net algorithm draws on the idea of ​​U-Net to improve the network. The algorithm deployed in the acquisition process should ensure its lightweight and real-time performance.

[0144] Specifically, the following improvements were made in the VU-Net network: the downsampling branch and upsampling branch used 7-layer convolutional layers and 6-layer convolutional layers for feature extraction, respectively. The downsampling branch added a channel attention mechanism to help better extract features. The upsampling branch adjusted SENet, that is, the compression operation in SENet was changed to expansion, as the attention mechanism DeSENet of the upsampling branch. The improvement of the VU-Net network is mainly the addition of the attention mechanism (Senet and DeSenet). The network structure of VU-Net is reused in two tasks, one for adaptive dimming and the other for ROI extraction.

[0145] The improved VU-Net network processes the segmented image for the next step. The number of vein pixels can be directly calculated in the adaptive dimming algorithm. In the ROI extraction process, subsequent operations can realize ROI extraction. That is, the VU-Net network can be used for calculating the number of vein pixels and extracting ROI images in the image preprocessing process.

[0146] After passing through the VU-Net network, the number of pixels belonging to the vein can be calculated based on the output results of the network.

[0147] The process of vein ROI extraction is to first input the vein image to be extracted into the VU-Net network. After network calculation, the vein segmentation image will be obtained. We need to cut out the ROI image based on this, find the rectangle that just contains all the vein information, calculate the vertex position of the rectangle, and then cut it to get the ROI image.

[0148] The image input size of the VU-Net network is adjusted to 256×256. The processing flow of the VU-Net network is as follows:

[0149] In the downsampling part, the convolution kernel with a size of 3×3×64 and a step size of 1 is first used to obtain a feature map with a size of 256×256×64. After that, the image size is unchanged after passing SENet. Then, the first downsampling convolution is performed with a convolution kernel size of 3×3×128 and a step size of 2 to obtain a feature map with a size of 128×128×128. After that, the convolution kernel size is 3×3×128 and a step size of 1 and SENet are performed to obtain a feature map with a size of 128×128×128. After that, the image size is unchanged after passing the second downsampling convolution. Convolution, the convolution kernel size is 3×3×256, the step size is 2, and the size of the feature map is 64×64×256. After the convolution kernel size is 3×3×256 and the step size is 1, and the SENet, the image size remains unchanged; then after the third downsampling convolution, the convolution kernel size is 3×3×512, the step size is 2, the size of the feature map is 32×32×512, and then the convolution kernel size is 3×3×512 and the step size is 1, and the image size remains unchanged.

[0150] In the upsampling part, bilinear interpolation is first used to change the dimension to 64×64×512, and then a 3×3×256 convolution kernel is used to obtain 64×64×256, which is then concatenated with the feature map of the second downsampling to obtain a feature map of size 64×64×512, and then a 3×3×256 convolution kernel is used to obtain a size of 64×64×256, and then DeSENet is used to keep the image size unchanged; then a second upsampling convolution is performed, bilinear interpolation is used, the dimension becomes 128×128×256, and then a 3×3×128 convolution kernel is used to obtain 128×128×128, which is then concatenated with the feature map of the first downsampling to obtain a feature map of size 128×128×256, and then pass through the convolution kernel of 3×3×128 to get a size of 128×128×128, and then pass through DeSENet, the image size remains unchanged; then after the third upsampling convolution, using bilinear interpolation, the dimension becomes 256×256×128, and then pass through the convolution kernel of 3×3×64 to get 256×256×64, and concatenate it with the feature map after the first convolution, the size of the feature map is 256×256×128, and then pass through the convolution kernel of 3×3×64 to get 256×256×64, and then pass through DeSENet, the image size remains unchanged; then pass through the convolution kernel of 3×3×2 to get the segmented image.

[0151] During the dimming process, VU-Net processes the segmented image to obtain the number of vein pixels N. During the image preprocessing process, VU-Net segments the collected vein image and obtains the ROI image after further processing.

[0152] like Figure 8 and Fig. 9 The network structure diagrams of SENet and DeSENet are shown respectively.

[0153] like Figure 8 As shown, the processing flow of SENet is as follows:

[0154] SENet first compresses the feature map W×H×C into a global feature vector of 1×1×C through global average pooling.

[0155]

[0156] Among them, sq means squeeze, F sq (f c ) represents the squeezing operation on the feature map fc, f c (i,j) represents the value of the i-th row and j-th column in the c-th channel of the feature map f.

[0157] Then it passes through a two-layer fully connected network. The first layer compresses the C-dimensional vector into C / r dimensions, where r is 4, and uses the ReLU activation function.

[0158] S2=δ(W1S1) (2)

[0159] Among them, δ represents the RELU activation function, and W1∈R in SENet C×C / r represents a fully connected layer that reduces the dimension, r is an attenuation factor, and the present invention sets it to 4.

[0160] The second layer expands the C / r-dimensional vector back to C dimension and uses the Sigmoid activation function to generate a 1×1×C weight vector.

[0161] S3=σ(W2S2) (3)

[0162] Among them, σ represents the sigmoid activation function, and W2∈R in SENet C / r×C It represents a fully connected layer with increased dimension, and the attenuation factor r is set to 4 in the present invention.

[0163] Finally, the weight vector is multiplied by the original feature map channel by channel to obtain a weighted feature map with a dimension of W×H×C.

[0164] like Fig. 9 As shown, the processing flow of DeSENet is as follows:

[0165] DeSENet first compresses the feature map W×H×C into a global feature vector of 1×1×C through global average pooling.

[0166]

[0167] Among them, sq means squeeze, F sq (f c ) represents the squeezing operation on the feature map fc, f c (i,j) represents the value of the i-th row and j-th column in the c-th channel of the feature map f.

[0168] Then it passes through a two-layer fully connected network. The first layer expands the C-dimensional vector to C*r dimensions, where r is 4, and uses the ReLU activation function.

[0169] S2'=δ(W1S1') (2)

[0170] Among them, δ represents the RELU activation function, and W1∈R C×C*r represents a fully connected layer with increased dimension, r is an attenuation factor, which is set to 4 in the present invention.

[0171] The second layer expands the C / r-dimensional vector back to C dimension and uses the Sigmoid activation function to generate a 1×1×C weight vector.

[0172] S3'=σ(W2S2') (3)

[0173] Where σ represents the sigmoid activation function, W2∈R C*r×C represents a fully connected layer that reduces the dimension, r is an attenuation factor, and the present invention sets it to 4.

[0174] Finally, the weight vector is multiplied by the original feature map channel by channel to obtain a weighted feature map with a dimension of W×H×C.

[0175] like Fig.10 The step of extracting a vein ROI is shown. Before training, the ROI is extracted from the acquired vein image.

[0176] The vein data set collected by the device is sent to VU-Net to obtain the segmented vein image. The largest rectangular area is taken and the coordinates of the four vertices are recorded. Then, the original image is cropped to obtain the vein ROI image.

[0177] The extracted ROI image is sent to the identity recognition network, which is the CVit multimodal network with a dual-branch structure built below. In the recognition network part, a multimodal network is designed for the multimodal data of the same subject. The idea of ​​cross-attention in transformer is used to extract and highlight the related features between the two modalities to improve the recognition accuracy of the network.

[0178] Step 3. Build a CVit multimodal network with a dual-branch structure, such as Fig.11 shown.

[0179] The constructed CVit multimodal network includes feature extraction network, fusion network, feature reconstruction network and classification network.

[0180] The feature extraction part uses the VGG16 network, such as Fig.12 As shown. The first 13 layers of the VGG16 network are used to extract the features of the back of the hand vein and the finger vein. As shown in the figure above, the convolution layer is 3×3 in size, the convolution step is 1, the activation function is ReLU, and the pooling layer is 2×2 in size. This part can fully extract the features between each modality.

[0181] like Fig.12 As shown, the image processing flow in the feature extraction network is as follows:

[0182] First, the input image is converted to 224×224×3, and after two 3×3×64 convolution layers, the size of the feature map is 224×224×64. After the maximum pooling layer, the height and width of the feature map are halved, and the size of the feature map becomes 112×112×64; then after two 3×3×128 convolution kernels, the size of the feature map becomes 112×112×128, and after the maximum pooling layer, the size becomes 56×56×128; then after three 3 ×3×256 convolutional layer, the size becomes 56×56×256, and then passes through the maximum pooling layer, the size becomes 28×28×256; then passes through three 3×3×512 convolutional layers, the size becomes 28×28×512, and then passes through the maximum pooling layer, the size becomes 14×14×512, and then passes through three 3×3×512 convolutional layers, the size becomes 14×14×512, and finally passes through the maximum pooling layer to obtain a feature map of size 7×7×512.

[0183] The dorsal vein ROI image is input into the feature extraction network to obtain a feature map f with a size of 7×7×512 DHV ; Input the finger vein ROI image into the feature extraction network, and also obtain a feature map f with a size of 7×7×512 FV .

[0184] like Fig.11 As shown in the figure, a dual-branch network structure is used in the feature fusion part. The first branch and the second branch network are used for feature extraction. The SENet channel attention mechanism in the first branch can help the network learn the correlation between the features of each modality more effectively. SENet can enhance the correlation of features within the modality. The SE module can improve the global learning ability of the model, mine low-level spatial features and high-level semantic features in the channel dimension, and improve network performance. The Cross-Attention mutual attention mechanism in the second branch can learn the correlation between the two modalities. The extracted features are sent to the feature reconstruction network for feature fusion, and finally the classification results are obtained through two fully connected layers.

[0185] The feature map output by the feature extraction network is input into the first branch of the fusion network, such as Fig.13 As shown in the figure, the input feature map is subjected to global average pooling to obtain a one-dimensional vector S1″ of dimension 1×1×512, and then S1″ is input into a fully connected layer with dimension 128 to obtain a vector S2″ of 1×1×512, and then a fully connected layer of 512 dimensions is used to obtain a vector S3″ of 1×1×512. Finally, S3″ is multiplied by the input feature to obtain the output feature map; the dorsal vein feature map f DHV After the feature extraction of the first branch, the feature map f is obtained DHV_SE , finger vein feature map f FVAfter the feature extraction of the first branch, the feature map f is obtained FV_SE .

[0186] The feature map output by the feature extraction network is input into the second branch of the fusion network, such as Fig.14 As shown, f DHV and f FV Divide the space into one-dimensional vectors, and get a vector f of size 49×512 DHV_token1 and f FV_token1 , the resulting vector f DHV_token1 and f FV_token1 , respectively input into the Transformer network, and swap the Q in the two Transformer networks to generate a vector f of dimension 49×512 DHV_token2 and f FV_token2 , and then restore the vector to 7×7×512 size and add it to f DHV and f FV Add them together to get f DHV_ViT and f FV_ViT . Get the vector f DHV_token2 and f FV_token2 The process is as follows:

[0187] Q DHV =f DHV_token1 W q1 ;

[0188] K DHV =f DHV_token1 W k1 ;

[0189] V DHV =f DHV_token1 W v1 ;

[0190] Q FV =f FV_token1 W q2 ;

[0191] K FV =f FV_token1 W k2 ;

[0192] V FV =f FV_token1 W v2 ;

[0193] Where W q1 , W k1 , W v1 , W q2 , W k2 , W v2 is a learnable parameter, which is matrix multiplied with the vector to obtain Q DHV, K DHV 、V DHV , Q FV , K FV 、V FV ; Query (Query, Q), key (Key, K) and value (Value, V) are the key components of the attention mechanism in Transformer. DHV , K DHV 、V DHV They are the query, key and value of the hand back vein feature; Q FV , K FV 、V FV They are the Query, Key and Value of the finger vein feature respectively.

[0194]

[0195] Among them, d k It's Q DHV , K DHV , Q FV , K FV The vector dimension of Cross-Attention() represents the DHV_token2 and f FV_token2 Perform mutual attention calculation.

[0196] The feature map f DHV_SE 、f FV_SE 、f DHV_ViT 、f FV_ViT Perform concat splicing to obtain the spliced ​​feature map F fusion , the dimension size is 7×7×2048, and then the concatenated feature map F fusion After the feature reconstruction network, features are further extracted, and finally the identity recognition result is obtained through two fully connected layers.

[0197] F fusion =Concat(f DHV_SE ,f FV_SE ,f DHV_ViT ,f FV_ViT );

[0198] Among them, Concat represents a concatenation operation on the channel dimension of the feature map matrix.

[0199] The present invention proposes a new fusion method based on the idea of ​​cross-attention in transformer, which improves the accuracy of identity authentication and enhances the correlation between inter-modal features. By processing image data using mobileVit and improving the mobileVit network, a cross-attention mechanism is added to provide inter-modal correlation features and provide richer feature information for classification tasks.

[0200] like Fig.15 As shown, the feature reconstruction network includes point convolution, ReLU activation function and convolution.

[0201] The point convolution includes the first point convolution, the second point convolution, and there are three ReLU activation functions, namely the first, second, and third ReLU activation functions; the signal processing flow in the feature reconstruction network is:

[0202] The signal first passes through the first point convolution, then the first ReLU activation function, then convolution, then the second ReLU activation function, then the second point convolution, and then the third ReLU activation function before output.

[0203] The feature map F has a dimension of 7×7×2048 fusion After a 1×1×1024 point convolution, the feature map dimension is 7×7×1024, and then a 3×3×1024 convolution kernel is performed to obtain a feature map dimension of 7×7×1024. Finally, after a 1×1×512 point convolution, the number of parameters of the fully connected layer is reduced, and the dimension of the final feature map F is 7×7×512.

[0204] Finally, the fused features are sent to the classification network. The classification network consists of three fully connected layers. The three fully connected layers contain 2048 / 1024 and n nodes respectively. The n of the last fully connected layer needs to be determined according to the category of the input network. The first two layers use the relu activation function, and the last layer uses the softmax activation function. The classification result is obtained after passing through the classification network.

[0205] During the training process of the CVit multimodal network, the network parameter batch_size is set to 16, the learning rate is set to 0.0001, the cross entropy loss function is used, and the training is performed for 100 rounds. After the CVit multimodal network is trained, it is deployed.

[0206] Before identity recognition, the vein features of the registered person are first extracted.

[0207] First, place the palm of the person to be registered on the vein image acquisition device. The vein image acquisition device will collect the finger vein images and hand vein images of the registrant, extract the ROI image and send it to the deployed CVit multimodal network. The network outputs a 7×7×1024 feature map and saves it in the local database.

[0208] Secondly, the vein image of the person to be identified is obtained and the identity is identified. Specifically, the palm of the person to be registered is placed on the vein image acquisition device, which will collect the finger vein image and the back vein image of the person to be identified; the ROI of the image to be identified is extracted and sent to the deployed CVit multimodal network to obtain a 7×7×1024 feature map, which is compared with the features in the local database, and the classification with the highest similarity is output as the identity recognition result.

[0209] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. An identity recognition device based on multimodal hand veins, comprising a vein image acquisition device and a computer device; characterized in that: The vein image acquisition device includes a housing, a camera, a host computer, a light source board and a light source core board; An opening is provided in the middle of the front side of the shell for inserting a hand; There are two cameras, defined as the first camera and the second camera; The first camera is arranged at the top of the front side of the housing, and the lens of the first camera is directed downward to the back of the hand; the second camera is arranged at the bottom of the right side of the housing, and the lens of the second camera is directed upward to the finger position; There are four light source panels, which are defined as a back-of-hand reflection light source panel, a back-of-hand transmission light source panel, a finger reflection light source panel, and a finger transmission light source panel; the four light source panels are arranged in the housing as follows: The hand back reflection light source plate is located below the first camera, and a filter is installed at the lens of the first camera; The finger reflection light source plate is located above the second camera, and a filter is installed at the lens of the second camera; The hand back transmission light source panel is located in the middle of the front side of the housing, and the installation height of the hand back transmission light source panel is lower than the height of the opening, so that when the hand is fully inserted into the housing, the hand back transmission light source panel is located below the hand; The finger transmission light source plate is aligned with the first camera at the upper and lower positions; The finger-transmitting light source board is located in the middle of the rear side of the housing, and the installation height of the finger-transmitting light source board is higher than the height of the opening, so that when the hand is fully inserted into the housing, the finger-transmitting light source board is located above the hand; The finger transmits the light source plate and aligns it with the second camera at the upper and lower positions; The four light source boards are connected to the light source core board through lines respectively, and the light source core board is connected to the host computer through lines; The first camera and the second camera are connected to the host computer through lines.

2. The identity recognition device based on multimodal hand vein according to claim 1, characterized in that: The host computer stores an adaptive dimming algorithm program, including a hand back vein transmission dimming algorithm program, a finger vein transmission dimming algorithm program, a hand back vein reflection dimming algorithm program, and a finger vein reflection dimming algorithm program; When the hand vein transmission dimming algorithm is executed, the following steps are followed: Step I.

1. Rough adjustment stage; Initialize the brightness matrix. At this stage, the brightness of the back-of-hand transmission light source panel is adjusted simultaneously; In order to adjust the brightness value of each LED light on the back of the hand transmission light source board, it is preset according to the structure of the hand. The back of the hand transmission light source board has a 6×6 LED array, and the 36 LEDs are divided into an area according to each four adjacent LEDs, and a total of 9 areas are divided. The brightness values ​​of the four LEDs in the same area are the same, and a 3×3 LED brightness matrix Brt is preset; Where Brt[i][j] is the brightness of region [i][j]: Where x is a variable, and when dimming is performed, x takes values ​​from 30 to 250, with a step length of 5; where i and j represent the i+1th row and j+1th column area of ​​the LED array on the back-of-hand transmission light source board, respectively, and i, j = 0, 1, 2; Step I.

2. Set the traversal range of x; Set the initial value of x to 30, traverse to 250 and end with a step length of 5; Step I.3.x: In the traversal range, use the VU-Net algorithm to calculate the number of vein pixels N for each x value, store the N and LED brightness matrix Brt obtained for each x into a list, and add 5 to the value of x; Step I.4.x traversal ends; Take out the LED brightness matrix Brt corresponding to the maximum N value in the list as the optimal brightness matrix in the coarse adjustment stage; Step I.

5. Fine-tuning phase; In this stage, the brightness of the nine areas is adjusted in turn; the brightness threshold of the area of ​​the i-th row and j-th column of the LED light source board is set to {Brt[i][j]-3,Brt[i][j]+3}, where Brt is the optimal brightness matrix obtained in step I.4; Step I.

6. First, adjust the LED brightness of the first area [0,0] within the brightness threshold to obtain the optimal brightness, and then calculate the optimal brightness of the remaining eight areas in turn to complete the fine-tuning process; Step I.

7. Adjust the nine areas in sequence to obtain the final brightness matrix Brt of the light source panel; When the finger vein transmission dimming algorithm program is executed, the processing is carried out according to the following steps: Step II.

1. Rough adjustment stage; Initialize the brightness matrix. At this stage, the brightness of the light source panel is adjusted simultaneously; The brightness value of each LED light on the finger transmission light source board is preset according to the structure of the hand. The finger transmission light source board has a 1×7 LED array. Each LED is used as a region, and a 1×7 LED brightness matrix Brt is preset, where Brt[i][j] is the brightness of region [i][j]: Brt=[x-9,x-6,x-3,x,x+3,x+6,x+9]; Among them, x ranges from 30 to 250 with a step length of 5; i and j represent the i+1th row and j+1th column area of ​​the LED array on the back of the hand transmission light source board, i=0, j=0,1,2,3,4,5,6; Step II.

2. Set the traversal range of x; Set the initial value of x to 10, traverse to 240 and end with a step length of 10; Step II.3.x: In the traversal range, use the VU-Net algorithm to calculate the number of vein pixels N for each x value, store the N obtained for each x and the LED brightness matrix Brt in a list, and add 10 to the value of x; Step II.4.x traversal ends; Take out the LED brightness matrix Brt corresponding to the maximum N value in the list as the optimal brightness value in the coarse adjustment stage; Step II.

5. Fine-tuning phase; In this stage, the brightness of the LED lights in the seven areas is adjusted in turn; the brightness threshold of the LED light source board is set to [Brt[i][j]-5,Brt[i][j]+5], where Brt is the optimal brightness matrix obtained in step II.4; Step II.

6. Traverse the brightness of the LED in the first area within the threshold to obtain the optimal brightness, and then calculate the optimal brightness of the remaining six LEDs in turn to complete the fine-tuning process; Step II.

7. Take the matrix with the largest number of vein pixels as the final brightness matrix Brt in the fine-tuning stage; When the hand vein reflection dimming algorithm and finger vein reflection dimming algorithm are executed, there is no need to control the LEDs individually, only the overall brightness needs to be adjusted. The specific steps are as follows: Step III.

1. Rough adjustment stage; Set the traversal range of x, set the initial value of x to 10, traverse to 255 and end, with a step length of 10; Step III.2.x: In the traversal range, use the VU-Net algorithm to calculate the number of vein pixels N for each x value, store the N and LED brightness matrix Brt obtained for each x into a list, and add 10 to the value of x; Step III.3.x traversal ends; Take out the LED brightness Brt corresponding to the maximum N value in the list as the optimal brightness value in the coarse adjustment stage; Step III.

4. Fine-tuning phase; Set the brightness threshold of the LED light source board to [Brt-5,Brt+5]; where Brt is the optimal brightness obtained in step III.3; Step III.

5. Traverse the brightness of the LED within the threshold value to obtain the optimal brightness and complete the fine-tuning process.

3. A method for identifying a person based on multimodal hand veins, based on the device for identifying a person based on multimodal hand veins as claimed in any one of claims 1 to 2, characterized in that: The multi-modal hand vein based identity recognition method comprises the following steps: Step 1. Place the palm to be collected on the vein image acquisition device, select the acquisition method and perform adaptive dimming. Set the optimal LED brightness through coarse adjustment and fine adjustment, and collect the back of the hand and finger modal data at the same time; Step 2. First, the vein data set collected in step 1 is sent to the VU-Net network for vein segmentation, and then the minimum rectangle containing all vein information is found, and the vertex information of the minimum rectangle is recorded. Finally, the vertex position of the minimum rectangle is found on the original image, and the rectangular area is cut out to obtain the vein ROI image; Step 3. Build a CVit multimodal network with a dual-branch structure; The built CVit multimodal network includes feature extraction network, fusion network, feature reconstruction network and classification network; The fusion network includes a first branch and a second branch structure; The SENet channel attention mechanism in the first branch helps the network learn the correlation between the features of each modality more effectively, and the cross-attention mechanism in the second branch learns the correlation between the two modalities; The signal processing flow in the CVit multimodal network is as follows: The hand back vein image and finger vein image are input into the feature extraction network to obtain the hand back vein feature map f DHV And finger vein feature map f FV ; The feature map output by the feature extraction network is input into the first branch of the fusion network. The processing flow of the first branch is as follows: The input feature map is subjected to global average pooling to obtain a one-dimensional vector S1″ of dimension 1×1×512, and then S1″ is input into a fully connected layer of dimension 128 to obtain a vector S2″ of dimension 1×1×512, and then passed through a 512-dimensional fully connected layer to obtain a vector S3″ of dimension 1×1×512. Finally, S3″ is multiplied by the input feature to obtain the output feature map; Characteristics of hand veins DHV After the feature extraction of the first branch, the feature map f is obtained DHV_SE , finger vein feature map f FV After the feature extraction of the first branch, the feature map f is obtained FV_SE ; The feature map output by the feature extraction network is input into the second branch of the fusion network. The processing flow of the second branch is as follows: f DHV and f FV Divide the space into one-dimensional vectors, and get a vector f of size 49×512 DHV_token1 and f FV_token1 , the vector f DHV_token1 and f FV_token1 Input them into a Transformer network respectively, and swap the Q in the two Transformer networks to generate a vector f of dimension 49×512 DHV_token2 and f FV_token2 , and then restore the vector to 7×7×512 size and add it to f DHV and f FV Add them together to get f DHV_ViT and f FV_ViT ; The feature map f DHV_SE 、f FV_SE 、f DHV_ViT 、f FV_ViT Concatenate to get the feature map F fusion , and then through the feature reconstruction network, further features are extracted, and finally through two fully connected layers to obtain the identity recognition result; Among them, the VU-Net network and the CVit multimodal network are both deployed on computer devices.

4. The method for identifying a person based on multimodal hand veins according to claim 3, characterized in that: The VU-Net network has been improved as follows: In the improved VU-Net network, the downsampling branch and upsampling branch use 7-layer convolutional layers and 6-layer convolutional layers for feature extraction respectively; the downsampling branch adds a channel attention mechanism to help better extract features, and the upsampling branch improves SENet by changing the compression operation in SENet to expansion, which serves as the attention mechanism of the upsampling branch, DeSENet.

5. The method for identifying a person based on multimodal hand veins according to claim 4, characterized in that: The image input size of the VU-Net network is adjusted to 256×256. The processing flow of the VU-Net network is as follows: In the downsampling part, the convolution kernel with a size of 3×3×64 and a step size of 1 is first used to obtain a feature map with a size of 256×256×64. After that, the image size is unchanged after passing SENet. Then, the first downsampling convolution is performed with a convolution kernel size of 3×3×128 and a step size of 2 to obtain a feature map with a size of 128×128×128. After that, the convolution kernel size is 3×3×128 and a step size of 1 and SENet are performed to obtain a feature map with a size of 128×128×128. After that, the image size is unchanged after passing the second downsampling convolution. Convolution, the convolution kernel size is 3×3×256, the step size is 2, and the size of the feature map is 64×64×256. After the convolution kernel size is 3×3×256 and the step size is 1, and the image size remains unchanged; then after the third downsampling convolution, the convolution kernel size is 3×3×512 and the step size is 2, the size of the feature map is 32×32×512, and then after the convolution kernel size is 3×3×512 and the step size is 1, and the image size remains unchanged; In the upsampling part, bilinear interpolation is first used to change the dimension to 64×64×512, and then a 3×3×256 convolution kernel is used to obtain 64×64×256, which is then concatenated with the feature map of the second downsampling to obtain a feature map of size 64×64×512, and then a 3×3×256 convolution kernel is used to obtain a size of 64×64×256, and then DeSENet is used to keep the image size unchanged; then a second upsampling convolution is performed, bilinear interpolation is used, the dimension becomes 128×128×256, and then a 3×3×128 convolution kernel is used to obtain 128×128×128, which is then concatenated with the feature map of the first downsampling. , the size of the feature map is 128×128×256, and then it passes through the convolution kernel of 3×3×128 to get a size of 128×128×128, and then passes through DeSENet, the image size remains unchanged; then after the third upsampling convolution, using bilinear interpolation, the dimension becomes 256×256×128, and then passes through the convolution kernel of 3×3×64 to get 256×256×64, and is concatenated with the feature map after the first convolution, the size of the feature map is 256×256×128, and then passes through the convolution kernel of 3×3×64 to get 256×256×64, and then passes through DeSENet, the image size remains unchanged; Then, after a 3×3×2 convolution kernel, the segmented image is obtained.

6. The method for identifying a person based on multimodal hand veins according to claim 3, characterized in that: In step 3, the feature extraction network uses the VGG16 model, and the first 13 layers of the network are used as the feature extraction part of the model. The image processing flow in the feature extraction network is as follows: First, the input image is converted to 224×224×3, and after two 3×3×64 convolution layers, the size of the feature map is 224×224×64. After the maximum pooling layer, the height and width of the feature map are halved, and the size of the feature map becomes 112×112×64; then after two 3×3×128 convolution kernels, the size of the feature map becomes 112×112×128, and after the maximum pooling layer, the size becomes 56×56×128; then after three 3 ×3×256 convolutional layer, the size becomes 56×56×256, and then after the maximum pooling layer, the size becomes 28×28×256; then after three 3×3×512 convolutional layers, the size becomes 28×28×512, and then after the maximum pooling layer, the size becomes 14×14×512, and then after three 3×3×512 convolutional layers, the size becomes 14×14×512, and finally after the maximum pooling layer, a feature map of size 7×7×512 is obtained; The dorsal vein ROI image is input into the feature extraction network to obtain a feature map f with a size of 7×7×512 DHV ; Input the finger vein ROI image into the feature extraction network, and also obtain a feature map f with a size of 7×7×512 FV .

7. The method for identifying a person based on multimodal hand veins according to claim 3, characterized in that: In step 3, we get the vector f DHV_token2 and f FV_token2 The process is as follows: Q DHV =f DHV_token1 W q1 ; K DHV =f DHV_token1 W k1 ; V DHV =f DHV_token1 W v1 ; Q FV =f FV_token1 W q2 ; K FV =f FV_token1 W k2 ; V FV =f FV_token1 W v2 ; Where W q1 , W k1 , W v1 , W q2 , W k2 , W v2 is a learnable parameter, which is matrix multiplied with the vector to obtain Q DHV , K DHV 、V DHV , Q FV , K FV 、V FV ;Q DHV , K DHV 、V DHV They are the query, key and value of the hand back vein feature; Q FV , K FV 、V FV They are the Query, Key and Value of the finger vein feature; Among them, d k It's Q DHV , K DHV , Q FV , K FV The vector dimension of Cross-Attention() represents the DHV_token2 and f FV_token2 Perform mutual attention calculation.

8. The method for identifying a person based on multimodal hand veins according to claim 3, characterized in that: The feature reconstruction network includes point convolution, ReLU activation function and convolution; The point convolution includes the first point convolution, the second point convolution, and there are three ReLU activation functions, namely the first, second, and third ReLU activation functions; the signal processing flow in the feature reconstruction network is: The signal first passes through the first point convolution, then the first ReLU activation function, then convolution, then the second ReLU activation function, then the second point convolution, and then the third ReLU activation function before output.

9. The method for identifying a person based on multimodal hand veins according to claim 3, characterized in that: During the training process of the CVit multimodal network, the network parameter batch_size was set to 16, the learning rate was set to 0.0001, the cross entropy loss function was used, and training was performed for 100 rounds.

10. The method for identity recognition based on multimodal hand veins according to claim 3, characterized in that: First, place the palm of the person to be registered on the vein image acquisition device. The vein image acquisition device will collect the finger vein image and the back vein image of the registrant, extract the ROI image and send it to the deployed CVit multimodal network. The network outputs a 7×7×1024 feature map and saves it to the local database. Secondly, the vein image of the person to be identified is obtained. The palm of the person to be registered is placed on the vein image acquisition device, which will collect the finger vein images and hand vein images of the person to be identified. After the ROI is extracted from the image to be identified, it is sent to the deployed CVit multimodal network to obtain a 7×7×1024 feature map, which is compared with the features in the local database, and the classification category with the highest similarity is output as the identity recognition result.

Images