Biometric information identification method, device, equipment and medium

Abstract

The present invention provides a biometric information recognition method, apparatus, equipment and medium, including: obtaining a biometric information image of a target to be identified; performing feature extraction on the biometric information image to obtain single-modal color features and single-modal depth features at different levels; fusing the single-modal color features and single-modal depth features step by step to obtain multimodal features at each level; fusing the multimodal features with the single-modal color features and the single-modal depth features respectively to obtain cross-modal color convolution features and cross-modal depth convolution features; fusing the cross-modal color convolution features and the cross-modal depth convolution features to obtain a final feature vector; performing information recognition based on the final feature vector of the biometric information image. The present invention, through cross-modal processing, can not only extract more significant features, but also effectively capture cross-modal information from color images and depth images, which helps to detect more complete targets, but also improves recognition accuracy in biometric information recognition.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, equipment and medium for biometric information recognition. Background Art

[0002] With the continuous development of society and the continuous advancement of technology, the demand for secure personal identity authentication has become more urgent. Among the many identity recognition methods, the superiority of human biometrics is unmatched by other peripheral methods. Its security and reliability have been widely recognized. Among them, human biometrics include but are not limited to face, palm print, fingerprint, iris, vein and other biometric information recognition.

[0003] In actual application scenarios, two-dimensional biometric information recognition devices are affected by conditions such as lighting, background, posture, and clarity, which causes huge changes in image texture and reduces the recognition accuracy. However, three-dimensional biometric information will not be affected by texture changes under the same lighting, background, posture, clarity and other conditions, and can be applied to a wider range of scenarios.

[0004] However, in related technologies, when two-dimensional biometric information (first image) is fused with three-dimensional biometric information (second image), the weight ratios of the two-dimensional biometric information and the three-dimensional biometric information cannot be dynamically adjusted according to the current environment, so that the fused image cannot fully reflect the biometric information, thereby reducing the accuracy of biometric information recognition. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a biometric information recognition method, device, equipment and medium biometric information recognition, which is used to solve the problem in the prior art that the accuracy of biometric information recognition cannot be ensured when biometric information recognition is performed after the fusion of two-dimensional biometric information and three-dimensional biometric information.

[0006] To achieve the above and other related objectives, the present invention provides a method for biometric information recognition, comprising the following steps:

[0007] Acquire a biometric image of a target to be identified;

[0008] Extracting features from the bio-information image to obtain single-modal color features at different levels and single-modal depth features at different levels;

[0009] The unimodal color features and the unimodal depth features are gradually integrated to obtain multimodal features at each level;

[0010] The multimodal features are respectively fused with the single-modal color features and the single-modal depth features to obtain cross-modal color convolution features and cross-modal depth convolution features;

[0011] Fusing the cross-modal color convolutional features with the cross-modal depth convolutional features to obtain a final feature vector;

[0012] Information recognition is performed based on the final feature vector of the biological information image.

[0013] The present invention also provides a biometric information recognition device, comprising:

[0014] An acquisition module is used to acquire a biological information image of a target to be identified;

[0015] An extraction module is used to extract features from the biological information image to obtain single-modal color features of different levels and single-modal depth features of different levels;

[0016] A step-by-step fusion module, configured to fuse the single-modal color features and the single-modal depth features step-by-step to obtain multimodal features at each level;

[0017] A cross-modal generation module, configured to fuse the multi-modal features with the unimodal color features and the unimodal depth features to obtain cross-modal color convolution features and cross-modal depth convolution features;

[0018] A cross-modal interaction module, configured to fuse the cross-modal color convolutional features with the cross-modal depth convolutional features to obtain a final feature vector;

[0019] An information recognition module performs information recognition based on the final feature vector of the biological information image.

[0020] The present invention further provides an electronic device, comprising:

[0021] one or more processors; and

[0022] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the device to perform one or more of the methods described above.

[0023] The present invention also provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause a device to perform one or more of the methods described above.

[0024] As described above, the present invention provides a biometric information recognition method, device, equipment, and medium, which have the following beneficial effects:

[0025] By extracting features from the biometric image of the target to be identified, single-modal color features and single-modal depth features of different levels are obtained; the single-modal color features and single-modal depth features are fused step by step to obtain multimodal features at each level, and the complementarity between color features and depth features is improved by the fused multimodal features; the multimodal features are fused with the single-modal color features and the single-modal depth features respectively to obtain cross-modal color convolution features and cross-modal depth convolution features; the cross-modal color convolution features and the cross-modal depth convolution features are fused to obtain the final feature vector. Through the above method, not only the low-level details and high-level semantic information of the input image are effectively captured, but also the number of feature levels is avoided. The sharp increase in the overall network parameters caused by too many levels is avoided. At the same time, the difficulty of network training is reduced. The final feature vector determined by cross-modal interaction can extract more significant features, effectively capture cross-modal information from the first image and the second image, and help detect more complete targets. Information recognition based on the final feature vector of the biometric image also improves the accuracy of biometric recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of a process for biometric information recognition according to an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a multimodal feature fusion process in a biometric information recognition method provided by one embodiment of the present invention;

[0028] Figure 3 A schematic diagram of a cross-modal generation process in a biometric information recognition method provided by one embodiment of the present invention;

[0029] Figure 4 A schematic diagram of a cross-modal interaction process in a biometric information recognition method provided by one embodiment of the present invention;

[0030] Figure 5 A block diagram of the structure of a biometric information recognition device provided by one embodiment of the present invention;

[0031] Figure 6 A schematic diagram of a step-by-step multimodal feature fusion network structure provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the hardware structure of a terminal device provided in one embodiment of the present invention;

[0033] Figure 8 A schematic diagram of the hardware structure of a terminal device provided in one embodiment of the present invention.

[0034] Component number description

[0035] 500 Biometric Information Recognition Device

[0036] 501 Get Module

[0037] 502 Extract Module

[0038] 503 Step-by-step fusion module

[0039] 504 Cross-modal Generation Module

[0040] 505 Cross-modal interaction module

[0041] 506 Information Identification Module

[0042] 1100 Input Devices

[0043] 1101 First Processor

[0044] 1102 Output Devices

[0045] 1103 First Memory

[0046] 1104 Communication Bus

[0047] 1200 Processing Components

[0048] 1201 Second Processor

[0049] 1202 Second Memory

[0050] 1203 Communication Components

[0051] 1204 Power Supply Assembly

[0052] 1205 Multimedia Components

[0053] 1206 Voice Component

[0054] 1207 Input / Output Interface

[0055] 1208 sensor assembly DETAILED DESCRIPTION

[0056] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0057] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0058] In related technologies in this field, insufficient data volume during training and testing of three-dimensional images can lead to modality loss, which can easily result in low-quality and noisy depth maps in the data set collected by the Kinect device. This can obscure certain detailed features of the biometric information and reduce the accuracy of biometric recognition. Furthermore, traditional biometric recognition methods, when fusing two-dimensional and three-dimensional images, particularly those employing signal-layer fusion, feature-layer fusion, and decision-layer fusion, suffer from the following drawbacks: For example, they are unable to dynamically determine the weight coefficients of the first image and the depth image based on current lighting and posture change conditions, and are unable to fuse artificial features with depth features based on the complementary features extracted from the first image and the depth image. Consequently, they are unable to obtain cross-modal complementary information, thereby reducing the accuracy of biometric recognition.

[0059] For example, during signal-layer fusion, in the initial stages of biometric recognition, pixels from two-dimensional and three-dimensional images are concatenated, or the coordinates of a point cloud are converted into a three-channel image. This is then combined with a color image to create a six-channel image, which is then fed into a convolutional neural network for feature extraction. This fusion method can easily lead to redundant feature information, increase data space usage, and significantly increase computation time.

[0060] For example, when fusion is performed at the feature level, features are typically extracted from both 2D and 3D images. Human-designed methods are used to extract artificial features, such as using Gabor filters to extract depth map features and grayscale map features separately. Adaboost is then used to select effective features, and deep learning-based methods are used to extract 3D LBP features of shape and texture from the 3D face mesh. These features are then concatenated and identified based on their distance. This simple fusion strategy, which relies on concatenation or element-by-element addition, loses the complementary information of the bimodal features.

[0061] For example, decision-layer fusion uses two or more biometric identifiers to identify two-dimensional images and three-dimensional images respectively, obtains corresponding scores, and uses corresponding strategies to fuse the scores to obtain the final face similarity score. Fusion at the decision layer is relatively simple and convenient, but the complementary features and common features between multiple modalities are not effectively retained, and the advantages of depth images cannot be brought into play.

[0062] Based on the problems existing in the above solutions, the present invention discloses a biometric information recognition method, a biometric information recognition device, an electronic device and a storage medium.

[0063] See also Figure 1 The present invention provides a flow chart of a biometric information recognition method, comprising the following steps:

[0064] Step S1, obtaining a biometric image of a target to be identified;

[0065] Among them, the biometric information image of the target to be identified includes at least one of the following: retina, iris, eyeball, eye pattern, eye shape, mouth shape, tooth shape, face, vein. In this embodiment, the biometric information of the target to be identified (target user) takes a facial image as an example, and the facial image of the target user is obtained in advance through the legal authorization of the target user. The facial image includes an RGB facial image and a depth facial image.

[0066] It should be noted that the biometric information image includes a first image and a second image of the target to be identified, the first image is a color image, and the second image is a depth image, that is, a color face image and a depth face image.

[0067] Specifically, an RGB-D camera is constructed by combining a depth sensor with a 2D color camera. After obtaining legal authorization from the target user, the RGB-D camera is used to capture an RGB-D image. The RGB first image (RGB image) and the depth image are then cropped to capture the corresponding facial area. It should be noted that the first image can be either a 3D or 2D image; the RGB first image is a color standard, resulting from the variations and superposition of the three color channels (red (R), green (G), and blue (B)). The depth image is an image or image channel containing information related to the distance to the surface of scene objects from the viewpoint.

[0068] For example, in specific fields (such as transportation, finance, and security), with the legal authorization of the target user, facial data can be obtained using front-end devices such as structured light (projectors and cameras), TOF (time of flight) cameras, light field cameras, lidar, and other devices.

[0069] Step S2, performing feature extraction on the bio-information image to obtain single-modal color features of different levels and single-modal depth features of different levels;

[0070] Specifically, a preset image feature extraction algorithm can be used to extract the features of the color image and the depth image respectively. The above-mentioned image feature extraction algorithm can be SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradient), CNN (Convolutional Neural Networks), ResNet (Residual Network), etc.

[0071] Step S3, fusing the unimodal color features and the unimodal depth features step by step to obtain multimodal features at each level;

[0072] Specifically, since the network model structure for extracting color images and depth images is the same, the unimodal color features and unimodal depth features of each network level are fused step by step to obtain multimodal features of each level, realizing the extraction of cross-modal features and improving the complementarity between cross-modal features.

[0073] Step S4, fusing the multimodal features with the unimodal color features and the unimodal depth features to obtain cross-modal color convolution features and cross-modal depth convolution features;

[0074] Specifically, since the first image is extracted through a convolutional neural network or a residual convolutional neural network, the unimodal color features obtained are two-dimensional information, and the unimodal depth features are three-dimensional information, the multimodal features are fused with the unimodal color features and the unimodal depth features respectively to obtain cross-modal color convolution features and cross-modal depth convolution features.

[0075] Step S5, fusing the cross-modal color convolution feature and the cross-modal depth convolution feature to obtain a final feature vector;

[0076] Specifically, using a dynamic fusion method, even when the amount of data is insufficient, by obtaining the relationship between two-dimensional data (the color convolution features across modalities) and three-dimensional data (the depth convolution features across modalities), the common (shared) characteristics between them are extracted, and by effectively capturing cross-modal complementary information, the integrity of the final feature vector is improved, achieving more accurate reflection features.

[0077] Step S6: performing information recognition based on the final feature vector of the biological information image.

[0078] Specifically, the final feature vector of the target user's biometric information is obtained to construct a face database, and the face database is used to train an information recognition model. For example, a neural network model is assigned to the training sample constructed from the face database, and the final feature vector in the training sample is input into the corresponding neural network model for recognition training, thereby obtaining an information recognition model, and the final feature vector of the target user to be identified is input into the information recognition model to obtain the recognition result of the target user.

[0079] Through the above method, deep-level feature extraction is performed on RGB images and depth maps. Compared with the 2D method for processing depth maps, this embodiment adopts deep-level cross-modal feature interaction to efficiently extract complementary features from two-dimensional information and three-dimensional information, effectively capturing cross-modal complementary information, so that the 3D face recognition technology based on this method achieves good results in both computational efficiency and accuracy.

[0080] In an exemplary embodiment, before the step S2 of extracting features from the bio-information image to obtain unimodal color features at different levels and unimodal depth features at different levels, the method further includes:

[0081] The depth image is enhanced using a generative adversarial network to obtain a high-quality (image quality enhanced) depth image (ie, a second image).

[0082] Specifically, by adding some noise to the Bosphorus dataset (3D face database) and the FRGCV2 dataset (3D face database) to simulate point cloud data (a set of vectors in a 3D coordinate system), the data corresponding to the same target object in the two datasets are associated to form paired data. The paired data is used as a training set to train the pixel2pixel network (GAN, generative adversarial network), and image conversion is performed through pixel enhancement to generate high-quality images. The trained pixel2pixel network is used to process low-quality depth images to obtain high-quality depth images. At the same time, it can accurately capture the low-frequency information in the image and enhance the quality of the depth image.

[0083] In an exemplary embodiment, step S2 is to perform feature extraction on the biometric image to obtain unimodal color features at different levels and unimodal depth features at different levels, as detailed below:

[0084] Using a convolutional neural network, a feature extraction is performed on the first image and the second image in the biological information image to obtain a color feature vector in the first image and a depth feature vector in the second image;

[0085] A deep residual network is used to perform secondary feature extraction on the color feature vector and the depth feature vector respectively, to obtain unimodal color features of different levels and unimodal depth features of different levels.

[0086] Specifically, the convolutional neural network includes an input layer, a convolution layer, a maximum pooling layer, a dropout layer, a fully connected layer, and a Softmax layer. For example, for pixels of RGB images or depth images, the structure of the convolutional neural network model includes: the input layer is an image of size 32×32; the first layer is the convolution layer C1, the convolution kernel size is 5×5, a total of 8; the second layer is the maximum pooling layer MaxPool1; the third layer is the convolution layer C2, the convolution kernel size is 3×3, a total of 16; the fourth layer is the convolution layer C3, the convolution kernel size is 3×3, a total of 32; The 5th layer is the maximum pooling layer MaxPool2; the 6th layer is the convolution layer C4, the convolution kernel size is 3×3, and there are 32 in total; the 7th layer is the convolution layer C5, the convolution kernel size is 3×3, and there are 32 in total; the 8th layer is the fully connected layer FC1, with a total of 120 neurons; the dropout layer is added after FC1; the 9th layer is the fully connected layer FC2, with a total of 84 neurons; the 10th layer is the Softmax layer, which outputs a color feature vector or a depth feature vector. It can be understood that the size of the input layer and the size of the convolution kernel can be adaptively adjusted according to the size of the image.

[0087] For another example, a first color image (color image) and a second image (depth image) of the same target object are paired, where the first image is (224, 224, 3) and the depth image is (224, 224, 1), the number is n, and the dimension is d0. The color feature vector in the RGB image and the depth feature vector in the depth image are extracted respectively through the above-mentioned convolutional neural network.

[0088] For example, a deep residual network is used to extract features of the color feature vector of an RGB image, and a deep residual network with the same network structure is used to extract the depth feature vector in the depth image, wherein the deep residual network is a ResNet-50 network. For example, the ResNet-50 network includes multiple convolution blocks, and the step size of the last convolution layer in each convolution block is changed from 2 to 1. By changing the convolution kernel step size of the last convolution layer to 1, the resolution of the feature map is improved while increasing the amount of computation very little, and the granularity of the features is enriched. For example, the ResNet-50 network is composed of four residual convolution modules connected end to end (not all of which are shown in the figure), and the four residual convolution modules have 256, 512, 1024 and 2048 filters respectively. Each of the residual convolution modules is connected to a pooling layer with a convolution kernel step size of 2, and the convolution kernel step size of the last convolution layer is changed to 1.

[0089] Specifically, the RGB convolution feature is output by the fourth residual convolution module, and the single-modal depth feature output by the last two residual convolution modules of the depth image is weighted and fused to obtain the cross-modal depth convolution feature.

[0090] See Figure 6 , a schematic diagram of the step-by-step multimodal feature fusion network structure provided by an embodiment of the present invention, wherein the first-level deep residual network interaction module is contained in the dotted box on the left side of the figure. The network structure is composed of a convolutional neural network (a one-layer network structure) and a deep residual network (a four-layer network structure), and the output of the convolutional neural network serves as the input of the deep residual network.

[0091] Specifically, two input images of the same target to be identified are an RGB image and a depth image; RGB feature vectors and depth feature vectors are extracted respectively in two identical convolutional neural networks; RGB feature vectors and depth feature vectors are extracted respectively through two identical deep residual networks, and single-modal color features of different depths and single-modal depth features of different depths are generated in turn.

[0092] According to the network structure, the unimodal color features and the unimodal depth features are fused step by step to obtain the multimodal features of each level, wherein the multimodal features of the first level are obtained by the fusion of the color feature vector and the depth feature vector; the multimodal features of the remaining levels are obtained by the fusion of the unimodal color features of the previous level of the current level, the unimodal depth features of the previous level of the current level, and the multimodal features of the previous level of the current level.

[0093] For example, in the first-level deep residual network interaction structure, the initial layer is composed of a convolutional neural network (CNN). In the initial layer, CNN extracts features from RGB images and depth images respectively, and obtains color feature vectors and depth feature vectors respectively. The multimodal features of the first layer are obtained by fusing the color feature vectors and the depth feature vectors in the initial layer.

[0094] For another example, the multimodal features of the second level are obtained by fusing the multimodal features of the first level, the unimodal color features of the first level in the deep residual network, and the unimodal depth features of the first level in the deep residual network.

[0095] For another example, the multimodal features of the third level are obtained by fusing the multimodal features of the second level, the unimodal color features of the second level in the deep residual network, and the unimodal depth features of the second level in the deep residual network.

[0096] For another example, the multimodal features of the fourth level are obtained by fusing the multimodal features of the third level, the unimodal color features of the third level in the deep residual network, and the unimodal depth features of the third level in the deep residual network.

[0097] It should be noted that, two deep residual networks with the same network structure (i.e., the first deep residual network and the second deep residual network), wherein the unimodal color features of the fourth level in the first deep residual network are fused with the multimodal features of the fourth level, can obtain the output of the last residual convolution module in the first deep residual network; the unimodal depth features of the fourth level in the second deep residual network are fused with the multimodal features of the fourth level to obtain the output of the last residual convolution module in the second deep residual network.

[0098] See also Figure 2 , which is a schematic diagram of a multimodal feature fusion process in a biometric information recognition method provided by an embodiment of the present invention, is described in detail as follows:

[0099] Step S201, fusing the single-modal color features at the same level with the multi-modal features at the same level step by step to obtain the single-modal color features that are enhanced step by step;

[0100] Specifically, in the deep residual level network of the step-by-step multimodal feature fusion network, the unimodal color features at the same level are fused step-by-step with the multimodal features at the same level to ensure that the unimodal color features are enhanced step-by-step.

[0101] Step S202: determining a cross-modal color convolution feature based on the progressively enhanced single-modal color feature output by the last residual convolution module in the deep residual network;

[0102] Specifically, the single-modal color feature that is gradually enhanced and output by the last residual network module in the deep residual network is a cross-modal color convolution feature.

[0103] It should be noted that after the fusion of the single-modal color features of the previous level that are gradually enhanced, they will inevitably have an impact on the single-modal color features of the current level that are gradually enhanced, that is, a step-by-step fusion enhancement method is implemented to obtain cross-modal color convolution features.

[0104] Step S203, fusing the unimodal depth features at the same level with the multimodal features at the same level step by step to obtain the unimodal depth features that are enhanced step by step;

[0105] It should be noted that, after the fusion of the step-by-step enhanced unimodal depth features of the previous level, it is bound to have an impact on the step-by-step enhanced unimodal depth features of the current level.

[0106] Among them, in the deep residual level network of the step-by-step multimodal feature fusion network, the single-modal deep features of the same level are fused step by step with the multimodal features of the same level, that is, a step-by-step fusion enhancement method is implemented to obtain cross-modal deep features.

[0107] Step S204: performing weighted fusion on the progressively enhanced single-modal deep features output by the last two residual convolution modules in the deep residual network to obtain cross-modal deep convolution features.

[0108] Specifically, two step-by-step enhanced single-modal deep features are weightedly fused to obtain cross-modal deep convolutional features, where the weighting coefficients can be fine-tuned on the dataset using the pre-trained VGGFace2 weights.

[0109] In order to assist in fine-tuning the convolutional layers in the entire network (the first-level deep residual network), two additional auxiliary branches are introduced for the loss term of each modality. The branch consists of two fully connected (FC) layers, each with 1024 nodes, where the number of nodes is equal to the number of classes in each dataset. By backpropagating the error to the early layers of the network, the network fine-tuning is achieved by learning the weights of the convolution extractor to improve the accuracy of feature extraction.

[0110] In the above manner, on the basis of single-modal depth features and single-modal color features, they are fused step by step with the multimodal features of the same level, and then cross-modal depth convolution features and cross-modal color convolution features are obtained respectively. Since the step-by-step cross-modal fusion captures the cross-channel correlation between all features of different modalities from RGB images and depth images, more significant features can be extracted, and cross-modal information from RGB and depth images can be effectively captured, which is helpful to detect more complete and consistent targets and improve the modal matching capability of RGB features and depth features.

[0111] See also Figure 3 A schematic diagram of a cross-modal generation process in a biometric information recognition method according to an embodiment of the present invention is described in detail as follows:

[0112] Step S301: determining the interactive spatial attention between the cross-modal color convolutional features and the cross-modal depth convolutional features based on a multi-head attention mechanism, and obtaining an attention map of the color features and an attention map of the depth features;

[0113] Specifically, based on the multi-head attention mechanism in Transformer, it is used to capture the respective importance of each pair of RGB and depth features. The cross-modal interaction module based on the co-attention mechanism enhances the co-attention into multiple attention heads to capture the complex relationships in different latent spaces, and takes their average as the attention result.

[0114] For example, using a multi-head attention mechanism means that the same feature needs to be mapped multiple times. Each time it is mapped to a space, the RGB and depth features are given new meanings, allowing the Transformer to pay attention to information in the subspace. Another example is using a multi-head self-attention mechanism to capture feature information from different aspects.

[0115] Step S302: multiplying the attention map of the color feature with the cross-modal color convolution feature to obtain the modality-enhanced cross-modal color convolution feature; and multiplying the attention map of the depth feature with the cross-modal depth convolution feature to obtain the modality-enhanced cross-modal depth convolution feature;

[0116] Specifically, the attention map of the color feature is multiplied with the cross-modal color convolution feature, or the attention map of the depth feature is multiplied with the cross-modal depth convolution feature, which can capture the interaction between the dimensions of the two modal features.

[0117] Step S303 : Dynamically fuse the modality-enhanced cross-modal color convolution features and the modality-enhanced cross-modal depth convolution features to obtain a final feature vector.

[0118] Specifically, the gate function is used to dynamically fuse the cross-modal color convolution features enhanced by modality and the cross-modal depth convolution features enhanced by modality to obtain the final feature vector, wherein,

[0119] M=Tanh(W R M R +W D M D )

[0120] Where M is the final feature vector, tanh is the activation function, and W R is the weight coefficient of the cross-modal color convolution feature for modality enhancement, W D is the weight coefficient of the cross-modal depth convolution feature for modality enhancement, M D The cross-modal deep convolutional features for modality enhancement; M R The cross-modal color convolutional features for modality enhancement.

[0121] In this way, the final feature vector obtained by cross-modal interaction can extract more significant features, effectively capture cross-modal information from RGB images and depth images, and help detect more complete targets. In the subsequent model training process, the final feature vector is used to train the recognition model, thereby greatly improving the recognition accuracy of biological information.

[0122] See also Figure 4 , which is a schematic diagram of a cross-modal interaction process in a biometric information recognition method according to an embodiment of the present invention, wherein step S301 is described in detail as follows:

[0123] Step S401, mapping the color convolution feature and the depth convolution feature to the same dimension to obtain a two-dimensional pixel sequence of a flattened convolution feature map;

[0124] Specifically, due to the difference in feature dimensions between RGB and weighted integration, it is impossible to directly perform inner product operations on features between modalities. Therefore, preprocessing is required through the above operations.

[0125] Step S402: Map the two-dimensional pixel sequences of the same dimension into a feature space of a preset dimension to obtain attention maps corresponding to the color convolution feature and the depth convolution feature respectively.

[0126] Specifically, calculating the attention maps corresponding to the color convolution features and the depth convolution features based on the attention mechanism is conducive to improving the accuracy of biometric information recognition.

[0127] In step S403, the attention maps corresponding to the color convolution feature and the depth convolution feature are processed by the standardized inner product, and the attention maps are summed and the average value of the attention maps is determined to obtain the attention map of the color feature and the attention map of the depth feature.

[0128] Specifically, combined Figure 3 and Figure 4 For example, first, the two features (color convolution features and the depth convolution features) are mapped to the same dimension (d), and the convolution feature map of size (N; C; H; W) is reshaped into a flattened two-dimensional pixel sequence of size (N; H × W; C), where (H; W) is the resolution of the original feature map and C is the number of channels. Secondly, the two d-dimensional multimodal features (color convolution features and depth convolution features) are mapped to H d - dimensional feature space, Then the i-th RGB attention map and the i-th Depth attention map can be expressed as:

[0129] R h =Linear(R;θ R,h ),D h=Linear(D;θ D,h )

[0130] Where Linear(·) represents a fully connected layer with parameter θ, R is the RGB image, D is the depth image, and h is the current d - The number of dimensional feature spaces;

[0131] Again, the attention maps corresponding to the color convolution features and the depth convolution features are calculated through the row-normalized inner product:

[0132]

[0133]

[0134] Where R is the RGB image, D is the depth image, and h is the current d - The number of dimensional feature spaces is mapped using the softmax function. These are the attention maps corresponding to color convolution features and depth convolution features respectively.

[0135] Then, the H attention maps are summed and the corresponding average is calculated, and then multiplied with the corresponding convolutional feature to obtain the modality enhancement feature in which each pixel carries semantic information;

[0136]

[0137] M R =A R ·R,M D =A D ·D

[0138] Among them, M R Semantically, the input depth map guides the network to “focus on” which visual features in the RGB image, while M D The semantically represented input RGB image guides the network to focus on "what is meaningful for recognition" in the given depth image. After obtaining the two-way enhanced modal features, the final 3D features are dynamically fused by using a gate function.

[0139] Through the above method, the interaction between cross-modal features can be quickly obtained, and cross-modal complementary information can be effectively captured. It not only effectively captures the low-level details and high-level semantic information of the input image, but also avoids the excessive number of feature levels that leads to a sharp increase in the overall network parameters, and reduces the difficulty of network training. The final feature vector determined by cross-modal interaction can extract more significant features, effectively capture cross-modal information from the first image and the depth image, and help detect more complete targets. Information recognition is performed based on the final feature vector of biological information, which also improves the accuracy of biological information recognition.

[0140] See also Figure 5 , a biometric information recognition device 500 provided in an embodiment of the present invention, comprising:

[0141] An acquisition module 501 is used to acquire a biological information image of a target to be identified;

[0142] The biometric image includes a first image and a second image of the target to be identified, wherein the first image is a color image and the second image is a depth image. The biometric image is at least one of a retina, an iris, an eyeball, eye lines, an eye shape, a mouth shape, a face, and veins.

[0143] An extraction module 502 is configured to perform feature extraction on the bio-information image to obtain single-modal color features at different levels and single-modal depth features at different levels;

[0144] A step-by-step fusion module 503 is used to fuse the single-modal color features and the single-modal depth features step by step to obtain multimodal features at each level;

[0145] A cross-modal generation module 504 is configured to fuse the multi-modal features with the unimodal color features and the unimodal depth features to obtain cross-modal color convolution features and cross-modal depth convolution features;

[0146] A cross-modal interaction module 505 is configured to fuse the cross-modal color convolutional features with the cross-modal depth convolutional features to obtain a final feature vector;

[0147] The information recognition module 506 performs information recognition based on the final feature vector of the biological information image.

[0148] In an exemplary embodiment, before the extraction module 502, the biometric information recognition device further includes: an image enhancement module, which uses a generative adversarial network to enhance the depth image to obtain a high-quality (image quality enhanced) depth image.

[0149] In an exemplary embodiment, the extraction module 502 includes:

[0150] A first extraction unit is configured to perform feature extraction on the first image and the second image in the biological information using a convolutional neural network to obtain a color feature vector in the first image and a depth feature vector in the second image;

[0151] The second extraction unit uses a deep residual network to perform secondary feature extraction on the color feature vector and the depth feature vector respectively, to obtain unimodal color features of different levels and unimodal depth features of different levels.

[0152] In an exemplary embodiment, the step-by-step fusion module 503 further includes: fusing unimodal color features and unimodal depth features step by step according to the network structure to obtain multimodal features of each level, wherein the multimodal features of the first level are obtained by fusing the color feature vector and the depth feature vector; the multimodal features of the remaining levels are obtained by fusing the multimodal features of the previous level of the current level, the unimodal color features of the previous level of the current level, and the unimodal depth features of the previous level of the current level.

[0153] In an exemplary embodiment, the cross-modal generation module 504 further includes:

[0154] The unimodal color features at the same level are gradually fused with the multimodal features at the same level to obtain the unimodal color features that are gradually enhanced; and cross-modal color convolution features are determined based on the gradually enhanced unimodal color features output by the last residual convolution module in the deep residual network;

[0155] The unimodal deep features of the same level are fused step by step with the multimodal features of the same level to obtain the unimodal deep features that are enhanced step by step; the unimodal deep features that are enhanced step by step are weightedly fused according to the outputs of the last two residual convolution modules in the deep residual network to obtain cross-modal deep convolution features.

[0156] In an exemplary embodiment, the cross-modal interaction module 505 further includes:

[0157] Determine the interactive spatial attention between the cross-modal color convolutional features and the cross-modal depth convolutional features based on a multi-head attention mechanism, and obtain an attention map of the color features and an attention map of the depth features;

[0158] Among them, the color convolution feature and the depth convolution feature are mapped to the same dimension to obtain a two-dimensional pixel sequence of the convolution feature map flattened; the two-dimensional pixel sequences of the same dimension are respectively mapped to the feature space of preset dimensions to obtain the attention maps corresponding to the color convolution feature and the depth convolution feature; the attention maps corresponding to the color convolution feature and the depth convolution feature are processed by standardized inner product, and the attention maps of each of the attention maps are summed and the average value of the attention maps is determined to obtain the attention map of the color feature and the attention map of the depth feature.

[0159] Multiplying the attention map of the color feature with the cross-modal color convolution feature to obtain the modality-enhanced cross-modal color convolution feature; and multiplying the attention map of the depth feature with the cross-modal depth convolution feature to obtain the modality-enhanced cross-modal depth convolution feature;

[0160] The modality-enhanced cross-modal color convolution features and the modality-enhanced cross-modal depth convolution features are dynamically fused to obtain a final feature vector.

[0161] The gate function is used to dynamically fuse the modality-enhanced cross-modal color convolution features and the modality-enhanced cross-modal depth convolution features to obtain the final feature vector, wherein,

[0162] M=Tanh(W R M R +W D M D )

[0163] Where M is the final feature vector, tanh is the activation function, and W R is the weight coefficient of the cross-modal color convolution feature for modality enhancement, W D is the weight coefficient of the cross-modal depth convolution feature for modality enhancement, M D The cross-modal deep convolutional features for modality enhancement; M R The cross-modal color convolutional features for modality enhancement.

[0164] In this embodiment, the biometric information recognition device and the biometric information recognition method have a one-to-one correspondence. For detailed technical details, technical functions and technical effects, please refer to the above embodiments, which will not be described in detail here.

[0165] In summary, the present invention provides a biometric information recognition device, which obtains single-modal color features of different depths and single-modal depth features of different depths by extracting a first image and a depth image respectively; the single-modal color features and the single-modal depth features are fused step by step to obtain multimodal features at each level, and the complementarity (correlation) between the color features and the depth features is improved by the fused multimodal features; the multimodal features are fused with the single-modal color features and the single-modal depth features respectively to obtain cross-modal color convolution features and cross-modal depth convolution features; the cross-modal color convolution features and the cross-modal depth convolution features are fused to obtain a final feature vector. Through the above method, not only the low-level details and high-level semantic information of the input image are effectively captured, but also the number of feature levels is avoided. The sharp increase in the overall network parameters caused by too many levels is avoided, and the difficulty of network training is reduced. The final feature vector determined by cross-modal interaction can extract more significant features, effectively capture cross-modal information from the first image and the depth image, and help to detect more complete targets. Information recognition is performed based on the final feature vector of biometric information, which also improves the accuracy of biometric information recognition.

[0166] An embodiment of the present invention further provides a device, which may include: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enable the device to execute Figure 1 In practical applications, the device can be used as a terminal device or a server. Examples of terminal devices may include: smartphones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc. The embodiments of the present invention do not limit the specific devices.

[0167] The embodiment of the present invention further provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute the embodiment of the present invention. Figure 1 Instructions for the steps of the method described herein.

[0168] Figure 7 Schematic diagram of the hardware structure of a terminal device provided by an embodiment of the present invention. Figure 7 As shown, the terminal device may include: an input device 1100, a first processor 1101, an output device 1102, a first memory 1103, and at least one communication bus 1104. Communication bus 1104 is used to implement communication connections between components. First memory 1103 may include high-speed RAM memory or non-volatile storage NVM, such as at least one disk storage. First memory 1103 may store various programs for performing various processing functions and implementing the method steps of this embodiment.

[0169] Optionally, the first processor 1101 may be implemented as, for example, a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and the first processor 1101 is coupled to the input device 1100 and the output device 1102 via a wired or wireless connection.

[0170] Optionally, the input device 1100 may include multiple input devices, such as at least one of a user interface for a user, a device interface for a device, a software programmable interface, a camera, and a sensor. Optionally, the device interface for a device may be a wired interface for data transmission between devices, or a hardware plug-in interface for data transmission between devices (such as a USB interface, a serial port, etc.); optionally, the user interface for a user may be, for example, a user-oriented control button, a voice input device for receiving voice input, and a touch sensing device for receiving user touch input (such as a touch screen or touchpad with touch sensing function); optionally, the software programmable interface may be, for example, an entry for a user to edit or modify a program, such as an input pin interface or input interface of a chip; the output device 1102 may include output devices such as a display and a speaker.

[0171] In this embodiment, the processor of the terminal device includes functions for executing each module of the voice recognition device in each device. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0172] Figure 8 A schematic diagram of the hardware structure of a terminal device provided in one embodiment of the present invention. Figure 6 Yes Figure 5 A specific embodiment in the implementation process. Figure 8 As shown, the terminal device of this embodiment may include a second processor 1201 and a second memory 1202.

[0173] The second processor 1201 executes the computer program code stored in the second memory 1202 to implement the above embodiment. Figure 1 The method.

[0174] The second memory 1202 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, such as messages, images, videos, etc. The second memory 1202 may include random access memory (RAM) and may also include non-volatile memory (non-volatile memory), such as at least one disk storage device.

[0175] Optionally, the second processor 1201 is provided in the processing component 1200. The terminal device may further include: a communication component 1203, a power component 1204, a multimedia component 1205, a voice component 1206, an input / output interface 1207, and / or a sensor component 1208. The specific components included in the terminal device are set according to actual needs and are not limited in this embodiment.

[0176] The processing component 1200 generally controls the overall operation of the terminal device. The processing component 1200 may include one or more second processors 1201 to execute instructions to complete all or part of the steps in the above-described data processing method. In addition, the processing component 1200 may include one or more modules to facilitate interaction between the processing component 1200 and other components. For example, the processing component 1200 may include a multimedia module to facilitate interaction between the multimedia component 1205 and the processing component 1200.

[0177] The power supply component 1204 provides power to various components of the terminal device. The power supply component 1204 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0178] The multimedia component 1205 includes a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen may include a liquid crystal display (LCD) and a touch panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0179] The voice component 1206 is configured to output and / or input voice signals. For example, the voice component 1206 includes a microphone (MIC), which is configured to receive external voice signals when the terminal device is in an operating mode, such as a voice recognition mode. The received voice signals can be further stored in the second memory 1202 or transmitted via the communication component 1203. In some embodiments, the voice component 1206 also includes a speaker for outputting voice signals.

[0180] The input / output interface 1207 provides an interface between the processing component 1200 and peripheral interface modules, which may be click wheels, buttons, etc. These buttons may include but are not limited to: volume buttons, start buttons, and lock buttons.

[0181] Sensor component 1208 includes one or more sensors for providing various status assessments for the terminal device. For example, sensor component 1208 can detect the open / closed state of the terminal device, the relative positioning of components, and the presence or absence of user contact with the terminal device. Sensor component 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, sensor component 1208 may also include a camera, etc.

[0182] The communication component 1203 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In one embodiment, the terminal device may include a SIM card slot for inserting a SIM card, allowing the terminal device to log into a GPRS network and establish communication with a server via the Internet.

[0183] From the above, we can see that Figure 8 The communication component 1203, the voice component 1206, the input / output interface 1207, and the sensor component 1208 involved in the embodiment can all be used as Figure 7 Implementation of the input device in the embodiment.

[0184] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A biometric information recognition method, characterized in that: The following steps are involved: Acquire a biometric image of a target to be identified; Extracting features from the bio-information image to obtain single-modal color features at different levels and single-modal depth features at different levels; The unimodal color features and the unimodal depth features are gradually integrated to obtain multimodal features at each level; The multimodal features are respectively fused with the single-modal color features and the single-modal depth features to obtain cross-modal color convolution features and cross-modal depth convolution features; The cross-modal color convolution feature and the cross-modal depth convolution feature are integrated to obtain the final feature vector; specifically including: mapping the color convolution feature and the depth convolution feature to the same dimension to obtain a two-dimensional pixel sequence of the convolution feature map flattened; mapping the two-dimensional pixel sequences of the same dimension in two ways to the feature space of preset dimensions respectively to obtain the attention maps corresponding to the color convolution feature and the depth convolution feature; processing the attention maps corresponding to the color convolution feature and the depth convolution feature by normalizing the inner product, summing up the attention maps and determining the average value of the attention maps, and then obtaining the attention map of the color feature and the attention map of the depth feature; multiplying the attention map of the color feature with the cross-modal color convolution feature to obtain the modality-enhanced cross-modal color convolution feature; and multiplying the attention map of the depth feature with the cross-modal depth convolution feature to obtain the modality-enhanced cross-modal depth convolution feature; dynamically fusing the modality-enhanced cross-modal color convolution feature and the modality-enhanced cross-modal depth convolution feature to obtain the final feature vector; Information recognition is performed based on the final feature vector of the biological information image.

2. The biometric information recognition method according to claim 1, wherein: The biological information image includes a first image and a second image of a target to be identified, wherein the first image is a color image and the second image is a depth image.

3. The biometric information recognition method according to claim 2, wherein: Also includes: The second image is enhanced using a generative adversarial network to obtain a second image with enhanced image quality.

4. The biometric information recognition method according to claim 1 or 2, characterized in that: The biological information image includes at least one of the following: retina, iris, eyeball, eye pattern, eye shape, mouth shape, human face, and vein.

5. The biometric information recognition method according to claim 1 or 2, characterized in that: The step of extracting features from the biological information image to obtain single-modal color features at different levels and single-modal depth features at different levels includes: Using a convolutional neural network to perform feature extraction on the first image and the second image in the biological information, respectively, to obtain color features in the first image and depth features in the second image; A deep residual network is used to perform secondary feature extraction on the color features and the depth features respectively, to obtain unimodal color features of different levels and unimodal depth features of different levels.

6. The biometric information recognition method according to claim 5, characterized in that: The step of fusing the single-modal color features and the single-modal depth features step by step to obtain multimodal features at each level includes: According to the network structure, the unimodal color features and the unimodal depth features are fused step by step to obtain the multimodal features of each level, wherein the multimodal features of the first level are obtained by the fusion of the color features and the depth features; the multimodal features of the remaining levels are obtained by the fusion of the unimodal color features of the previous level of the current level, the unimodal depth features of the previous level of the current level, and the multimodal features of the previous level of the current level.

7. The biometric information recognition method according to claim 1, wherein: The step of fusing the multimodal features with the unimodal color features and the unimodal depth features to obtain cross-modal color convolution features and cross-modal depth convolution features includes: The single-modal color features at the same level are gradually integrated with the multi-modal features at the same level to obtain the single-modal color features that are enhanced step by step; Determining a cross-modal color convolution feature based on the progressively enhanced single-modal color feature output by the last residual convolution module in the deep residual network; The unimodal depth features at the same level are gradually integrated with the multimodal features at the same level to obtain the unimodal depth features that are enhanced step by step; The weighted fusion of the progressively enhanced single-modal deep features output by the last two residual convolution modules in the deep residual network is performed to obtain cross-modal deep convolution features.

8. The biometric information recognition method according to claim 1, wherein: Also includes: Dynamically fuse the modality-enhanced cross-modality color convolution features and the modality-enhanced cross-modality depth convolution features to obtain the final feature vector, where: Where M is the final feature vector, tanh is the activation function, and W R is the weight coefficient of the cross-modal color convolution feature for modality enhancement, W D is the weight coefficient of the cross-modal depth convolution feature for modality enhancement, The cross-modal deep convolutional features for modality enhancement; The cross-modal color convolutional features for modality enhancement.

9. A biometric information recognition device, characterized in that: include: An acquisition module is used to acquire a biological information image of a target to be identified; An extraction module is used to extract features from the biological information image to obtain single-modal color features of different levels and single-modal depth features of different levels; A step-by-step fusion module, configured to fuse the single-modal color features and the single-modal depth features step-by-step to obtain multimodal features at each level; A cross-modal generation module is used to fuse the multimodal features with the unimodal color features and the unimodal depth features to obtain cross-modal color convolution features and cross-modal depth convolution features; specifically, the module includes: mapping the color convolution features and the depth convolution features to the same dimension to obtain a two-dimensional pixel sequence of a convolution feature map; mapping the two-dimensional pixel sequences of the same dimension to a feature space of a preset dimension to obtain the attention maps corresponding to the color convolution features and the depth convolution features; and performing normalized inner product processing on the attention maps corresponding to the color convolution features and the depth convolution features. , summing up each of the attention maps and determining the average of the attention maps, thereby obtaining the attention map of the color feature and the attention map of the depth feature; multiplying the attention map of the color feature with the cross-modal color convolution feature to obtain the modality-enhanced cross-modal color convolution feature; and multiplying the attention map of the depth feature with the cross-modal depth convolution feature to obtain the modality-enhanced cross-modal depth convolution feature; dynamically fusing the modality-enhanced cross-modal color convolution feature and the modality-enhanced cross-modal depth convolution feature to obtain the final feature vector; A cross-modal interaction module, configured to fuse the cross-modal color convolutional features with the cross-modal depth convolutional features to obtain a final feature vector; An information recognition module performs information recognition based on the final feature vector of the biological information image.

10. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the apparatus to perform the method of one or more of claims 1-8.

11. One or more machine-readable media, characterized in that Instructions are stored thereon, which, when executed by one or more processors, cause the apparatus to perform the method according to one or more of claims 1-8.

Images