An end-side fingerprint representation identification method based on multi-task learning

Abstract

The application discloses an end-side fingerprint representation recognition method based on multi-task learning. First, the fingerprint picture is preprocessed, and then the preprocessed training data is input into a backbone neural network to obtain basic features; then the basic features are input into a minutia extraction network, and texture information maps are generated through multiple layers of convolution and deconvolution. The basic features are input into a multi-layer perceptron to generate topological information and obtain corresponding category information. Finally, the basic features, the texture information and the topological information are updated through a joint loss function and backward propagation in three network modules to achieve the purpose of converting the basic features into fingerprint representation information with the assistance of the texture information and the topological information. The application uses a multi-task learning method to learn fingerprint feature information in multiple scales, effectively improving the fingerprint recognition accuracy. The application fuses fingerprint multi-scale information into a fingerprint representation, making the robustness stronger. The application adopts a lightweight network throughout, saving hardware resource overhead and being more suitable for end-side devices.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and relates to pattern recognition and multi-task learning techniques in deep learning, specifically to an edge fingerprint representation recognition method based on multi-task learning. Background Technology

[0002] Deep learning technology has brought revolutionary progress and development to the field of computer vision. Most of the current biometric technologies with excellent recognition effects (such as face, iris, gait, etc.) can be attributed to the use of deep neural networks and large-scale training sets.

[0003] In the field of fingerprint recognition, many large-scale model networks are widely known for their high accuracy and strong fitting performance, such as DeepPrint and Transformer. However, when these model networks are deployed on edge embedded devices (platforms characterized by limited memory resources, low processor performance, and power consumption), they cannot be deployed or run in real time on these platforms due to limitations in storage space and computing power. Lightweight networks, with their low power consumption and small size, have become an effective solution to meet this requirement. However, current lightweight networks suffer from low recognition rates in the fingerprint recognition field and cannot adapt to real-world application scenarios.

[0004] Therefore, this patent proposes an edge fingerprint representation recognition method based on multi-task learning. This method achieves the final recognition effect through multi-task learning, with each task assisting the other. It can achieve higher recognition accuracy and computational efficiency while maintaining a small size. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an edge fingerprint representation recognition method based on multi-task learning, aiming to obtain a lightweight model capable of fingerprint recognition technology, which can be deployed on edge devices and embedded devices to achieve low parameters and high performance.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] An edge fingerprint representation recognition method based on multi-task learning includes the following steps:

[0008] Step (1) Batch fingerprint image input and preprocessing: The batch fingerprint images are scaled, then binarized and standardized, and the processed image data is randomly divided into training data and validation data.

[0009] Step (2) Basic Feature Task Learning: Input the preprocessed training data into a weight-shared backbone neural network to obtain basic features, which will serve as the starting point for subsequent multi-task learning.

[0010] Step (3) Minute Extraction Task Learning: Input the basic features into the minute extraction network, and generate a texture information map through multiple convolutions and deconvolutions.

[0011] Step (4) Fingerprint category task learning: Input the basic features into the multilayer perceptron to generate topological information and obtain the corresponding category information.

[0012] Step (5) updates the basic features, texture information and topology information by backpropagating them to the three network modules through a joint loss function, so as to transform the basic features into fingerprint representation information with the help of texture information and topology information.

[0013] Step (6) Test the trained network model and evaluate its accuracy.

[0014] Furthermore, the specific steps of step (1) are as follows:

[0015] Step (1-1) Obtain the dataset and preprocess the fingerprint images in the dataset, including: cropping based on ROI regions, filling the cropped images into square images, and scaling the images to M*M size.

[0016] Steps (1-2) involve iterating through the grayscale values ​​L from 0 to 255 in the preprocessed fingerprint image, dividing the image's pixel values ​​into target and background parts based on the grayscale values, and simultaneously counting the number of pixels N0, grayscale value, Sum0, and mean grayscale value of the target part. The number of pixels N1, grayscale value, Sum1, and average grayscale value of the background area. And the proportion of each type of pixel in the fingerprint image. Finally, the inter-class variance σ is obtained. 2 :

[0017] σ 2 =ω0ω1(μ0-μ1) 2 ;

[0018] This will make σ 2 The maximum gray value L is set as the threshold, and the pixels are binarized using this threshold to obtain the binarized image.

[0019] Steps (1-3) standardize the binarized image to make it conform to a normal distribution.

[0020] Steps (1-4) randomly divide the standardized fingerprint images into training data and validation data for further network training.

[0021] Furthermore, the specific steps of step (2) are as follows:

[0022] The backbone neural network with weight sharing described in step (2-1) adopts the AIFingerNet model and is pre-trained using the SimCLR method.

[0023] Step (2-2) inputs the training data into the pre-trained AIFingerNet model to obtain the output of the last layer of the AIFingerNet model, i.e. the basic features, which are used as inputs for the remaining learning tasks. The feature vector size is 192*5*5.

[0024] Furthermore, the specific steps in step (3) are as follows:

[0025] Step (3-1) takes the basic features extracted by the AIFingerNet model as input, and passes them through a minutiae extraction network consisting of two layers of convolution and one layer of deconvolution, generating a texture information map of size layer*128*128 as x.

[0026] Step (3-2) obtains a set T consisting of minutiae on the fingerprint ridges based on the preprocessed fingerprint image from step (1-1):

[0027] T = {m1, m2, ..., m t …,m n};

[0028] Where m t =(x t ,y t ,θ t ), t=1,…,n, where n represents the number of detail points, (x t ,y t θ represents the coordinates of the t-th minutiae in the fingerprint image. t Indicates the direction of the t-th detail point, in the range [0, 2π].

[0029] Step (3-3) encodes the detail point set T into a heatmap of the same size as x, which is then used as the true label input y. The encoding method is as follows:

[0030] Iterate through each point in y and represent it as (i,j,k), where (i,j) represents the coordinates of the point in y, and k represents the channel dimension of the point in y. Let the value of this point be M(i,j,k), then:

[0031]

[0032] Wherein, Cs((x t ,y t ),(i,j))·Co(θ t ,2kπ / layer) represent the detail points (x t ,y t ,θt The contribution of Cs((x) to M(i,j,j) by the spatial coordinates and rotation direction, specifically, Cs((x) t ,y t ),(i,j)) means (x t ,y t The Euclidean distance between (i,j) and (i,j), Co(θ) t ,2kπ / layer) represents θ t The directional difference between 2x / layer and 2x / layer.

[0033] Step (3-4) uses the mean loss function L based on the texture information map x and the ground truth label input y. map Calculate the contribution of texture information to the final representation information:

[0034]

[0035] Here, weight is a two-dimensional matrix with all values ​​of 1 and a size of (1, a), and Δ(x, y) is a matrix of size (a, 1) composed of (x, y). 2 The result is reconstructed into a matrix of size (a, -1), where -1 indicates that the size of this dimension is adaptive, and then the average of the adaptive second dimension is taken.

[0036] Furthermore, the specific steps in step (4) are as follows:

[0037] Step (4-1) takes the basic features extracted by the AIFingerNet model as input, passes them through a multilayer perceptron consisting of two fully connected layers and one drop layer, and obtains the probability of the fingerprint image in each category. The final category of the fingerprint image is determined by the category with the highest probability.

[0038] Step (4-2) uses the class probability of the fingerprint image and the true label probability after softmax as the cross-entropy loss function L. cross The input is used to calculate the contribution of topological information to the final representation information:

[0039]

[0040] Where p represents the class probability. This represents the probability of the true label.

[0041] Furthermore, the specific steps in step (5) are as follows:

[0042] Step (5-1) combines the basic features and the true label probabilities using the center loss function L. center Calculate the contribution of the basic features to the final representation information:

[0043]

[0044] Among them, c x Representing basic features, c y represents the true label probability, and m represents the number of features.

[0045] Step (5-2) Establish the joint loss function;

[0046] L=λ1*L map +λ2*L center +λ3*L cross ;

[0047] Where λ1, λ2, and λ3 are all hyperparameters used to balance the mean loss function L at detail points. map Cross-entropy loss function L cross and center loss function L center The proportions are λ1 = 0.3, λ2 = 0.125, and λ3 = 1.

[0048] Step (5-3) uses a joint loss function to monitor the backbone network, minutiae extraction network, and multilayer perceptron, and backpropagates to update the parameters of the three, so that the basic features extracted by the backbone network are the representation information.

[0049] Furthermore, the specific steps in step (6) are as follows:

[0050] Step (6-1) inputs each fingerprint image of the verification data into the trained backbone network. The output of the backbone network is the final representation information, which is a 192-dimensional feature vector.

[0051] Step (6-2) takes the representation information of a random fingerprint image under each class as a reference sample, and calculates the similarity between the representation information of the remaining fingerprint images and the reference sample. When the similarity between the two is higher than the set threshold, they are judged to be fingerprints of the same class; otherwise, they are judged to be fingerprints of the same class. The same error rate is obtained by batch comparison and is used as an evaluation accuracy index.

[0052] The beneficial effects of this invention are as follows:

[0053] 1. This invention uses a multi-task learning method to learn fingerprint feature information at multiple scales, effectively improving fingerprint recognition accuracy.

[0054] 2. This invention fuses multi-scale fingerprint information into a single fingerprint representation, resulting in stronger robustness.

[0055] 3. This invention uses a lightweight network throughout, which saves hardware resource costs and is more compatible with end-side devices. Attached Figure Description

[0056] Figure 1 Flowchart of the method of this invention. Detailed Implementation

[0057] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0058] This invention mainly proposes an edge fingerprint representation recognition method based on multi-task learning. Figure 1 The specific implementation steps of this invention are as follows:

[0059] Step (1) Batch fingerprint image input and preprocessing: The batch fingerprint images are scaled, then binarized and standardized. The processed image data is randomly divided into training data and validation data. The specific steps are:

[0060] Step (1-1) Obtain the dataset and preprocess the fingerprint images in the dataset, including: cropping based on ROI region, filling the cropped image into a square image, and scaling the image to M*M size. In this embodiment, M=160.

[0061] Steps (1-2) involve iterating through the grayscale values ​​L from 0 to 255 in the preprocessed fingerprint image, dividing the image's pixel values ​​into target and background parts based on the grayscale values, and simultaneously counting the number of pixels N0, grayscale value, Sum0, and mean grayscale value of the target part. The number of pixels N1, grayscale value, Sum1, and average grayscale value of the background area. And the proportion of each type of pixel in the fingerprint image. Finally, the inter-class variance σ is obtained. 2 :

[0062] σ 2 =ω0ω1(μ0-μ1) 2 ;

[0063] This will make σ 2 The maximum gray value L is set as the threshold, and the pixels are binarized using this threshold to obtain the binarized image.

[0064] Steps (1-3) standardize the binarized image to make it conform to a normal distribution.

[0065] Steps (1-4) randomly divide the standardized fingerprint images into training data and validation data for further network training.

[0066] Step (2) Basic Feature Learning: The preprocessed training data is input into a weight-sharing backbone neural network to obtain basic features, which serve as the starting point for subsequent multi-task learning. The specific steps are:

[0067] The backbone neural network with weight sharing described in step (2-1) adopts the AIFingerNet model and is pre-trained using the SimCLR method. The AIFingerNet model is an improvement on AI85FaceIDNet and can be deployed on edge embedded devices to achieve high precision.

[0068] Step (2-2) inputs the training data into the pre-trained AIFingerNet model to obtain the output of the last layer of the AIFingerNet model, i.e. the basic features, which are used as inputs for the remaining learning tasks. The feature vector size is 192*5*5.

[0069] Step (3) Minute Extraction Task Learning: Input the basic features into the minute extraction network, and generate a texture information map through multiple convolutions and deconvolutions.

[0070] Step (3-1) takes the basic features extracted by the AIFingerNet model as input, and passes them through a minutiae extraction network consisting of two convolutional layers and one deconvolutional layer, generating a texture information map of size layer*128*128 as x. In this embodiment, layer=6.

[0071] Step (3-2) uses a fingerprint minutiae extractor to obtain a set T consisting of minutiae such as bifurcation points and endpoints on the fingerprint ridges, based on the preprocessed fingerprint image from step (1-1).

[0072] T = {m1, m2, ..., m t …,m n};

[0073] Where m t =(x t ,y t ,θ t ), t=1,…,n, where n represents the number of detail points, (x t ,y t θ represents the coordinates of the t-th minutiae in the fingerprint image. t Indicates the direction of the t-th detail point, in the range [0, 2π].

[0074] Step (3-3) encodes the detail point set T into a heatmap of the same size as x, which is then used as the true label input y. The encoding method is as follows:

[0075] Iterate through each point in y and represent it as (i,j,k), where (i,j) represents the coordinates of the point in y, and k represents the channel dimension of the point in y. Let the value of this point be M(i,j,k), then:

[0076]

[0077] Wherein, Cs((x t ,y t ),(i,j))·Co(θ t ,2kπ / layer) represent the detail points (x t ,y t ,θ t The contribution of the spatial coordinates and rotation direction of Cs((x) to M(i,j,k), specifically, Cs((x) t ,y t ),(i,j)) means (x t ,y t The Euclidean distance between (i,j) and (i,j), Co(θ) t ,2kπ / layer) represents θ t The directional difference between 2x / layer and 2x / layer.

[0078] Step (3-4) uses the mean loss function L based on the texture information map x and the ground truth label input y. map Calculate the contribution of texture information to the final representation information:

[0079]

[0080] Here, weight is a two-dimensional matrix with all values ​​of 1 and a size of (1, a). In this embodiment, a = 16. Δ(x, y) is a matrix of size (a, 1), which is composed of (x, y). 2 The result is reconstructed into a matrix of size (a, -1), where -1 indicates that the size of this dimension is adaptive, and then the average of the adaptive second dimension is taken.

[0081] Step (4) Fingerprint category task learning: Input the basic features into the multilayer perceptron to generate topological information and obtain the corresponding category information.

[0082] Step (4-1) takes the basic features extracted by the AIFingerNet model as input, passes them through a multilayer perceptron consisting of two fully connected layers and one drop layer, and obtains the probability of the fingerprint image in each category. The final category of the fingerprint image is determined by the category with the highest probability.

[0083] Step (4-2) uses the class probability of the fingerprint image and the true label probability after softmax as the cross-entropy loss function L. cross The input is used to calculate the contribution of topological information to the final representation information:

[0084]

[0085] Where p represents the class probability. This represents the probability of the true label.

[0086] Step (5) updates the basic features, texture information and topology information by backpropagating them to the three network modules through a joint loss function, so as to transform the basic features into fingerprint representation information with the help of texture information and topology information.

[0087] Step (5-1) combines the basic features and the true label probabilities using the center loss function L. center Calculate the contribution of the basic features to the final representation information:

[0088]

[0089] Among them, c x Representing basic features, c y represents the true label probability, and m represents the number of features.

[0090] Step (5-2) Establish the joint loss function;

[0091] L=λ1*L map +λ2*L center +λ3*L cross ;

[0092] Where λ1, λ2, and λ3 are all hyperparameters used to balance the mean loss function L at detail points. map Cross-entropy loss function L cross and center loss function L center The proportions are λ1 = 0.3, λ2 = 0.125, and λ3 = 1.

[0093] Step (5-3) uses a joint loss function to monitor the backbone network, minutiae extraction network, and multilayer perceptron, and backpropagates to update the parameters of the three. The main purpose is to ensure that the basic features extracted by the backbone network are the representation information.

[0094] Step (6) Test the trained network model and evaluate its accuracy.

[0095] Step (6-1) inputs each fingerprint image of the verification data into the trained backbone network. The output of the backbone network is the final representation information, which is a 192-dimensional feature vector.

[0096] Step (6-2) takes the representation information of a random fingerprint image under each class as a reference sample, and calculates the similarity between the representation information of the remaining fingerprint images and the reference sample. When the similarity between the two is higher than the set threshold, they are judged to be fingerprints of the same class; otherwise, they are judged to be fingerprints of the same class. The same error rate is obtained by batch comparison and is used as an evaluation accuracy index.

[0097] To verify the effectiveness of the proposed method, we used a capacitive fingerprint scanner to collect fingerprints from six fingers of both hands: the thumb, index finger, and middle finger. 70 images were collected from each finger, resulting in a total of 12,740 fingerprint images, which were then used to construct the NB-2034-S2 dataset.

[0098] Based on the above dataset, we validated existing lightweight network methods, and the experimental results are shown in Table 1 below:

[0099] Table 1 Comparison with Existing Mainstream Lightweight Network Models

[0100] method Number of parameters (M) Computational complexity (M) Runtime (ms) Equality error rate (%) MobileNetV1 3.22 294.82 121.36 4.9 MobileNetV2 2.24 159.65 184.23 4.61 ShuffleNetV2 2.26 148.69 94.96 4.14 Ours 0.19 80.61 51.84 1.65

[0101] Experimental results show that our method achieves a lower error rate than existing mainstream lightweight network models, with higher accuracy, while also improving the efficiency in terms of parameter count, computational cost, and runtime.

[0102] Meanwhile, we conducted comparative experiments on large-scale models in the field of fingerprint recognition, and the experimental results are shown in Table 2 below:

[0103] Table 2 Comparison with Large Model Network Models

[0104] method Number of parameters (M) Computational complexity (M) Runtime (ms) Equality error rate (%) DeepPrint 65.95 44595.75 34976.85 0.68 Transformer 24.72 12245.53 4237.80 0.54 Ours 0.19 80.61 51.84 1.65

[0105] Experimental results show that our method has an error rate that is less than 1% higher than that of large model networks, and the accuracy is not much different. However, our method has a much smaller number of parameters, computational cost and running time, making it more suitable for edge devices.

[0106] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0107] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for end-side fingerprint representation recognition based on multi-task learning, characterized in that, Includes the following steps: Step (1) Batch fingerprint image input and preprocessing: The batch fingerprint images are scaled, then binarized and standardized, and the processed image data is randomly divided into training data and validation data; Step (2) Basic Feature Task Learning: Input the preprocessed training data into a weight-shared backbone neural network to obtain basic features, and use the basic features as the starting point for subsequent multi-task learning; Step (3) Minute detail extraction task learning: Input the basic features into the minute detail extraction network, and generate a texture information map through multiple convolutions and deconvolutions; Step (4) Fingerprint category task learning: Input the basic features into the multilayer perceptron to generate topological information and obtain the corresponding category information; Step (5) updates the basic features, texture information and topology information by backpropagating them to the three network modules through a joint loss function, so as to transform the basic features into fingerprint representation information with the help of texture information and topology information. Step (6) Test the trained network model and evaluate its accuracy; The specific steps of step (2) are as follows: The backbone neural network with weight sharing described in step (2-1) adopts the AIFingerNet model and is pre-trained using the SimCLR method; Step (2-2) Input the training data into the pre-trained AIFingerNet model to obtain the output of the last layer of the AIFingerNet model, i.e. the basic features, as the input for the remaining learning tasks. The feature vector size is 192*5*5. The specific steps in step (3) are as follows: Step (3-1) takes the basic features extracted by the AIFingerNet model as input, passes them through a minutiae extraction network consisting of two convolutional layers and one deconvolutional layer, and generates a model of size [size missing]. A 128*128 texture information map is used as ; Step (3-2) Based on the preprocessed fingerprint image from step (1-1), obtain a set consisting of detail points on the fingerprint ridges. : ; in , Indicates the number of detail points. This represents the coordinates of the t-th minutiae in the fingerprint image. Indicates the direction of the t-th detail point, ranging from [0, 2π]. Step (3-3) sets the detail points Encoding as size and The same heatmap was used as the input for the real labels. The encoding method is as follows: Traversal For each point in the equation, represent it as... , Indicates the point at coordinates in Indicates the point at In the channel dimension, the value of this point is denoted as ,but: ； in, They represent details respectively. Spatial coordinates and rotation direction The contribution value, express and The Euclidean distance between them express and The directional difference between them; Steps (3-4) are based on texture information maps. and real label input Using the mean loss function at detail points L map Calculate the contribution of texture information to the final representation information: ； in, Given a two-dimensional matrix with all values ​​of 1, the matrix has a size of (1, a). Let be a matrix of size (a, 1), which is formed by... The result is reconstructed into a matrix of size (a, -1), where -1 indicates that the size of this dimension is adaptive, and then the average is taken over the adaptive second dimension. The specific steps in step (4) are as follows: Step (4-1) takes the basic features extracted by the AIFingerNet model as input, passes them through a multilayer perceptron consisting of two fully connected layers and one drop layer, and obtains the probability of the fingerprint image in each category. The final category of the fingerprint image is determined by the category with the highest probability. Step (4-2) uses the class probability of the fingerprint image and the true label probability after softmax as the cross-entropy loss function L. cross The input is used to calculate the contribution of topological information to the final representation information: ； Where p represents the class probability. This represents the probability of the true label.

2. The end-side fingerprint representation recognition method based on multi-task learning according to claim 1, characterized in that, The specific steps of step (1) are as follows: Step (1-1) Obtain the dataset and preprocess the fingerprint images in the dataset, including: cropping based on ROI region, filling the cropped image into a square image, and scaling the image to M*M size; Steps (1-2) involve iterating through the grayscale values ​​L from 0 to 255 in the preprocessed fingerprint image, dividing the image's pixel values ​​into target and background parts based on the grayscale values, and simultaneously counting the number of pixels N0, grayscale value, Sum0, and mean grayscale value of the target part. The number of pixels N1, grayscale value, Sum1, and mean grayscale value of the background area. And the proportion of each type of pixel in the fingerprint image. Finally, the inter-class variance of the two classes was obtained. : ； Will make The maximum gray value L is set as the threshold, and the pixels are binarized to obtain the binarized image. Steps (1-3) standardize the binarized image to ensure it follows a normal distribution; Steps (1-4) randomly divide the standardized fingerprint images into training data and validation data for further network training.

3. The end-side fingerprint representation recognition method based on multi-task learning according to claim 1, characterized in that, The specific steps in step (5) are as follows: Step (5-1) combines the basic features and the true label probabilities using the center loss function L. center Calculate the contribution of the basic features to the final representation information: ； in, Indicates basic features, represents the probability of the true label, and m represents the number of features; Step (5-2) Establish the joint loss function; ； in These are all hyperparameters used to balance the mean loss function L at detail points. map Cross-entropy loss function L cross and center loss function L center proportion, ; Step (5-3) uses a joint loss function to monitor the backbone network, minutiae extraction network, and multilayer perceptron, and backpropagates to update the parameters of the three, so that the basic features extracted by the backbone network are the representation information.

4. The end-side fingerprint representation recognition method based on multi-task learning according to claim 3, characterized in that, The specific steps in step (6) are as follows: Step (6-1) Input each fingerprint image of the verification data into the trained backbone network. The output of the backbone network is the final representation information, which is a 192-dimensional feature vector. Step (6-2) takes the representation information of a random fingerprint image under each class as a reference sample, and calculates the similarity between the representation information of the remaining fingerprint images and the reference sample. When the similarity between the two is higher than the set threshold, they are judged to be fingerprints of the same class; otherwise, they are judged to be fingerprints of the same class. The same error rate is obtained by batch comparison and is used as an evaluation accuracy index.

Images